Throughout my career working for tech startups I noticed a strange phenomenon. Throughout the beginning of my career I had a fair bit of what some would consider relatively “simple” web development work. Things like developing crud apps, building an API integration, etc. I noticed that even though I would give estimates for my work, my managers and their managers all the way up to the founder would have certain expectations about how long work should take regardless of my estimates, and if my estimates outpaced their expectations then there would be problems. Often times this would be in spite of the fact that many times those above me didn’t have nearly the technical expertise at all to make such an assessment. I chalked it off at the time as simply just a display of systemic mistrust within that organization and tried my best to meet their expectations in spite of them sometimes being unrealistic.
However, as the years have gone by I’ve noticed that the problem extended to almost every company I worked for. The problem became worse when I graduated from simple web development work to distributed computing and what many people would consider “big data” (though I hate the term to be honest). Many of the problems I was now solving were significantly more involved, cluster management, event driven systems, high availability, functional programing, complex distributed graph computations, topics of research and scalable data science. However, I found that the unspoken expectations of the managers above me in terms of how long certain tasks should take still on average remained roughly the same as was when I was doing simple web development work!
They wouldn’t say this at first, you would give your estimates and thoughtfully break your tasks into reasonable chunks and factor in for uncertainties and testing, but still if you sat down and pressed them they would still expect anything that would take more than a couple of weeks to be too involved, and they would assume that the problem was how you are approaching the problem, regardless of how hard the problem actually was. I found myself absolutely astonished that tech founders could be so clueless as to assume that a simple rest api integration should take the same amount of time as a real time transactional distributed ward’s clustering implementation for peta bytes of data, or a highly available complex distributed metastore. Had engineering really come that far in those few short years for this much harder work to be commoditized already? No.
After all of these years, I finally came to one simple conclusion. With all due respect: we are completely clueless about how long things should take.
Next stop: Unicorn Island
Lets explore this point more by means of an extended analogy. Suppose that you wanted to start a new business as a yachting captain. Suppose that you had watched a show my wife will indulge in from time to time, “Below Deck”, and you had determined that you needed one of those gigantic multi million dollar yachts to serve the best of the best, baseball stars and movie directors will be your quarry. This is in many ways analogous to when a startup company decides that they want to serve the fortune 500, companies that have petabytes and beyond of data. However, you as a startup founder have to operate lean, and you are only willing to spend $10,000 on a boat. If you were to walk up to the owner of the multi million dollar yacht and say, I’ll give you $10,000 for that boat, you would be laughed off the dock.
Similarly if you were to walk to a boat construction company (because what does that yacht owner know) and ask them to build a comparable boat, they wouldn’t take you seriously. So what do you do instead? Hire someone that’s an expert in boats to make it work for you. Shift the culpability of making the impossible happen to someone else. So you hire your enthusiastic new employee and and think to yourself, “He will find me exactly what I want, a tremendous yacht that can serve 20 people with luxury amenities, full kitchen, hot tub, etc.” essentially you envision in a piecemeal fashion what you saw before but couldn’t afford. Now suppose that you were to pay them simply a flat fee for their services, and then you give them the budget of $10,000. Now suppose additionally, you didn’t give them the full vision of the boat that you wanted, but simply told them “I want a boat that can transport 20 people” and left out the rest of it.
The boat expert may think to themselves, this is impossible, but I dont want to dissapoint them on my first week, so maybe I can find some type of discounted used ferry for under $50,000 and says “ok lets do this but lets up the budget to $50,000”. Its more than the founder had wanted to spend, but they are willing to make this concession. So the boat finder begins his work, however, as the different parts of the vision start to unfold, and the luxury nature of the yacht starts to reveal itself, the boat expert suddenly realizes they have just become victim to a case of misplaced culpability, and any additional needs that they surface to satisfy the needs of the “luxury yacht” vision is now a issue of their own invention! Even though the business requirements are coming from the owner, they are still surfacing the additional technical requirements needed to satisfy the vision and thus are in many ways now an obstacle to the founders original intention of acquiring a boat for under $10,000. Suddenly, the boat expert is producing problems, and not solutions, suddenly he is a problem.
The parsimonious yachtman isn’t being malicious here. Its just inevietable, that no matter what the boat finding expert does, he will never be able to overcome the fact that the founder really only “values” his new acquisition with a price tag of $10,000 so anything he does contrary to this reality simply weakens his position, REGARDLESS of how right he is in his on going estimation, and additionally he’s already started the engagement by asking for a concession.
Expectations vs Reality
Back to the tech startup, suppose that you think of the development of a piece of software in terms of its TCA (total cost of acquisition), but multiplying the development hours that you spend on something by the developer salaries, and for simplicity suppose that you don’t factor in things like infrastructure costs or license costs for other software. The startup manager says they want a system built that can support X write latency/throughput, Y read/latency and throughput, but the exact nature of the product itself is a WIP due to the fact that they are exploring the market and trying to find the right clients (namely any clients) so need to keep some doors open. Additionally the manager has an idea in the back of his mind that it should take about 2 weeks to build the system by comparing it to another system that he was involved with (unbeknownst to him that system was much much simpler and didn’t have anywhere near the same requirements). The engineer comes back with this simplified description and says he can get a first version produced, but it will take a month instead of 2 weeks. The engineer was being optimistic, and it was a stretch goal already, but he’s new to the company and doesn’t want to seem like he’s a slow poke.
The manager after 2 weeks (his original expectations) has already started breathing down the engineers neck. But why is that? Didn’t the engineer say it would take a month? Not to mention that now that the client focus has shifted, new requirements are now pushing the estimate well beyond the month, but the estimate is not allowed to shift with shifting requirements. It is now the written gospel of expectation for the project, regardless of lip service the company may pay to “agile”.
I could take the comparison farther, but nothing can change the fact that the founder was so far off his original unspoken 2 week expectation that nothing the engineer can do will satisfy what he has in mind and his original budget of 2 weeks was all that he “really” was willing to spend, and a month was a stretch but he was willing to pay a bit more. He was doomed from the start by the manager/founders inaccurate expectation. What happens next? Most likely the engineering lead fails to live up to the expectation and is replaced on the project with another one. But that sounds kind of like an extreme scenario doesn’t it? Would you be surprised to hear that most companies I’ve worked for fell prey to some form of this pattern? Its clearly not an isolated problem.
Startup founders fool themselves into thinking that they can procure assets orders of magnitude beyond what they are willing to pay to procure those assets. In essence, when it comes to engineering assets and engineering time they fall prey to “magical thinking” (likely inspired by some depiction of a hacker they saw in a movie once). If founders/managers were to step back and think about it simply in terms of TCA and compare what they are building to what a company is charging for a similar product they might realize that they are trying to get a $10 million yacht for $10,000. Again, no one is being malicious, they are simply learning from what the history of tech enterprises that have gone before us have taught them. That skilled engineers working in a basement totally clueless to business requirements can produce tremendous value. They don’t realize that so many factors have gone into play in those success stories that went before including luck, timing, market conditions to name a few.
Why can't you just be more like Boris?
The underlying problem still remains that the founder/managers (albeit uninformed, ultimately unchangeable) subconcious expectations about how long the task should take inspired by how much they value that particular component is not even in the same galaxy to what it will cost to build such a system. Some “product development methodologies” like Agile have tried to solve the problem but given us nothing more than a slightly different means to shift culpability to engineering when things aren’t happening as fast as we expect. So how do you solve that problem? The question I have started asking managers and founders is simple. “How much time do YOU want me to spend on this”. This little question has often made them pause. “What do you mean, you are supposed to provide me with an estimate and then I approve it.” Its just not that simple.
Estimates are such a loaded and dangerous endeavor for an engineer. To the point where companies mistakenly talk about estimates like its a “skill” that an engineer can learn. But the reality is that if you can make a probabalistically accurate estimate, then its likely that the task should have been automated by some other means already. In other words, its easy to estimate a task that essentially amounts to copy and pasting some well known CRUD API end point patterns, but any even remotely creative or novel work is almost guaranteed to be totally unknown. Its almost always a way to trap an engineer in a bargaining conundrum. Ie, if you make me provide an estimate first and its no where near what you have in mind (because it won’t be), then you have placed me automatically in a position of begging for additional time. Its not so surprising that this subtle form of manipulation in the sales tactic of getting us to put our cards out the table first comes from the business end of the company. So as engineers, what do we have to do?
But estimates are our friends, it conveys expectations back to the company. The truth is that isn’t how it actually plays out, and this needs to become our new engineering battle cry. If engineers stop giving estimates for their work and simply ask for deadlines then it changes the dynamic of the conversation. If they say that they want to spend $10,000 on acquiring the boat, then we can come back and say, well I won’t be able to give you the luxury yacht you had in mind, however, I can find a used sailboat and after we fix it up you can possible charge high end clientele for exclusive sailing excursions. It puts us back in the superior position of negotiation. Then as engineers, we have the freedom to come up with a solution that meets your expectations and simultaneously doesn’t put us in an impossible situation. Or at a minimum at least veto the idea all together as not being possible to achieve in the desired time spans. Besides 99% of the time you end up getting a deadline anyway, so why even go through the excercise of making estimates?
Engineering is not getting simpler, its getting more and more complex, because we are solving harder and harder problems, which means that these things (which all haven’t been commoditized yet, in spite of the fact that many founders secretly think that all engineering problems have already been reduced to the drag and drop simplicity of building Wix websites) will take longer to solve and will have a higher TCA. As a result, the gap between the expectations of founders/managers in terms of time to implementation and the amount of time we should be spending building these systems is becoming light years apart. As a result, we continue to push out half baked solutions cobbled together with duct tape and chicken wire just barely hobbling across the finish line so we can toss it off to operational teams and burden them with our rushed decision making for the rest of the life of the company or until they burn out and leave (guess which will come first).
The problem will get much worse until engineers start hitting their employers with much needed reality checks on what it takes to solve these problems. For startup founders/managers, if I cannot appeal to your sense of pity for your employees, I hope I can at least appeal to your sense of pride: Don’t be the guy asking the $10 million yacht captain if he wants to sell it for $10,000.
Kyle Prifogle