Programming is not strictly an engineering exercise but a creative one. It’s not a matter of time invested, intelligence or effort to whether or not AI will replace programmers. It’s a fundamental limitation to how AI is currently organized in relation to its users. All AI in its current form can do is synthesize and recombine existing ideas created by humans. It requires our ideas to function, it feeds on them like nectar, without which it would stagnate.

Think of AI in its current organization like a parrot or a DJ, not a musician. It does not create new ideas on its own because the creative process doesn’t come from just mimicking other artists’ ideas. The creative process is rooted in an embodied agent reflecting and subverting their relationship to their environment which results in reputational risk and possible societal reward. In other words, we are in the real world, we transcribe whats actually happening here into forms like text, which is a profoundly lossy down sampling of information when you think about all the things we do on a daily basis.

I will sample the cover of the cover of the cover of the cover of "Imagine"

AI doesn’t really have reputational risk, it doesn’t have embodiment and it doesn’t have a relationship to its local environment because its environment is the entire social sphere of those who interact with it according to how its RLHF feedback is specified. Until those things exist AI can’t really create, and its responses will be a muted average or pastiche of societies collective sentiments, not the reflection of an independent agent striving to orient themselves in relationship to its environment.

Because of this IF everyone stops creating and depends on AI to create, like the scaremongers and doomsayers are saying, then its tap of ideas which it can plagiarize is cut off and all human intellectual endeavor is stunted. AI and ourselves both die a slow death of lack of ingenuity.

Can AI become embodied and achieve agency? Sure! But it’s a much different conversation if we are talking about the neighbor’s android spewing misinformation on the internet than if we are talking about ChatGPT doing so. A much more human conversation.

A large part of my career has been spent vacillating between what I would call entrenched platform teams with increasingly vague notions of how their solutions are utilized and scrappy reactive startup teams hobbling by on bailing wire and duct tape solutions. I’m usually the person sitting on the fence trying to balance the situation out, but some situations are beyond repair. That said, I spent a large part of my early career being in the former camp, until slowly over time, and through owning my own business, I started to realize that part of the utility of software itself is the ability to move fast. Ask someone in microprocessor manufacturing sales where the lead times for development are 3-5 years if they would be happy about being able to push out a product in two months. The tendency for engineers to fall exclusively into the former camp comes from a number of factors, not the least of which is cultural bias amongst their engineering peers.

So the question is how do you bridge those two modes and create a funnel for long term investment which doesn’t divorce it entirely from the opportunities that the business needs to pivot to chase while still creating foundations that can be grown upon and make an impact in the long term?

A lot of us are familiar with the concept of skunk works projects. Companies can embrace this mode of working, and usually reserve it for those who are closest to the customer, data analysts for example. But the universal attitude of engineering towards these projects is that they are “throw away”. IE once the engineering team gets in there, there is almost nothing salvageable that can be leveraged, the solution is so fundamentally flawed that no “proper” engineering can be applied to it, they have to go back to square one.

A more conventional way of solving this problem is through what is referred to as “product development”. This is a process of learning the purely economic and business strategic dimensions of a project and collaborating with engineering to make sure that the two are aligned. The problem frequently with this approach is that product development doesn’t usually have the vocabulary to be able to speak effectively with engineering and visa versa that they are frequently talking past one another.

Another problem is that just the sheer weight of additional process prevents the business from being able to move quickly in reaction to macro-economic fluctuations that affect the business. You don’t want to be purely reactive, but you also don’t want to put your head in a cave and watch the world go by. At the end of the day you want to be able to benefit from both of those mindsets.

These problems become even more exaggerated when you start to operate in a complex space, such as one with heavy data science or machine learning. In these environments, data scientists and analysts start to function more properly like the product development team because the context is too complex for product development to be effective at it. Now we have three parties, product, engineering and analysis from three distinct disciplines speaking different languages.

What’s more, treating the results of data science or machine learning projects as just one offs or skunk works projects is sometimes not possible because engineering usually doesn’t have the capability to start back at square one with those projects, it may be too complex or involve mathematical machinery that’s not relevant to engineering experience. Platforms specifically at this intersection like Tensorflow help with this problem but it eschews a rather narrow path through the space. With engineering being forward focused and thinking about secondary and tertiary effects of engineering decisions engineering teams usually stand distinct from the other two, creating an ever widening gap to analytics and product. The problem becomes exaggerated when engineering teams have their own product development team, which usually has the counter intuitive affect of widening the gap, and entrenching / defining the platform to an exaggerated degree making it drift further and further away from the opportunities which sustain the business. Gasoline to the fire here are various biases (superiority complices?) about throroughness and attention to detail that engineers tend to hold up as a crutch or as an integral part of their identities.

Opportunity Engineers

This is where I introduce the idea of “Opportunity Engineering”. Engineers who have a dual purpose of extending and building out the platforms which power the team and working with platform teams to extend their work in measured ways while simultaneously helping to build short term opportunity projects which are not simple one offs but build on and extend the long term vision. If the engineering team and platform are a cruise ship, then the opportunity engineers are the tugboats using engineering to pull them gently making small course corrections to ensure that they are veering into the right dock at the right time. Opportunity engineers wear many hats, sit on the fence between several teams and move rapidly between ideation, implementation, maintenance and extension.

Unlike solutions engineers who are often times extensions of a sales team, the opportunity engineer is a full engineer, capable of working at any level of engineering complexity, but with the specific mission of working projects which accelerate the companies ability exploit opportunities which may otherwise be unattainable by traditional engineering teams. It’s often said that engineering teams need to keep their feet on the ground, whereas sales and product function to pull the company into the clouds, but what this often causes is a misalignment problem, a bimodal working environment in which engineering teams are locked into tunnel vision exclusive prioritizing its own needs, leaving end to end teams to fend for themselves. Over time the gap widens and the only thing to resolve the problem is total reorganization. It’s the job of the opportunity engineer to serve as an elevator between the two to make sure that one is never veering too far away from the other. While its traditionally thought of as the job of the product development team to do this, there are some environments, such as complex machine learning or data science environments where this is practically impossible without having someone with engineering experience embedded on those teams.

Additionally having an engineer function in this space gives freedom for other engineers to focus on the well-defined and more certain tasks in the near term knowing that they don’t have to worry as much about generalization. An opportunity engineer can observe that the business requests fall into a certain pattern of behavior and then advocate for extending the platform to have degrees of flexibility which will serve the business’s end goals. While it seems like have a specific focus on this aspect could lead to over-generalization it actually has opposite effect. Because the opportunity engineers are closer to the opportunities of the business, they know which generalizations are likely to bear fruit for the business. Platform engineering teams on the other hand are often making best guesses of what dimensions to generalize based upon direction from product and trying to “read the tea leaves” of where the business is going based upon second hand signals. This then frees up the platform and product engineers to only build what’s been well defined and laid out before them without having to worry about whether or not its well aligned with the chaotic and ever changing business environment around them.

One thing needs to be clear though: An opporutnity engineer wouldn’t just be shooting from the hip and chasing butterflies and shiny objects. They would be deliberately exploring the space around the team like a reinforcement learning algorithm in an “exploration” mode freeing up the rest of the team to be in the “exploitative” phase of running with what they discovered.

Ultimately every engineer needs to develop a product mindset and orient themselves as a kind of opportunity engineer, after all, you have to know how the bread is getting on the table in order to make sure you are making the right moves to ensure it keeps happening, however, not all engineers have a preference to work in this way. Therefore we should exploit as much as possible the engineers which are willing to be scrappy and teach them how to think long term so that they can bridge the rest of the engineering team to the business and keep their efforts honest and well oriented.

At the end of the day all engineering is opportunity engineering, but it also takes one of every kind to make a ship float.

~ Kyle Prifogle

• An employee mischaracterizes a coworker to their manager behind their back
• An employee ices a coworker out of a domain and hordes work because they want to protect their territory
• An employee power plays their coworker in order to establish dominance in a domain
• A manager uses a carrot to motivate an employee to work harder but then doesn’t follow through on it
• A manager sets up situations that pit two employees against one another in order to get them to work harder
• A manager undercuts or fails to support an employees work because they find them intimidating
• A manager isolates employees from one another rather then urging them to work out their differences
• Teams become pre-occupied with defining boundaries and defending territories rather than sprinting towards the goal together

This type of environment is toxic to it’s core, it takes individuals who excel out of genuine curiousity and puts them at the mercy of a power heirarchy with no motivation other than to preserve it’s own existence. It is like Zamaytin describes his dystopian society in “We”:

The weight of the system crushes the dream; 
under the burden of fear and suspicion, 
life becomes a shadow play, a farce.

Work on a team is NOT a zero sum game

The common theme of all these scenarios is the idea that when one employee/manager/team wins the other loses and visa versa, in other words that achievement within a team is a zero sum game. This type of mindset has been called a lot of things, the” fixed mindset” for example but it’s ultimately rooted in the idea that the enemy is “within the gates”, meaning that you need to be on guard against those you work with. However, the realization that needs to be made is that the enemy is “outside the gates”. If there is an enemy at all, the competition is that enemy, and that adversary is outside of any individual or team in the company, it stands as a harrowing specter shadowing the team motivating them to charge out to the gates to meet it. Combat analogies seem to really drive this point home, for example, in Braveheart after William Wallace gives his speech, the soldiers don’t then turn to one another and start stabbing each other in the back, but they focus their attention outwards towards the enemy, because they have no choice.

The realization that needs to be made is that collaboration and working together ultimately results in a scenario where everybody wins to a far greater degree than if the individuals within the team were defeating one another in a zero sum game, especially when faced with what is or may become an existential threat. Whatever aphorism you want to use for this concept “the sum is greater than the parts”, “make more pies”, “growth mindset”, this is the exciting opportunity that can be embraced and seized for teams with the capability to do so. To address my previous curiosity about how Spotify is different, I think its possible that someone at the company, maybe even Daniel Ek himself realizes this principal very deeply, and the ripples of that intention are why it is such a great place to work.

See how many of the above scenarios begin to dissolve away when applying this new mindset

• An employee no longer wants to mischaracterize a coworker because this limits their coworkers ability to work, which in turn affects the employees ability to thrive too
• An employee doesn’t want to ice a coworker from a domain because without that employee in the domain the domains is less likely to be successful, they want to share work and the credit that comes with doing that work well
• An employee wouldn’t power play a coworker because the more their coworker feels powerless, the less likely they are to be able to be impactful on the team
• A manager wouldn’t use a carrot to motivate an employee because once the carrot is removed the employee becomes burnt out
• A manager wouldn’t want to pit employees against one another to work harder because while that gets a short term productivity boost it leads to burn out and resentment
• A manager would want to support an employee regardless of how intimidating they find them because the further their employee goes, the more likely they are to succeed too
• A manager would never isolate employees from one another but rather try to get them to resolve differences because united they are more strong
• Teams would become less pre-occupied about functional boundaries and territories and realize that the entire companies territory is what really matters

In other words, the best way to serve ourselves is to be selfless to those around us! Within a team, we have to support one another’s dreams, ambitions and goals better and stop viewing the world through an adversarial lens. I think Paul said it best:

Do not let any unwholesome talk come out of your mouths, 
but only what is helpful for building others up according to their needs, 
that it may benefit those who listen.

Realizing this I think back on my career and how imperfectly I may have realized this idea in the past and the impact that it may have had. Just as a single individual in an environment embracing a growth mindset can have rippling effects on everyone around them for the better, a single act of self interest, no matter how benign can have devestating effects on a team and spread like a fire from the mind of one individual to the next until it burns up every ounce of energy that we could have used to create something truly great and beautiful for the world. It places on us a heavy burden, but its not without any hope for redemption when we don’t get it perfectly right.

~ Kyle

This theory goes something like this:

1. OpenAI opening up their api will result in the internet being flooded with ChatGPT content.

2. ChatGPT cannot actually tell if content is written by itself, especially short form content, 100% of the time.

3. ChatGPT in order to be retrained must now train on data that it produced itself. There will be an attempt to filter out generated content, but the training process will inadequately identify generated content and not be able to do a good enough job of this. We will in fact discover that in a lot of scenarios its impossible to distinguish generated content from real.

4. This will create strange artifacts and behavior in ChatGPT, how this plays out exactly I’m not sure. Open to suggestions on how it starts to decohere. This is the key point to discuss. The outcome could be much more severe. Can ChatGPT succesfully train on its own generated data and continue to progress? Will the outcome of this be decoherence, hyperbolic exageration of certain behaviors, or something else, maybe even evolution/progression? This is a classic feedback loop either way.

5. We will have no choice but to only train ChatGPT on versions of the internet that existed before OpenAI made their API’s publicly available. Hello Internet Archive!

Will the feedback loop be constructive or destructive?

There are some ways OpenAI can attempt to mitigate this issue, such as somehow encoding a signature into generated content that indicates that content is generated, but this is ultimately an arms race scenario and it will be a constant battle to fight ways to forge or bypass that functionality.

The end result is that ChatGPT will be stuck in this day and age forever. If this theory is correct enjoy it while it’s current! It will soon be stuck in a time capsule.

Alot of the talk around ChatGPT focuses around apolyptic images of humanity enslaved by a matrix like machine world.

Calm down this isn't the future

While this is fun and compelling to think about, its not really practical and plays on fantasy more than reality. The reality is that ChatGPT has only exhibited “intelligence” in the most basic sense of borrowing and recycling human intelligence.

However, ChatGPT is a threat but not in the way that most are anticipating. The threat of ChatGPT is that it completely destroys the internet as we currently know it. While it is true that the portion of the population incapable of distinguishing ChatGPT from real human intelligence will continue to be enalsved to the internet under their new ChatGPT master, and this is a sad state of affairs, the net effect isn’t enslavement of humanity as a whole, but a near total abandonment of the internet itself by most reasonable people.

Here is a likely course of actions based on recent events:

1. Noticing the potential of not having to pay writers and just generating articles based on click-bait by-lines internet content creators will fire all their creative staff and use ChatGPT to generate all their content. This results in the internet being flooded with non-human created content. We are on the verge of this point.
2. OpenAI will create browser plugins to detect when machine generated text is on your website. They have already done this essentially.
3. Third parties will notice the conflict of interest and potential for abuse and cencorship of OpenAI having both the algorithm and the cure for it and will develop their own plugins to detect ChatGPT.
4. Third party plugins will be in an arms race with OpenAI for a number of years without much success, resulting in total confusion about whether anyone is being effective at detecting generated content.
5. People craving genuine human interaction and creativity will begin abandoning the internet altogether as an unreliabe source of information.

While its maybe a good thing we collectively take a break from the internet as a whole, if you are connected to any company that makes is living off of advertisements on text based content of any kind, this includes Meta, Twitter, Reddit, any news website, then I would run away from them as fast as you possibly can because they are on the verge of catastrophic collapse. The best we can hope to come from their ashes is a kind of Wikipedia alternative, but likely Google will buy them all in a firesale once they have fallen. Instagram will continue to be safe for a time because detecting generated images is a fundamentally easier thing to do, but over time they will fall prey as well.

This is much more likely

I’m fortunate in that I work for a music company and machine generated music hasn’t really reached a state of maturity yet…hold up…Nevermind we are good.

This begs the question though that many of the companies that are involved with OpenAI and ChatGPT and its underpinnings are also profiting from selling advertising on text based content. Isn’t this self destructive? Maybe they don’t realize that the tool that they have developed to “help” their products may inadvertantly end up flooding it with useless content and destroying it. Or maybe it’s a meta strategy to level the playing field or knock out the competition. The main controller of OpenAI right now is Microsoft, who has failed as of yet to get a foothold in social media. Maybe this is ultimately their way of evening the odds.

Its a pattern that we know all to well because we see its presence emerge naturally overtime in the companies we work at. Companies organizing their technology to be complementary in such a way that supplements productivity and uses the right tools for the right job. Javascript in the front, ruby in the middle java in the back. Or maybe javascript in the frontend, python in the middle, C++ in the backend. The reasoning for this type of organization is usually a combination of culture and hiring conveniences, and because those technologies happen to complement one another in particular ways, but have we stopped and really thought about specifically why this type of structure has emerged over time? An interesting answer to that question comes in thinking about the specific characteristics of those languages and their strengths and weaknesses in the context of a concept I call “churn”.

Note: This diagram is deliberately a stereotype, think of it as a projection along the dimension of churn.

Before I go further I should explain what I mean by “churn”. Churn is a terminology I have borrowed from marketing that is typically referred to as “customer churn” or the rate of change with which customers turn over. In the context of software systems I refer to churn as meaning more generally “rate of change” within a software system. You can think of this as the churn rate of the code, but it also can refer to the rate of change of the use cases, configurations, and business systems surrounding the code itself. In essence it is the rate of change imposed by external systems, end users, business requirements on the desired software system to solve that particular business problem.

Thinking about the previous “classic enterprise architecture” in this context presents some interesting ways of viewing the phenomenon. You start to see that on the edges of the software system where it is interacting with external systems (typically end users), there is more churn in the system, and that we compensate for that greater rate of churn by adopting languages and paradigms that are more dynamic and ammenable to that higher rate of change.

Churn in analogy to thermodynamic entropy

One way to think about this is in terms of entropy of heat in materials engineering. You design materials to accomodate the highest heat rate on the outside then gradually develop different layers of materials with other desirable characteristics that need to be more insulated from the heat going gradually internal to the material. Another analogy that I use alot when thinking about designing reactive systems to be able to handle a large amount of data is thinking about traffic patterns, off ramps, speed zones as various tools you can use to take a heavy amount of traffic and gradually start to siphon off parts of it until you have distributed a manageable amount of traffic to the right places. Yet another analogy I use in this regard is that of water sluiceways and how they are used to prevent floods. You get the idea, but long story short the same analogy can be used here substituting “traffic volume”, or “water volume” for “churn volume”.

The Advantages of a Churn Perspective

This is the part of the argument where I get a bit controversial but not without a point. I argue that each language and its accomponying libraries, ecosystem and even the culture surrounding that language has very specific strengths that should be capitalized on. Its when we start to think of a single programming language as having “universal appeal” that we start to get into trouble. No offense to the node.js crew as this is purely my perspective and personal preference, but I cringe a bit when I think about designing back end systems with javascript. Typescript emerged on the scene and made that prospect a bit more sensible, but to a large extent typescript is just an attempt to extend the language to capitlize on a particular strength that it was not designed to have in mind, perhaps at the mercy of abandoning its key historical strengths. You could probbably make the same argument in reverse about scala.js or other attempts to introduce generic or dynamic programming to strongly typed languages when maybe the best thing to do is to embrace the dynamic nature of the underlying thing you are trying to model instead.

This is where I come to my recent experiences with Formation.ai and some of the insights that I have had recently around its architecture. Formation is an amazing and challenging place to work for a number of reasons, and if you are a data engineer I highly recommend it. The main kind of defining feature of Formation is that it has been seeded from the beginning with ultra brilliant often times phd level practitioners of functional programming. While simultaneously Formation also finds itself at the extremely churn-filled world of online marketing as it builds incredible reinforcement learning inspired customer loyalty programs for major brands that are bleeding edge and deliver tremendous value. The difference between these two worlds is tremendously challenging and produces some amazingly novel concepts.

A main desirable characterstic of functional programming is that it allows you to define extremely safe maintainable software systems. Systems that you know exactly what they are doing at any given moment. This is an over simplification obviously but relative to other approaches there is a kernal of truth here. However, one aspect of functional programming that makes it challenging is that it generally takes longer to build systems that are designed to be purely functional. I’ve had many conversations with my collegues and I realize that this is also an oversimplfication, and in their perspective perhaps a straight falsehood. They would argue, and I think this is completely valid, that for someone sufficiently experienced in FP it actually takes less time to implement systems with a pure FP approach than any other approach and that you get the added benefiets of pure programming as a bonus. The key phrase there is that it takes “someone sufficiently experienced” to be able to implement such a system, which if you look at the average of all the individuals that you have available to your hiring pool, and the amount of added implementation time added for lesser experience it averages out to a longer amount of time to implement than say throwing together a quick python script. And thats totally fine because you get so many added benefiets from FP that its often time worth the investment.

However, lets consider the aspect of churn that I brought up before. Imagine if you start to model a particular type of relationship you observe in the business patterns and develop the FP algebra that models that relationship over the period of a week and implement its requisite categories using a very pure FP approach only to find that within that week those relationships have fundmanetally changed or been abandoned altogether multiple times. In general, we find that the business world wants to move faster than the engineering world. That impendence mismatch is well known to us all so needs to be respected. Then the challenge becomes that (going back to the entropy analogy) given the business operations are “hot”, how do we arrangement our materials (layers of software systems) in such a way to best accomodate that reality and insulate systems that are less engineered to accomodate for that high rate of churn but have other characteristics such as maintainability which we desperately need if we are to succeed long term.

Credit: Toggl engineering team

Notice here that my argument is less that you can’t build an entire software system from the front end all the way to the back end in a single language (such as haskell, or on the other end javascript), but that sometimes doing so might be extremely impractical. Another way to put it is, if certain languages and paradigms require you to invest heavily in them, then it may make sense to be more choosy about when you chose to implement them, ie, dont use quantum mechanics to tie your shoes, and conversly, don’t try to use the bunny trick to trap an ion.

An Architecture That Embraces Churn

When developing the pilot architecture for Formation, we wanted to respect this concept of “churn”. As we anticipated given that many of these systems were pilot programs, they would be changing very rapidly and often times with little notice. As a result we sat back and took stock of the assets (both people and code) that the company had accumulated to that point. The company is a good mix of front end engineers (primarily javascript developers), data scientists (primarily python developers), data engineers (scala and python developers as well as SQL guru’s), and functional programmers (type level scala developers and haskell developers). Over time the company has moved in a direction to embrace for every time a more polyglot attitude which is good, but the remaining kind of core competencies remain. So given that set of assets at our disposal what type of system could we develop that respected the heavy “churn” idea. The following is what we came up with:

Notice a key aspect of this architecture is using SQL as what I call a “data configuration” language via means of Dremio. Dremio for those who aren’t familiar is a tool based on Apache Arrow and Apache Calcite that allows you to use SQL to specify virtual data tables which you can then access via means of JDBC for example to use in other Spark pipelines (for example). Think of it as a competitor to Presto with a materialization framework for external views and a snazzy front end with the ability to read from multiple types of data stores other than s3.

Its an interesting tool, that seems to be exactly the “right tool for the job” in this particular project, despite its lack of maturity around things like having a seperate persistent metadatabase that you can interact with directly (unlike hive), which we were able to get around by building a migration tool in scala that pulled the metadata into version control using the Dremio rest api. This allows Dremio to serve as the outer most, most flexible layer of our system, the one thats is most exposed to churn and is allowed to be deliberately client bespoke. What this allows us to do is massage the data into “canonical” tables which the rest of the system can use and dramatically siphons off a signifigant amount of churn. The other thing it allows us to do which is really intersting is defer product decision making until we have a statistically signifigant pattern emerge around our clients. In essensce Dremio becomes our product feature hopper, which we dig into and pull components from and develop later as we start to see patterns emerge in the Dremio queries between clients.

From there you can see the continuum from the very high churn Dremio SQL configuration layer down into the datascience which is still dynamic and flexible but requires more rigorous process around developing, down to the data engineering which is more type safe and written in scala and heavily leverages the canonical tables to prevent the code from having to change very much, all the way into the haskell layer which could be completely application agnostic and more “core” operations which don’t need to change constantly but need to be ultra safe and maintainable. I argue that this particular arrangement of using the right tools for the right job is both intuitive and puts the right pressures in the right places so that the company can operately at maximal efficiency in light of the highly churn based environment it finds itself in. Notice that the symmetry of the pattern of low to high churn exists even on the egress side of the fence where Formation has to configure sometimes very bespoke measurement metrics and guidelines to prove attribution of their value proposition. Basically any time the system is exposed to an “end user” or “non specialist user” of some sort we see these layers of churn insulation (going back to the thermodynamic analogy) emerge towards that eventual end goal.

Its also interesting to note how this parallels many of the data strategies that we’ve seen emerge over the years that goes from very unstructured data to highly use case specific structured data. In our case also the fact that we are using a data virtualization tool which helps to aleviate some of the heavier ETL that would be involved in such a data pipeline. The fact that Dremio is a distributed technology also means that it can scale to most our use cases in the batch realm within acceptable SLA’s (barring the complexity of reflection management which can sometimes make the SLA’s a bit more dicey). In our case we use it as part one of a one two punch for scalable data processing, the second being Apache Spark. Basically most use cases of Spark SQL we have moved into Dremio as its a tool that allows non specialist to query the data without having to know python or scala. This is really useful for things like debugging, QA and validation teams.

One note of caution, Dremio is still a green tool, and has its fair share of problems, particularly around materialization scheduling and handling. We realized after the fact that we could likely get the same type of behavior out of AWS Glue Data Catalog and Athena (Spark’s ability to point its metadata at the hive metastore-glue underlying athena is especially convenient) to handle the “services oriented metadata specification”, with some additional work around specifying certain views to be maintained as materialized views for quicker access. Moving forward this is likely what we will use in place of Dremio. A similar tool with alot of promise is being developed by LinkedIn called Dali which seems to be solving a similar problem as well, though its not yet open source. Another alternative could have been something like zeppelin or Databricks notebooks but it still would limit the users to people knowledgeable of python or scala. Deltalake SQL could have helped here but databricks overall felt like it was too disruptive to introduce at the time and still wasn’t quite as user friendly to non programmers.

Types of Complexity in Data Engineering

To summarize I want to discuss various types of complexity in software engineering for data systems. There are 3 broad types of complexity here that I would like to shed light on as a result of this architecture, the third being perhaps the most overlooked:

1. System Complexity (internal complexity): This is what we typically think of when we refer to complexity in software. In general, how well does the software function on its own when no external forces are changing it or exerting pressure on it, ie how well does the software do its job. How well does it deal with volume, throughput, reliability and stability, latency. How easy is it to provision and maintain the infrastructure to run the software.

2. Operational Complexity (external complexity): This is some of the types of complexity we refer to when we say complexity in software but not all of it. This essentially is how well does the software cope with external business processes and change. How easy is it to extend the software and evolve it over time. How easy is it to configure. How easy is it for users to get analytic or diagnostic information out of the software. Churn as it pertains to business configuration falls into this category and the concept Churn Based Programming focuses a light on this type of complexity.

3. Social Complexity (meta complexity): This is the most overlooked category and its the type of complexity that comes with the fact that programming languages are maintained by people that have social structures that either reinforce or tear apart the use of languages. Things that would matter here include categories such as rate of adoption of a language, which is impacted by how comprehensible it is for a new learner. How fluid development in the language feels. What types of IDE’s or other development tools exist to be productive. What are the written or unwritten dogmas surrounding a particular language and how does that impact intentionally or unintentionally the proliferation of the use of the language. What are the interests of the community surrounding a particular language historically. For example, python and data science, javascript and front end development. Churn Based programming pertains to this type of complexity in the sense that it encourages certain types of languages with profiles of social complexity to focus on problems well suited for that community rather than attempt to be a universally applicable. IE, it embraces the diversity and strengths of programming languages.

I will end my discussion here as its starting to get long, but I hope that Churn Based Programming is found as a useful concept that others will draw inspiration from and try to find other more formal ways of expressing it. I’m eager to learn of more obvious or orthodox expressions of the same idea. In my next post I’m going to continue the discussion about programming languages and talk about the dueling concepts of Cohesion and Coupling and how it affects certain aspects of what programming language you might chose for a particular task, and think about those ideas in the lens of program churn.

Kyle Prifogle

Throughout my career working for tech startups I noticed a strange phenomenon. Throughout the beginning of my career I had a fair bit of what some would consider relatively “simple” web development work. Things like developing crud apps, building an API integration, etc. I noticed that even though I would give estimates for my work, my managers and their managers all the way up to the founder would have certain expectations about how long work should take regardless of my estimates, and if my estimates outpaced their expectations then there would be problems. Often times this would be in spite of the fact that many times those above me didn’t have nearly the technical expertise at all to make such an assessment. I chalked it off at the time as simply just a display of systemic mistrust within that organization and tried my best to meet their expectations in spite of them sometimes being unrealistic.

However, as the years have gone by I’ve noticed that the problem extended to almost every company I worked for. The problem became worse when I graduated from simple web development work to distributed computing and what many people would consider “big data” (though I hate the term to be honest). Many of the problems I was now solving were significantly more involved, cluster management, event driven systems, high availability, functional programing, complex distributed graph computations, topics of research and scalable data science. However, I found that the unspoken expectations of the managers above me in terms of how long certain tasks should take still on average remained roughly the same as was when I was doing simple web development work!

They wouldn’t say this at first, you would give your estimates and thoughtfully break your tasks into reasonable chunks and factor in for uncertainties and testing, but still if you sat down and pressed them they would still expect anything that would take more than a couple of weeks to be too involved, and they would assume that the problem was how you are approaching the problem, regardless of how hard the problem actually was. I found myself absolutely astonished that tech founders could be so clueless as to assume that a simple rest api integration should take the same amount of time as a real time transactional distributed ward’s clustering implementation for peta bytes of data, or a highly available complex distributed metastore. Had engineering really come that far in those few short years for this much harder work to be commoditized already? No.

After all of these years, I finally came to one simple conclusion. With all due respect: we are completely clueless about how long things should take.

Next stop: Unicorn Island

The Parsimonious Yachtman

Lets explore this point more by means of an extended analogy. Suppose that you wanted to start a new business as a yachting captain. Suppose that you had watched a show my wife will indulge in from time to time, “Below Deck”, and you had determined that you needed one of those gigantic multi million dollar yachts to serve the best of the best, baseball stars and movie directors will be your quarry. This is in many ways analogous to when a startup company decides that they want to serve the fortune 500, companies that have petabytes and beyond of data. However, you as a startup founder have to operate lean, and you are only willing to spend $10,000 on a boat. If you were to walk up to the owner of the multi million dollar yacht and say, I’ll give you $10,000 for that boat, you would be laughed off the dock.

Similarly if you were to walk to a boat construction company (because what does that yacht owner know) and ask them to build a comparable boat, they wouldn’t take you seriously. So what do you do instead? Hire someone that’s an expert in boats to make it work for you. Shift the culpability of making the impossible happen to someone else. So you hire your enthusiastic new employee and and think to yourself, “He will find me exactly what I want, a tremendous yacht that can serve 20 people with luxury amenities, full kitchen, hot tub, etc.” essentially you envision in a piecemeal fashion what you saw before but couldn’t afford. Now suppose that you were to pay them simply a flat fee for their services, and then you give them the budget of $10,000. Now suppose additionally, you didn’t give them the full vision of the boat that you wanted, but simply told them “I want a boat that can transport 20 people” and left out the rest of it.

The boat expert may think to themselves, this is impossible, but I dont want to dissapoint them on my first week, so maybe I can find some type of discounted used ferry for under $50,000 and says “ok lets do this but lets up the budget to $50,000”. Its more than the founder had wanted to spend, but they are willing to make this concession. So the boat finder begins his work, however, as the different parts of the vision start to unfold, and the luxury nature of the yacht starts to reveal itself, the boat expert suddenly realizes they have just become victim to a case of misplaced culpability, and any additional needs that they surface to satisfy the needs of the “luxury yacht” vision is now a issue of their own invention! Even though the business requirements are coming from the owner, they are still surfacing the additional technical requirements needed to satisfy the vision and thus are in many ways now an obstacle to the founders original intention of acquiring a boat for under $10,000. Suddenly, the boat expert is producing problems, and not solutions, suddenly he is a problem.

The parsimonious yachtman isn’t being malicious here. Its just inevietable, that no matter what the boat finding expert does, he will never be able to overcome the fact that the founder really only “values” his new acquisition with a price tag of $10,000 so anything he does contrary to this reality simply weakens his position, REGARDLESS of how right he is in his on going estimation, and additionally he’s already started the engagement by asking for a concession.

Expectations vs Reality

Back to the tech startup, suppose that you think of the development of a piece of software in terms of its TCA (total cost of acquisition), but multiplying the development hours that you spend on something by the developer salaries, and for simplicity suppose that you don’t factor in things like infrastructure costs or license costs for other software. The startup manager says they want a system built that can support X write latency/throughput, Y read/latency and throughput, but the exact nature of the product itself is a WIP due to the fact that they are exploring the market and trying to find the right clients (namely any clients) so need to keep some doors open. Additionally the manager has an idea in the back of his mind that it should take about 2 weeks to build the system by comparing it to another system that he was involved with (unbeknownst to him that system was much much simpler and didn’t have anywhere near the same requirements). The engineer comes back with this simplified description and says he can get a first version produced, but it will take a month instead of 2 weeks. The engineer was being optimistic, and it was a stretch goal already, but he’s new to the company and doesn’t want to seem like he’s a slow poke.

The manager after 2 weeks (his original expectations) has already started breathing down the engineers neck. But why is that? Didn’t the engineer say it would take a month? Not to mention that now that the client focus has shifted, new requirements are now pushing the estimate well beyond the month, but the estimate is not allowed to shift with shifting requirements. It is now the written gospel of expectation for the project, regardless of lip service the company may pay to “agile”.

I could take the comparison farther, but nothing can change the fact that the founder was so far off his original unspoken 2 week expectation that nothing the engineer can do will satisfy what he has in mind and his original budget of 2 weeks was all that he “really” was willing to spend, and a month was a stretch but he was willing to pay a bit more. He was doomed from the start by the manager/founders inaccurate expectation. What happens next? Most likely the engineering lead fails to live up to the expectation and is replaced on the project with another one. But that sounds kind of like an extreme scenario doesn’t it? Would you be surprised to hear that most companies I’ve worked for fell prey to some form of this pattern? Its clearly not an isolated problem.

Startup founders fool themselves into thinking that they can procure assets orders of magnitude beyond what they are willing to pay to procure those assets. In essence, when it comes to engineering assets and engineering time they fall prey to “magical thinking” (likely inspired by some depiction of a hacker they saw in a movie once). If founders/managers were to step back and think about it simply in terms of TCA and compare what they are building to what a company is charging for a similar product they might realize that they are trying to get a $10 million yacht for $10,000. Again, no one is being malicious, they are simply learning from what the history of tech enterprises that have gone before us have taught them. That skilled engineers working in a basement totally clueless to business requirements can produce tremendous value. They don’t realize that so many factors have gone into play in those success stories that went before including luck, timing, market conditions to name a few.

Why can't you just be more like Boris?

A Radical Proposal

The underlying problem still remains that the founder/managers (albeit uninformed, ultimately unchangeable) subconcious expectations about how long the task should take inspired by how much they value that particular component is not even in the same galaxy to what it will cost to build such a system. Some “product development methodologies” like Agile have tried to solve the problem but given us nothing more than a slightly different means to shift culpability to engineering when things aren’t happening as fast as we expect. So how do you solve that problem? The question I have started asking managers and founders is simple. “How much time do YOU want me to spend on this”. This little question has often made them pause. “What do you mean, you are supposed to provide me with an estimate and then I approve it.” Its just not that simple.

Estimates are such a loaded and dangerous endeavor for an engineer. To the point where companies mistakenly talk about estimates like its a “skill” that an engineer can learn. But the reality is that if you can make a probabalistically accurate estimate, then its likely that the task should have been automated by some other means already. In other words, its easy to estimate a task that essentially amounts to copy and pasting some well known CRUD API end point patterns, but any even remotely creative or novel work is almost guaranteed to be totally unknown. Its almost always a way to trap an engineer in a bargaining conundrum. Ie, if you make me provide an estimate first and its no where near what you have in mind (because it won’t be), then you have placed me automatically in a position of begging for additional time. Its not so surprising that this subtle form of manipulation in the sales tactic of getting us to put our cards out the table first comes from the business end of the company. So as engineers, what do we have to do?

NO MORE ESTIMATES.

But estimates are our friends, it conveys expectations back to the company. The truth is that isn’t how it actually plays out, and this needs to become our new engineering battle cry. If engineers stop giving estimates for their work and simply ask for deadlines then it changes the dynamic of the conversation. If they say that they want to spend $10,000 on acquiring the boat, then we can come back and say, well I won’t be able to give you the luxury yacht you had in mind, however, I can find a used sailboat and after we fix it up you can possible charge high end clientele for exclusive sailing excursions. It puts us back in the superior position of negotiation. Then as engineers, we have the freedom to come up with a solution that meets your expectations and simultaneously doesn’t put us in an impossible situation. Or at a minimum at least veto the idea all together as not being possible to achieve in the desired time spans. Besides 99% of the time you end up getting a deadline anyway, so why even go through the excercise of making estimates?

It will only get worse

Engineering is not getting simpler, its getting more and more complex, because we are solving harder and harder problems, which means that these things (which all haven’t been commoditized yet, in spite of the fact that many founders secretly think that all engineering problems have already been reduced to the drag and drop simplicity of building Wix websites) will take longer to solve and will have a higher TCA. As a result, the gap between the expectations of founders/managers in terms of time to implementation and the amount of time we should be spending building these systems is becoming light years apart. As a result, we continue to push out half baked solutions cobbled together with duct tape and chicken wire just barely hobbling across the finish line so we can toss it off to operational teams and burden them with our rushed decision making for the rest of the life of the company or until they burn out and leave (guess which will come first).

The problem will get much worse until engineers start hitting their employers with much needed reality checks on what it takes to solve these problems. For startup founders/managers, if I cannot appeal to your sense of pity for your employees, I hope I can at least appeal to your sense of pride: Don’t be the guy asking the $10 million yacht captain if he wants to sell it for $10,000.

Kyle Prifogle

As an example lets take a very basic and contrived activity and walk through it from a services and from a product oriented approach. Remember, this is just an example, think about it in analogy to actual services and product like activities rather than taking it literally.

The log(x) excercise

Suppose the company needs to generate a data set representing log(x) with a sample size of 1001 maximum value of 3. A purely services approach would take some (poor soul) and request “Could you manually create data until we have 1000 items that follow log(x)?” For the sake of argument suppose they start to construct a spreadsheet in the simplest way by manually entering the values (and they can’t use autofill). First they might know that log(0) is 1. So they will add the values [0,1] to their spreadsheet. Then maybe they will say I know the maximum value is 3, and log(1000) = 3, so they will also add [1000,3] to the data set. Now you have a dataset that looks something like:

0,1
1000,3

For simplicity lets represent this as an array of arrays:

[
  [0,1]
  [1000,3]
]

They need to have 999 more values, and they have exactly that amount between 0 and 1000. So they start one by one adding each additional value of [x, log(x)] to their data set using a calculator, until they are done. Suppose it takes about 5 seconds for them to enter each item into their spreadsheet. This means that the whole task will take the services person about 1.4 hours to complete. Now remember, this is an analogy, of course no one would do this, but suppose that the [x, log(x)] activity is standing in place for some perhaps more complex activity that might be difficult to productize.

An early stage tech startup executive might look at this and say. This is great, this means that they can get about 6 of these done in a day, or 30 in a week. Maybe you need this done on a weekly basis 300 times. So you decide to hire 10 services people and you have met the capacity required.

Now lets look at a purely product drive approach to the same problem. The company comes to a software engineer and says, I need to be able to generate spreadsheets with data representing [x, log(x)]. The engineer might come up with a black box solution like:

def discrete_log(max, values)
# code
  end

which takes a max and the desired number of values and produces [x, log(x)]. In this case they can use the function to produce the dataset:

discrete_log(3, 1000)
  =>
  [
    [0,1]
    [1, 0.3]
    .
    .
    .
    [1000, 3]
  ]

Again for the sake of argument, imagine that log(x) actually some much more difficult task which the engineer estimates will take about 80 hours to build, or 2 entire weeks worth of work. Suppose that this particular function doesn’t require any maintenence to utilize (not a realistic assumption). This is a simplistic example, but automatically we can see the obvious advantage of automation through a purely Produt driven process: ( Services: 400 Hours x Week | Product: 80 Hours)

This is just to state the obvious and say that automation saves time. Granted, services employees tend to be easier to find, and engineering assets much more expensive but you can see just in the simple fact that the product driven approach is a one time cost (albeit an expensive one), that it will end up being much cheaper to sustain over the lifetime of the company.

Another complexity is in the fact that the engineering assets do not scale linearly like services. Suppose that the engineering task was actually estimated to take 400 hours up front. You can’t, like in the services example, simply hire 10 engineers and expect the task to get done in 1 week. Usually one engineer is working on a specific task, or if the task can be broken apart, it doesn’t scale so simply. Worst case scenario it might take the engineer 10 weeks, and you will be stuck waiting those 10 weeks.

Now lets take each of these and submit them to some pressures you might expect in a tech start up environment.

Pressure 1: Changing Requirements

Suppose that the company realizes that instead of doing log(x) it wanted to produce spreadsheets of ln(x) (the natural log base e).

Services: The services approach is unaffected, now when they go to their calculator instead of type log(x), they type ln(x). All is well, the estimates are the same and the number of head count required is the same.

Product: The product approach is not marginally better than the services approach in this case. In this case the product engineer has to create a new black box function:

def discrete_ln(max, values)
# code
  end

and then apply this to the scenario. This glosses over the fact that discrete_ln and discrete_log might have some shared functionality.

#psuedocode
class DiscreteData

def log(x)
# some hard function
  end

def ln(x)
# some other hard function
  end

  def print
  [x, ln(x)]
  end

  end

So if the implementation has been invested in sufficiently then its possible the Product approach is not totally susceptible to this pressure. However, once the new function is implemented continued hours required to sustain the solution are 0.

Pressure 2: Changing Parameters

Suppose that we wanted to return 2 log(x) instead of just log(x). We can think of 2 as being a parameters of the problem since it is the coefficient on the function.

Services: The services approach is still unaffected, now when they go to their calculator instead of type log(x), they type 2 log(x). All is well, the estimates are the same and the number of head count required is the same. You are left with a total number of hours roughly the same as before.

Product: The product is practically unaffected. The existing solution can be extended by doing something like:

discrete_log(3, 1000).map{|x| [x[0], 2 * x[1]] }

Because in this case the changing parameter equates to changing the coefficient in front of the function we get this added benefit practically for free.

Pressure 3: Changing Scope

Say for example instead of just returning x and log(x) we also wanted to return log(2x), something like: [x, log(x), log(2x)]. Suppose that we wanted this to be the data backing a 2D graph before and now its the data backing a 3D graph. Now we have increased the scope of the problem.

Services: The services approach is increased slightly but is still unaffected. Perhaps instead of taking 5 seconds for each row in the spreadsheet now it takes 8 seconds. This can still be easily measured and to meet the needed capacity we can simply scale up the amount of services employees that we need.

Product: Like with Pressure 1, the product approach now may or may not be faster to implement in the beginning depending upon how general the solution was originally implemented. For example if the engineer anticipated requiring more dimensions he might have implemented the solution in a more general way:

#pseudocode
class DiscreteData

def log(value)
# some hard function
  end

def print()
  [ x, log, log(2x)]
  end

  end

Thus the internals of the function are abstracted out and the print function handles the heavy lifting of being able to configure the operation to expand to the needed scope.

Now that we have looked at 3 different types of pressure that can be exerted to both pure approaches lets step back and look at the advantages and disadvantages of each:

Pure Services Approach:

Pure Product Approach:

The curious thing here is that I listed “Capacity Scales Linearly” as both an advantage and a disadvantage for a Pure Services Approach. How can this be?

In the above graph I show how linear scaling capacity increases with increasing assets (in this case additional services headcount), and three possible graphs for how Product scales with increasing assets (both tech assets as well as infrastructure and engineering assets). You can see that linear scaling is useful in the early stages of a tech startup, it says that you get out of your asset allocation roughly what you put into it, but most important about this, is that you can measure how much you get out of it. That certainty is worth its price in gold to an early stage tech startup. This explains why it is so appealing: it is better to have a Known Good, than an Unknown Excellence. However, as the demands on a tech company grow, so do the demands on capacity. This is especially true for the web, where network effects can take place and you might be suddenly faced with a capacity requirement unlike you imagined. Faced with the prospect of producing an operation a million times in a week that would require a services head count of 100,000 people, suddenly the need for scalability becomes apparent, and if this capacity is not present that equates to lost opportunity for the company.

Product capacity scaling however is not linear. Its hard to gauge exactly what it is, but when things are going well it can in the higher limits far outpace the linear scaling of services. As with services approach it is still subject to diminishing returns, however, due to capabilities of vertical scaling within infrastructure the gap could be very large.

Pure Services to Product Evolution: A Handicap?

So its obvious that services have problems scaling, but they provide a lot of benefit upfront, therefore does it make sense to think of services -> product as an evolution of a tech company as it starts to reach maturity? This is certainly the prevailing conventional wisdom on the topic. However, this is where you see start differences in companies in Silicon Valley (heavily funded companies) vs everywhere else (modest initial funding). A heavily funded tech startup can skip the initial stages of having a more services oriented product by operating in the red for a considerable amount of time while the build out the product and then, if the planning has been on the mark, you begin with considerable capacity to scale. This doesn’t have to be a black and white equation either. Maybe the company can take on a modest amount of revenue through having a striped down product (MVP), that way they are showing momentum and still in the red but not depedent upon a heavy services approach for their operating cash flow. This is the lean start up model.

However, companies that start with much smaller initial investments might have to be more strategic about exchanging a certain amount of services to generate operating cash while they are simultaneously building out the product. However, in some ways taking on services is like taking on VC investment. In the same way that VC investors may have different priorities than the company, services may start to make special demands on the product. The product cannot serve two masters and in the end the product and the entire company starts to mold itself around a services oriented approach. This is why it is so important that companies utilizing services take every opportunity to productize their services whenever they get a little bit of a bubble in their operating expenses: The longer you use services, the harder it is to get rid of them.

Kicking the Services Habit

Why is it so hard to move away from services once your company has started to differentiate in that direction? Think about the people problem, once you have hired your services team to a certain size, services are going to be hesitant and resistant to automating any of their tasks (even if it means that it frees them up to work on more interesting things). Think of the logistical problem, the processes that services use have differentiated in possibly thousands of different directions, so it becomes very hard to generalize the snowflake like patterns that they have adopted they have done into a single coherent product.

But perhaps the biggest reason services are hard to drop is psychological. Why change things when they are working fine? Where is the need for something that is going to cost a lot up front and is difficult to measure? (this is not actually true, more on how to measure product features in a later blog post). After all, in some sense services feel good to a company. It feels productive, when a new spreadsheet is generated. But its important to stop and ask, is that needed productivity?

So, why implement a product when services can produce the results I want now? Let me answer that question by asking another question. Why exercise? Its just going to hurt and Im going to feel worn out afterwards. Because when you exercise you increase your bodies ability to cope with daily function. In other words, you have increased your capacity, much in the same way a product oriented mindset increases a tech companies capacity to scale.

Services in many ways are like an addictive substance. They provide immediate value quickly, they are apparent measurable value and capacity scales linearly. However, the longer you use them, the harder it is to get off of them for your organization, and the more likely your company is to fall into a pattern of stagnation.

Services = Short term gain?

Is that to say that all services are simply chasing after short term gain? Not necessarily. They could be a very deliberate strategic decision. For example, one Pure Services approach that seems to work for companies is a “Truncated” Pure Services approach. The idea behind this approach is for your company to embrace services and more or less scrape by with limited capacity until it reaches the finish line. Usually that means some type of acquisition event. In this strategy the services approach is important as a kind of “planned obsolescence” of the company itself. In other words, it doesn’t make sense for the company to invest in very long term gains because the company more or less has a target and only needs to grow to that level. Whether or not that level can be achieved with linear growth in capacity depends on the company, maybe companies will have to exchange a greater amount of Product in order to reach that goal, and where that ratio lands, whether its 50:50, 70:30, 90:10 really depends on the company.

However, even this is quite a gamble. As a parent company investing in another company, it seems likely that you would like to probe into the technology of a company and know whether or not it is truly scalable, or merely appears to scale but is in reality propped up by an army of services.

Pure Product Approach: A Possibility?

Revisiting the above code examples, you can see that even an attempt to implement a “Pure” product approach in fact produces a result that is still on a spectrum between Services and Product.

The more general you make your solution the more potential pressure the solution can accommodate making it more similar to services. In this way you can really think of Product development as the process of encapsulate potential services patterns within a repeatable automated framework, hence Software as a Service.

So how do you define the scope of your product? One tactic could be “code only to the use cases that you know of. “ That is, define the user stories you wish to cover and cover just those. However, another approach that is more in the middle of the spectrum might be something like “code to the use cases that you know of, and can estimate are likely to arise.” This is much more art than it is science. Its the true mark of a skilled engineer. But there is also something to be said about how the microservices approach fits into a possible hybrid services / product type architecture.

Suppose you already have a robust services architecture and would like to an adopt more of a pure product approach. If you define the needed scope to capture your current services in terms of a single monolithic product then you might be hesitant to start down that path because the scope may seem too ambitious. However, if you envision it as a series of microservices, some which can be productized easily and others which might temporarily remain as a service type arrangement then there is a greater chance you can start to break the problem apart and actually reach the goal productizing your services architecture incrementally, one small bite at a time.

Kyle Prifogle

1. Do you automatically build after every approved PR, i.e. Continuous Deployment?
2. Do you utilize best practices for source management?
3. Do you log errors and monitor performance?
4. Do you enforce best practices by linting your code and performing static analysis?
5. Have you clearly defined testing responsibilities?
6. Do you write tests before fixing bugs?
7. Can your developers run migrations without worry?
8. Is there at least 1 database for each separate service?
9. Do you utilize feature flags, versioning or other techniques to facilitate deployment?
10. Do you have a consensus driven process for product development and prioritization?
11. Do you monitor and track the effectiveness of features?

Now I will breakdown each item and explain them in more detail:

1. Continuous Deployment

Item 3 on the joel test asks “Do you make daily builds?” The first item on my list expands this by asking if you make builds whenever a pull request is approved. In other words, “continuous deployment”. Many companies struggle with continuous deployment, citing difficulties in stability and executing deployments this frequently, however, this usually is a smell for other artifacts of bad design within a companies infrastructure (see number 7 and 8).

The reason continuous deployment is such a key component to developer happiness harkens back to the lean startup model itself. Frequent small iterative changes can be more easily assessed. I cannot tell you how many times i’ve worked with other developers with incredible talent and ambition, but that lacked the discipline to see their features in an incremental way and be able to break them apart to smaller pieces.

This not only makes it easier to review and deploy, but makes it easier to pinpoint where production problems may be arising. I’ve spent countless hours digging through massive pull requests trying to isolate a single bottleneck that slipped through a PR; like finding a digital needle in a haystack. Its best to avoid this by working more carefully, deliberately and with better isolation and encapsulation in your design. If there is a feature that has you particularly worried on deployment, free yourself of that worry by wrapping it in a feature flag and making roll backs a non issue (see item 9).

2. Do you utilize best practices for source management?

Item one on the Joel test list asks “Do you use source control?” Its amazing to think there was a period of time where this was not a standard practice where today its almost a given. So I think it makes sense to expand the rule a bit to ask if in addition to using source control, if you are using it in the right way. What does this mean? In the first item we emphasized how continuous deployment on smaller encapsulated features leads to greater stability and ease of debugging. In order to facilitate this then it makes sense that you should do the following:

1. Keep Pull Requests as Small as possible.
2. Keep commits as small as possible, and commit messages descriptive.

This makes reviewing pull requests a much easier task for the reviewer, and reviewing a pull request is an important part of the process. I have been in jobs before where some pull requests were so massive that it got to the point that no one wanted to review them! In regards to pull request etiquette I’ve seen some really interesting approaches. For example, once I submitted a pull request and the reviewer responded by rewriting the pull request making it substantially more complex to read and even broke several of the tests! The reviewer has to realize while their feedback is valuable, the originator of the work is still the author of the pull request. Its always best to be curteous by trying to understand the original intent behind the pull request and asking questions of the author rather than try to replace their input with your own approach.

3. Do you log errors and monitor performance?

This expands the 4th item on the Joel test to include performance monitoring. These are more items that are so common place its hard to imagine a tech company without them these days. Services like New Relic have made performance monitoring much more common place to the point where every company I have worked with has used New Relic.

However, infrastructure monitoring has blossomed in recent years as well. Services like Datadog have made it possible to have a very close eye on instracture exceptions and monitor them with incredible accuracy. These oversight tools have made it to the point where many companies have even taken an extreme mode of working solely from these metrics coining the term Metrics Driven Development, even abandoning tried and true methodologies like Test Driven Development. Personally, I cannot imagine a world without tests keeping things in line, but for some companies the scale at which they work makes maintaining test suites of that size a practical impossibility.

4. Do you lint your code?

I can’t describe the relief I had on the day that I integrated RuboCop and Brakeman as blocking parts of my last companies continuous integration scripts. No longer did anyone have to constantly remind one another of little violations in best practices, we just had to update our rubocop yml and rubocop did all of the heavy lifting. Once people started noticing that the tests were failing remotely, they started running them locally before they pushed up their pull requests. There were some that thought it was over the top, but in the end it saved so much time in automating the enforcement of best practices.

In the Ruby world, Brakeman can never replace common sense and careful coding around security practices and keeping a keen eye on CVE but still is better than no automated security testing and thus provides a considerable amount of value as well. If you like this approach, other tools like Pig and Bullet can even seek your code out for grevious violations in performance design.

5. Have you clearly defined testing responsibilities?

Item 10 on the joel test asks “do you have testers”. I found this a little odd the first time I read it. About 50% of the jobs i’ve experienced in the past thought of testing as solely the responsibility of the engineers. While personally, I find this is not an ideal practice, because the engineers the wrote the code are in fact the worst people to test it (because they are predisposed to think about it in the way they wrote it, thus missing major mistakes), the fact remains that many companies still make engineers do the majority of the testing as a way of keeping them accountable for their own work. Therefore, I decided to make this item less specific and simply ask if testing responsibilities are clearly defined and understood in the company.

This also includes the company establishing best practices about automated tests. Does the company use request, controller, and session specs and not BDD such as capybara or selenium? This is fine as long as the company comes to a consensus together about how they should approach testing and whether or not they will bring in additional head count to deal with functional and usability testing.

6. Do you write tests before fixing bugs?

The Joel test asks in item 5 if you fix bugs before writing new code. While this is important, I think it falls more under an implicit activity that arises from logging the errors and monitoring them. However, one practice seem ubiquitous in debugging bugs and that is writing tests. So many drawn out conversations I’ve had in the past could have been avoided by one small request “Could you send me a failing test to describe the issue you are describing?” I came to the point where I almost found it easier to communicate at times through tests rather than through English. The effectiveness of the approach is emphasized by the fact that Rails makes it a required practice for submitting issues. While getting 100% test coverage is not always a sensible goal, approaching bugfixes with a test first mentality seems like a no brainer.

7. Can your developers run migrations without worry?

This one is huge. If the relational database is overloaded in throughput in such a way that certain tables are locked out from being migrated upon this creates a huge obstacle for developer productivity. While certain types of migrations such as renaming, altering tables, or reindexing are fundamentally difficult to perform and should be forgone in favor of techniques such adding new columns and deprecating old ones, the fact remains that if migrations are a tenuous and difficult process then your developers are going to be hesitant to add them.

This ties somewhat into Continuous Deployment which becomes more difficult if migrations are an obstacle especially in a strict zero downtime environment. Features like InnoDB’s online DDL for MySQL can help with this, but nothing can replace have a very robust and well architected microservice design and multiple databases for different responsibilities. This eliminates the database as a single point of failure, and accommodates differing velocities of data

8. Is there at least 1 database for each seperate service?

In the previous point I linked to a wonderful GoTo conference talk by Fred George that describes how to approach microservice design. I would recommend watching this entire video, Fred George is a pioneer and every minute of this video is a goldmine. In it he describes how many companies make the mistake of having a single underlying database (often relational) even when they have several separate services. This single point of failure becomes an obstacle to the overall ecosystem of the company and should be avoided at all costs.

The reality is that each service should serve a distinct purpose (a telescoping of the Single Responsibility Principle of OOP), and that it may demand one or several different database technologies depending upon its function and goal. While ETL processes might be needed to keep separate services in sync, keeping a single massive underlying database as a single point of failure becomes a huge bottleneck as you begin to scale your system. For more information about the many database design and its biological inspiration watch this really awesome talk (also a GoTo conference video) by Chad Fowler.

9. Do you utilize feature flags, versioning or other techniques to facilitate deployment?

In point 3 I mentioned Metrics Driven development and how so companies utilize this in lieu of test driven development. While I don’t endorse this approach, the only way I would recommend it if it were in conjunction with some way to wrap risky features in feature flags. Deployment rollbacks simply put are time consuming, risky, difficult to pull off without event and if you can avoid them you almost always should. In companies that do not have a good microservice and database design deployment can become a challenge. I had the unfortunate experience of working in one such company in the past. When a separate dev-ops team is handling deployments, the back and forth becomes prohibitively challenging. I found over time that it was in fact easier to wrap risky features, features that I thought might cause some type of problem in feature flags which I could then activate or deactivate at my leisure.

As time went on I found that this tactic in fact worked as a solid practice in general and started doing it more often. After all, a deployment might take several minutes, but flipping a feature flag in a relational database takes milliseconds. This is not a universal solution, for example, changes that complect the object design destructively could not be safely added to a feature, however, many features that are more additive or even configuration oriented become a good fit for protection behind a feature flag. Not only this, but it also becomes a good way to approach AB testing to the point that one company LaunchDarkly built a business model around this approach. Separating some features from deployments only further increases the feasibility of continuous deployment and makes your deployments more stable.

10. Do you have a consensus driven process for product development and prioritization?

In this blog post I describe a consensus oriented product development life cycle. Why is this important for developer happiness? Because the worst thing that can happen for a developer is for them to pour their heart into a feature and for the company to shift directions and abandon what they have worked so hard on, of no fault of their own, simply because of the shifting priorities. This is extremely frustrating to developers and while it is inevitable, it should be avoided at all cost. Make sure that every stakeholder has signed off on a feature before development begins and make sure that your feature has been driven through a process of consensus building on it before work begins to avoid developer frustration about shifting priorities.

11. Do you monitor and track the effectiveness of features?

In the same blog post I reference in 10 I describe how monitoring of features is an essential part of a good product development lifecycle. Why is this important to developer happiness? Because it provides feedback and validation to their work as being effective. This is absolutely essential to provide developers with the motivation they need to keep working and building tools that they think will make a lasting impact on the company and the broader development community (in the case of open source).

Enjoy this list and let me know what you think. While not every item is required, the more you have, the more productive, happy and purpose driven your development team will become.

Kyle Prifogle

Then the company shifts direction, and what you’ve been working suddenly becomes a waste of time, in spite of your every best effort.

One unfortunate consequence of the Agile Culture is that many people assume that agile means “move first, think later.” As though the fail fast mentality should permeate every facet of how a tech business should operate, people often think that Agile gives you the freedom to begin your walk down the path in any direction, trusting that the process of iteration will eventually right your path. The problem with this line of thinking, is that iteration can sometimes fall into long term divergent behavior.

This concept is borrowed from my experience in biocomplexity. In studying population dynamics its critical where how you start. In other words, given fixed parameters different initial conditions can mean the difference between a population spiraling out of control, or dying off to extinction.

In many ways this paradigm applies to how a project evolves over time. Given the constant of how the engineering team operates, how you get a team started can mean the difference between a quickly executed, useful and even reuseable feature, and a black hole of iterations that lead nowhere and provide little value, like the spirals in the above Hopf bifurcation diagram.

After experiencing the sting of a dead project a few times, I eventually stumbled upon what was the most elucidative blog post I had seen in years:

Stuart Rimmel: System Traps

If you get a chance read both parts of this blog post you should, its worth every minute. Suddenly I realized that so many of those system traps I had encountered in my career. But it was hard to avoid many of those traps as an engineer speaking towards product development. There were too many specific instances to keep in mind and too much that would need to change. Eventually I started to put together a picture of a product development system educated by many of those system engineering traps that would provide a context for working with maximal collaboration between the different functions within a tech company. The process I came up with was as follows:

The diagram itself is actually fairly intuitive, and not too far from several other paradigms, including Agile itself. The key differentiator is that it is aimed upon developing a concept upon its consensus within the organization. Too often escalation results in organizations skipping some of these steps in order to deliver something that they view as being business critical. However, by forcing requirements through such a system it allows organizations to know and measure the velocity of their pipeline upon the level of consensus they have achieved within the organization. Here is a description of the above process in plain words:

1. Representatives from each team collaborate in brainstorming sessions to determine a list of “Pre-specs” or the seed of features to be developed within the software. Included in this brainstorming is assessments by engineering of technical feasibility and how particular features build upon one another in the context of their architecture and platform (a blog post on this to come later).

2. If it is unknown even by engineering whether or not a particular feature is technically feasible a Spike might need to be performed by engineering.

3. The next steps go in order of “difficulty or amount of time needed to modify”. Next the idea moves on to Product to be developed into a prototype or proof of concept. This varies depending on the organization but often equates to a high level description of the feature and the user stories it maps too, as well as some basic wireframes and technical descriptions.

4. Next the Prototype is passed off to design (which is more costly to change than the prototype). Design tools that can be easily be shared and collaborated with product and engineering are helpful tools. For this reason I have recently been using Adobe XD because of its presentation view which product can use to visualize the user experience workflow and assess how well it fits the defined user stories.

5. Next the design is passed off to engineering to be turned into a spec. This is the deep dive by engineering to think about technical feasibility of the individual components laid out by product and design. If any issues are encountered then the design is passed back to product to be re-evaluated and then to design to be tweaked. This cycle may continue several times before the Spec is finalized.

6. Depending upon the software product some products may have a heavy or light services component to using the software. For Product oriented SaaS companies this step may be very light or non existent. For more service oriented products this might be crucial, and will give Services a chance to assess how much personnel will be needed to support services engagements laid out in the spec. At this point Services gets a chance to see what services engagements will be needed in the proposed Specification, and while they are assessing also make their internal documentation to perform services tasks.

7. After this checks out they pass it on to client services who does a sanity check on the match between the user needs being escalated by client success and the scope of the Specification. These may iterate a couple more times.

8. The reason that Design/Product are seperated from Services/Client Success is because the first group represents a “planning” phase of the product whereas the later represents an “execution” phase. This separation is important because it presents the execution teams with something to respond to and iterate upon.

9. Once the Spec has been vetted by all the stakeholders in it, the engineering team drives the feature to execution. Separate from automated testing (such as rspec, capybara, selenium), manual Useability and Functional testing are performed by User Experience Design and Client Success, though many organization chose to include functional testing, or at least smoke testing as a responsibility of the engineering team. It often makes more sense for those teams to perform those tests since they will need to be familiar with the issues that may arise, though some organizations think its better for the engineering team to have greater accountability on their work.

10. After those tests are performed follow ups in the form of hotfixes happen.

11. The most important part of the process is remembering that a feature is NOT finished once it ships but rather needs to continue to be monitored and assessed in the form of analytics to determine effectiveness of changes, AB testing/multivariate testing, and support both from Client Success and Services.

In a later blog post I will talk about how this process can be executed rapidly by a small project oriented teams, which I call “cells” or “pods” and how this increases collaboration and accountability, as well as camaraderie within your organization.

Kyle Prifogle