The Monday That Disappeared
At 8:40 on a Monday morning, a senior backend engineer opens her laptop with a plan. Call her Dana. She has been at the company for four years, which makes her one of the longest-tenured people in an engineering group of thirty-one. The plan is written on an index card next to her keyboard, because she has learned that anything not written down on Monday morning ceases to exist by Monday afternoon.
The card says: tenant isolation for the reporting service.
This is not a small item. The company sells a B2B analytics product, revenue is up meaningfully year over year, and the last two enterprise deals both stalled in security review over the same question: how is one customer's reporting data isolated from another's? The answer today is "carefully, by application logic, mostly." Dana believes the answer needs to become "structurally, by design, verifiably." She estimates three weeks of focused work, of which she has maybe two clear days this week. Today is supposed to be one of them.
At 8:52 she opens Slack to mute it and sees a thread from the on-call channel. The overnight deployment to staging failed. Not production, staging. The message is from a mid-level engineer who joined seven months ago and who has appended the pipeline log and the phrase "any idea?" Dana reads the log. She recognizes the failure shape in about ninety seconds: the migration step timed out because the seeded dataset in staging has grown past the point where the migration runs inside the step's timeout window. She has seen this three times before, most recently in March. She knows the fix, which is to truncate two audit tables in staging and rerun. She also knows that nobody wrote this down anywhere durable, because the last time it happened she fixed it during an incident and moved on.
She fixes it. Elapsed: eighteen minutes, including the rerun watch. She does not count the four minutes she spends afterward re-reading her own notes on the isolation design, trying to recover the thought she had at 8:40.
At 9:35 a product manager pings her directly. A customer success manager has escalated a ticket: an enterprise account says a scheduled export "produced the wrong numbers" on Friday. Support has already asked the standard questions. Nobody can tell whether the export ran, whether it ran late, whether it ran against a partially loaded dataset, or whether the customer is misreading the report. Answering the question requires pulling the job history from one system, the ingestion timestamps from another, and the customer's timezone configuration from a third. Dana does this because she is the person who knows where all three live. Elapsed: forty minutes. The answer turns out to be that the ingestion job for that tenant finished eleven minutes after the export started, which is a real product problem worth fixing. She files a ticket. The ticket will be prioritized in two weeks.
At 10:30 she has an hour. She uses forty minutes of it on the isolation design and produces something she is reasonably happy with: a sketch of how the query layer could enforce tenant scoping at a level below application code. Then the Thursday release checklist reminder fires.
The checklist is a document. It has been a document for three years. It contains thirty-one items, of which perhaps nine require someone to think. The rest are verifications: confirm the feature flags for the release are in the expected state, confirm the migration has been reviewed, confirm the third-party integration sandbox is responding, confirm nobody has an unmerged hotfix branch, confirm the changelog draft exists, confirm the support team has been notified. Dana is the release captain this week, which rotates. Working the checklist takes her fifty-five minutes, most of which is spent in other people's tools, confirming things that the tools themselves already know.
Lunch is a sandwich at her desk while she reviews two pull requests.
At 13:20 an engineer on the platform team asks whether she can get him access to the analytics replica so he can debug a slow query. He has asked the right channel already and been told the right channel is a two-day approval. Dana has a standing permission she probably should not have. She grants him a temporary credential and makes a note to revoke it Friday. Elapsed: twelve minutes, plus a small ongoing security liability.
At 13:45 the integration test suite goes red on main. Dana looks. It is the payments sandbox again, which becomes flaky whenever the vendor deploys. She confirms this by checking the vendor's status page, then reruns the suite, then tells the channel it is fine. She does not open a ticket, because there is no ticket category for "third-party sandbox is unreliable and we have no isolation from it." Elapsed: twenty-five minutes, mostly waiting.
At 14:20 she gets ninety uninterrupted minutes, and they are good ones. This is the part of the day where the company gets what it is paying for.
At 15:50 a QA engineer asks for help. She needs test data representing an enterprise tenant with three years of history, multi-currency transactions, and a specific subscription state, because that is the configuration where last month's bug appeared. Building this by hand takes her most of a day. Dana knows a faster path involving a partially documented seeding script and a manual anonymization step. She walks through it. Elapsed: thirty-five minutes of Dana's time, and it saves the QA engineer four hours, which is why nobody has ever questioned whether this is a reasonable use of either person's time.
At 16:40 someone asks in the release channel whether tomorrow's release is on track. Answering this requires knowing the state of four things owned by three teams. Dana asks four people. Three answer within twenty minutes. The fourth answers at 18:10, after she has closed her laptop.
Before she stops, she spends fifteen minutes preparing the release notes draft by reading merged pull request titles and translating them into customer-readable language.
Add it up. Dana spent something in the range of four hours on work that required her to be Dana, specifically, only because of what she happened to know and what permissions she happened to hold. She spent roughly ninety minutes on the architectural problem that is currently blocking enterprise deals. Nothing she did was frivolous. Every single item was a legitimate thing that a competent engineer would do. Any one of them, described in isolation to a CTO, would sound like a reasonable Monday.
The company, meanwhile, has an open requisition for two more senior backend engineers. The business case for those hires is that the roadmap is slipping and the team is at capacity.
Both statements are true. The roadmap is slipping, and the team is at capacity. What is unresolved is whether the capacity is being consumed by engineering.
That is the question this piece is about, and it is worth stating plainly before anything else: are you short of engineers, or are you wasting engineering?
Most organizations never ask it, because the answer requires looking at a category of work that is almost invisible in the systems executives use to see their own company.
The Work Nobody Puts on the Roadmap
Every software organization has two kinds of work. The first kind is on a plan somewhere. It has a ticket, an epic, an owner, a quarter, and a slide. When leadership talks about what engineering is doing, this is what they mean. The second kind is the work required to make the first kind possible, and it lives in places that do not aggregate: Slack threads, meeting side channels, ad hoc calls, support escalations, recurring calendar reminders, checklists in shared documents, dashboards someone checks every morning out of habit, spreadsheets that reconcile two systems that should reconcile themselves, and the accumulated memory of a handful of long-tenured people.
The second kind of work does not appear in the roadmap because it is not a project. It is a condition. It is the ambient cost of operating what you have already built, at the scale you have already reached, with the tooling you happened to have when you got there.
Executives underestimate this work for a structural reason rather than a careless one. Nearly every instrument leadership uses to observe engineering measures deliberate output. Sprint boards show planned items. Roadmaps show intended outcomes. Delivery dashboards show throughput of tracked work. Even the DORA metrics, which are among the better-validated measures in the field, describe the performance of the delivery system rather than the composition of an engineer's week. None of these instruments has a column for "forty minutes reconstructing what happened to a customer's export job because no single system could answer the question."
This is also why developer productivity surveys often disappoint the executives who commission them. They produce sentiment. Sentiment is real and worth knowing, but it does not tell a CFO what to fund. The gap between "our engineers report frustration with tooling" and "here is the specific recurring workflow consuming eleven engineer-hours per week, and here is what it would cost to remove it" is the entire distance between a complaint and a decision.
There is a useful vocabulary for part of this already, and it comes from operations rather than product engineering. Google's Site Reliability Engineering practice defines toil as operational work that is manual, repetitive, automatable, tactical, devoid of enduring value, and that grows at least linearly with the size of the service. The SRE organization caps aggregate operational work at fifty percent of an SRE's time and treats a sustained breach of that cap as a management problem requiring intervention rather than an individual performance issue (Google SRE Book, "Eliminating Toil"). The framing is worth borrowing even if the number is not directly transferable to your organization, because it does something most companies never do: it treats the volume of repetitive operational work as a governed quantity with an owner and a threshold, rather than as weather.
The concept needs extending, though. Toil as originally defined is production-adjacent. What consumed Dana's Monday was broader. Some of it was production operations. Much of it was release coordination, environment maintenance, test data preparation, access brokering, cross-tool status reconciliation, and support translation. These are not reliability tasks. They are the connective tissue of a growing software organization, and they follow the same economics: manual, repetitive, automatable in principle, and expanding with scale.
Engineering overhead is not the enemy
It would be easy, and wrong, to treat all of this as waste. A software organization is not a factory floor, and the goal is not to drive coordination to zero. Code review is overhead and it is worth every minute. Design discussion is overhead. Incident postmortems are overhead. Onboarding is overhead. Documentation is overhead. A release conversation in which three senior people decide not to ship on a Thursday because two risky changes landed together is one of the highest-value hours that organization will spend that week.
The useful distinction is not between engineering work and overhead. It is between judgment and mechanical execution.
Judgment is the application of expertise to a situation whose correct handling is not predetermined. It requires context, weighing of competing considerations, and accountability for a decision that could reasonably have gone another way. Mechanical execution is the performance of a predetermined sequence whose correct outcome is known in advance, where the human contribution is reliability of execution rather than quality of decision.
The same activity can be either one, depending on where you cut it. This is the part most organizations get wrong. They evaluate whole activities rather than the components inside them, conclude that the activity "requires a senior engineer," and stop thinking.
| Activity | The judgment part (keep it human) | The mechanical part (a strong candidate for automation) |
|---|---|---|
| Database migration | Deciding whether this migration is safe to run against production now, given traffic, rollback options, and what else is shipping | Running the same fourteen pre-flight checks and capturing their output for the fiftieth time |
| Production incident | Forming a hypothesis about a failure mode nobody has seen before, and deciding what risk to accept to restore service | Collecting the same six log streams, dashboards, recent deploys, and config diffs that every investigation starts with |
| Release | Deciding whether the accumulated risk in this release is acceptable this week | Verifying that flags are in the expected state, the changelog exists, and the integration sandbox is responding |
| Quality engineering | Deciding what to test, at what depth, based on where risk actually concentrates in this change | Regenerating a predictable test dataset; re-running the same regression evidence collection each cycle |
| Support escalation | Diagnosing a genuinely novel product defect from ambiguous customer symptoms | Assembling the tenant's job history, timestamps, config, and recent errors so a diagnosis is possible at all |
| Access management | Deciding whether this person should have this level of access to this data | Provisioning, expiring, logging, and revoking it once the decision is made |
Read the right-hand column again and notice what it has in common. It is almost entirely evidence gathering and predetermined sequence execution. It is the setup for judgment rather than judgment itself. And in most organizations, the setup consumes more senior engineering time than the judgment does.
Dana's Monday, sorted into these two columns, comes out roughly ninety minutes of judgment and something close to four hours of setup. The company is not paying her for the setup. It is paying her for the judgment, and it is paying a premium for the specific judgment that a person with four years of institutional context can exercise. It receives that judgment for about a fifth of her day.
How Careful Organizations Accumulate Careless Work
The instinct, on reading a day like Dana's, is to conclude that somebody made a mistake. Somebody should have automated the release checklist. Somebody should have built self-service test data. Somebody should have fixed the access approval process. The engineering leadership must be weak, or the team must be undisciplined.
That reading is almost always wrong, and it is worth spending time on why, because organizations that misdiagnose this problem as a discipline problem tend to respond with exhortation, which does nothing.
Manual operational work accumulates through a sequence of individually correct decisions. Here is the mechanism, in the order it usually happens.
A process begins its life as an obviously reasonable manual step. When a company has five engineers and one environment, walking through a release checklist is faster than automating one, by a wide margin. Writing a script to seed test data would take two days; seeding it by hand takes twenty minutes and happens twice a month. The manual approach is not laziness. It is correct capital allocation for a company at that size. Anyone who automated it at that stage would have been over-engineering, and would have been told so in code review.
Then the company grows, and every input to that original calculation changes at once, without anyone recalculating. Release frequency goes from twice a month to twice a week. Environments go from one to five, because sales wants a demo environment, compliance wants an isolated environment, and the mobile team needs their own. Customers go from twelve to four hundred, which means support volume goes up and the tail of unusual configurations gets longer. Integrations go from two to eleven. A SOC 2 audit arrives and adds evidence requirements to processes that previously had none. The engineering team triples, which means the ratio of people who know why things work the way they do to people who do not gets much worse.
The twenty-minute task is now a forty-minute task that happens eight times as often, performed by people who did not design it, in an environment with more ways to go wrong. Nobody revisits it, because there is no moment at which anyone is asked to. Nothing in the operating rhythm of a growing software company includes a scheduled re-examination of decisions that were correct two years ago.
The forces that keep it in place
Once a manual process is established, several forces conspire to keep it there, and they are worth naming individually because they call for different responses.
The pain is distributed, and therefore invisible. If one person spent eleven hours a week on release coordination, that person would escalate loudly. Instead, eleven hours are spread across nine people in increments of twenty to forty minutes. No individual's experience is bad enough to trigger action. Aggregation is precisely the thing organizations are bad at doing informally.
Every individual occurrence looks cheap. Twenty minutes is not worth a meeting. It is not worth a ticket. It is certainly not worth a two-week automation project. The comparison people make is between twenty minutes and two weeks, rather than between two weeks and the annualized total of that twenty minutes across everyone who performs it. This is the single most common analytical error in this whole domain, and Act VII is devoted to correcting it.
Nobody owns the workflow end to end. This is the structural one. Consider what it takes to provision a QA environment with realistic data: infrastructure provisions the compute, the platform team owns the deployment tooling, the data team owns the anonymization rules, security owns the access policy, and the QA team owns the actual need. Every one of those five groups owns a step. None of them owns the outcome. Work that crosses organizational boundaries has no natural home, and improvement work with no natural home does not get done, no matter how motivated the individuals are.
Automation competes with the roadmap, and loses on visibility. An engineering manager choosing between shipping a customer-visible feature this quarter and building internal release automation is making a choice between an improvement someone will notice and an improvement that will be invisible if it works perfectly. The incentive gradient runs one way. It is not that leaders don't understand the value of the second option. It is that the first option is legible to the board and the second one is not.
Seniority becomes the safest workaround. When a process is fragile, the risk-minimizing move is to route it through the person most likely to get it right. That person is your best engineer. This is locally rational and globally corrosive, and it deserves its own treatment, which it gets in the next section.
Tribal knowledge inflates apparent complexity. After a process has been performed manually for two years by the same three people, it accumulates undocumented conditionals. Do this, except on Fridays. Skip step four unless the mobile release is in flight. When someone finally proposes automating it, the estimate comes back large, because the estimate has to account for all the exceptions. Much of the perceived complexity is not inherent. It is the residue of never having been forced to specify the process precisely. Writing it down usually reveals that a third of the exceptions are obsolete.
Nobody measures interruption cost. Organizations that do measure task time measure execution time. The eighteen minutes Dana spent on the staging deployment is what a time-tracking exercise would capture. It would not capture the cost of having been pulled out of an architectural problem and needing to rebuild that context. Research on interrupted programming tasks found that only about ten percent of programming sessions resumed coding activity within a minute of the session starting, and that resumption typically involved navigating around the codebase first to rebuild context (Parnin & Rugaber, ICPC 2009 / Software Quality Journal 2011). Earlier work in the same tradition found that tasks resumed after an interruption could take roughly twice as long as uninterrupted equivalents. These studies are old enough that the tooling has changed considerably, and neither justifies a precise multiplier. What they support is a directional claim that most cost models ignore entirely: the interruption is more expensive than the interrupting task.
Processes evolve faster than their automation. A team automates the release process. Six months later the architecture changes, a new service is added, and the automation covers eighty percent of the release. The remaining twenty percent is handled manually "for now." Within a year, the manual portion has grown back and the automation is a partial-coverage system that still requires a human operator. This is not a failure of the original automation. It is a failure to treat automation as something that requires ongoing maintenance the way any other production system does.
Automation degrades, and people quietly stop trusting it. This is the most under-discussed force of the lot. When an automated check produces false failures often enough, engineers stop treating its output as information. Google's testing team reported in 2016 that around 1.5 percent of their test runs produced flaky results, that roughly 16 percent of their tests exhibited some flakiness, and that about 84 percent of observed transitions from passing to failing involved a flaky test rather than a real regression (Google Testing Blog, 2016). Those figures come from one organization at extreme scale a decade ago, and should not be treated as a general rate. The behavioral consequence, though, generalizes well: once a signal is unreliable, humans route around it. They rerun. They eyeball the output manually. They add a "quick manual check" after the automated one. The manual work returns, now stacked on top of the maintenance cost of the automation that was supposed to eliminate it.
Steps get automated; workflows do not. A team writes a script that provisions an environment. Another team writes a script that loads data. A third writes a deployment job. Each step is automated. The workflow still requires a human to run three things in order, check the output of each, and handle the two failure modes in between. From the engineer's perspective, almost nothing has changed. From a management dashboard, the process is "automated." This gap between step automation and workflow automation is where a great deal of automation investment quietly fails to produce the promised relief.
The compounding tax of "just this once"
There is a specific pattern that deserves a name, because it is how a surprising amount of permanent operational load enters an organization.
Something urgent happens. A customer needs a data correction, a partner needs a report in a format nobody supports, a deployment needs to happen outside the normal window. An engineer handles it manually, correctly, in an hour. It was the right call. Building a proper mechanism would have taken a week and the customer needed it Thursday.
The next month, it happens again. The engineer who did it the first time does it again, faster, because now she knows how. The month after that, someone new asks, and is directed to her. By the sixth occurrence, it is no longer an exception. It is an undocumented service the company offers, with a single human implementation, no SLA, no error handling, no audit trail, and no owner other than the person who happened to say yes the first time.
The tax compounds in three directions simultaneously. It compounds in volume, because the number of customers and situations that trigger it grows with the business. It compounds in fragility, because the manual procedure accumulates exceptions faster than anyone documents them. And it compounds in concentration, because the more times one person performs it, the more efficient it is to keep routing it to her and the more expensive it becomes to move it anywhere else.
The decision that created this was correct. The decision that perpetuated it was never made at all.
The Expensive Engineer Doing Inexpensive Work
Now the economics, which is where most discussions of this topic go badly wrong in one of two directions. Either they reduce everything to a salary calculation, which is too crude to be useful, or they retreat into vague language about "unlocking potential," which is too soft to fund.
Start with the crude version, because it is at least a floor. In the United States, the Bureau of Labor Statistics reported a median annual wage of $133,080 for software developers as of May 2024, and $102,610 for software quality assurance analysts and testers (BLS Occupational Outlook Handbook). Those are national medians across all industries and all employers, including sectors that pay well below what a venture-funded SaaS company pays a senior engineer in a major market. Total compensation at product companies, including equity, runs materially higher. For internal modeling, most companies should use their own loaded cost, meaning salary plus benefits, taxes, equipment, software licenses, and a share of management and recruiting overhead. Depending on the market and the level, a fully loaded hourly figure for a senior engineer commonly lands somewhere between $75 and $200. Use your own number rather than any number in an article, including this one.
Run the crude calculation anyway, because it is instructive even when directionally imprecise. Suppose a release checklist consumes ninety minutes of one senior engineer's time per release, releases happen weekly, and a second person spends thirty minutes chasing status. At a loaded cost of $120 per hour, that is about $12,500 per year for one checklist. Most organizations have between six and twenty processes of comparable weight. The arithmetic gets uncomfortable quickly, and it is still the least interesting part of the argument.
The scarce resource is not hours
The reason the salary calculation understates the problem is that hours are not what is actually scarce in a software organization. What is scarce is high-quality engineering attention: sustained, uninterrupted, context-rich thinking applied by someone who understands the system deeply.
This resource has properties that make it unlike a generic hour of labor. It is non-fungible, in that two engineers with the same title cannot always substitute for one another on the same problem. It is fragile, in that it degrades sharply under interruption and takes real time to rebuild. It is concentrated, in that a small number of people in any organization can do the hardest work. And it does not accumulate, in that an hour of attention not spent today cannot be banked for next week.
When Dana spends three hours on repetitive operational work, the company does not lose three hours of salary. It loses three hours of the most constrained input it has, and the loss is measured in what those hours would otherwise have produced. In her case: a tenant isolation design that is currently a gating factor on enterprise deals. That is not an abstract opportunity cost. It has a revenue shape.
The general list of what gets displaced is worth stating explicitly, because it is what a founder is actually buying when they fund senior engineering: architectural work that prevents future rework, performance improvements that change unit economics, security work that removes deal friction, reliability work that reduces churn, mentoring that raises the ceiling of the whole team, technical risk reduction that prevents a bad quarter, and the unstructured experimentation that occasionally produces something nobody scoped.
None of these appear on a sprint board as clearly as the operational work does. All of them compound. That asymmetry is the core of the economic argument: repetitive operational work has a linear cost and displaces work with compounding returns.
The seniority paradox
Here is the part that surprises leaders when they first look carefully at where operational work concentrates. It does not distribute evenly. It flows uphill, toward the most capable people, and it does so for reasons that are entirely rational at every individual decision point.
Senior engineers attract operational work because they know the system, so they resolve ambiguity faster. They know the history, so they can explain why something works the way it does. They are trusted, so routing a sensitive task to them minimizes risk. They are fast, so the marginal cost of asking them looks lower than the marginal cost of asking anyone else. And they usually have the broadest permissions, because permissions accumulate with tenure, which means they are often the only person who can do a given thing without a two-day approval.
Every one of those is a good reason. Together they produce a perverse result: the more valuable an engineer becomes to an organization, the more efficiently that organization can waste them. Capability functions as a magnet for exactly the category of work that does not require capability.
The effect is self-reinforcing over time. Each time a task is routed to the senior person, two things happen. The organization's ability to handle it any other way atrophies slightly, and the senior person's knowledge advantage grows slightly. Eighteen months of this produces a situation where the routing is no longer a preference but a fact: she genuinely is the only person who can do it quickly, and the organization genuinely cannot afford for it to be done slowly.
The hero engineer trap
Most companies have a version of this person. She is the one who saves the release, who diagnoses the production issue at eleven at night, who knows why the payment reconciliation job has that strange conditional, who can get a stuck deployment moving. She is visibly, dramatically productive. She is often the highest-rated engineer in performance reviews and she deserves to be.
She is also, frequently, the most reliable indicator that the organization's operating system is under-built.
Consider what generates the opportunities for heroism. Someone must save a release, which implies releases are saveable-or-not rather than routine. Someone must diagnose the outage quickly, which implies diagnosis is hard, which implies the systems do not explain themselves. Someone must remember the strange conditional, which implies the conditional is not encoded anywhere a machine or a newcomer could find it. Every act of heroism is evidence of a gap, and the heroism obscures the gap by resolving its symptoms before they become visible enough to fund.
This is a delicate argument to make inside a company, and it is easy to make it badly. Nothing here suggests the hero is a problem, or that expertise is somehow suspect. Deep expertise is the most valuable thing a software organization accumulates. The claim is narrower and it is about where expertise should live: expertise that has been applied to the same situation more than a handful of times should be captured in a system, so that the expert can move on to the situations that have not been seen before.
The practical test is diagnostic rather than accusatory. Take your most valuable engineer and ask, over the last quarter, which of their rescues were novel and which were the fifth instance of a familiar pattern. Novel rescues are exactly what you want senior people doing. Repeat rescues are unclaimed automation work wearing a cape. The ratio tells you something important, and it is a ratio very few engineering organizations have ever calculated.
There is a second risk that founders tend to feel before CTOs do. An organization dependent on a small number of people for operational continuity has a key-person exposure that is invisible on any balance sheet. It shows up when that person takes three weeks of leave, or has a bad quarter, or leaves. The cost of that exposure is not the recruiting fee. It is the six to twelve months during which nobody else can do things the organization has come to assume are always possible.
Human Middleware
Software architects have a precise word for the layer that sits between systems and makes them able to work together: middleware. It moves data between components, translates formats, coordinates sequences, and handles the fact that System A and System B were not designed with each other in mind. Middleware is unglamorous, essential, and, in a well-built system, entirely automatic.
A useful way to see what has happened inside many software organizations is to notice how much of that layer is staffed by people.
A person is functioning as human middleware when their recurring contribution is to move information between systems, execute a predetermined sequence, reconcile two sources of truth, or bridge a gap between tools that should communicate directly. The defining characteristics are that the work is triggered by the state of a system rather than by a judgment call, that its correct output is knowable in advance, and that the human adds reliability rather than insight.
The pattern is easier to recognize in examples than in definitions.
| What it looks like | What the person is actually doing | What the system should be doing |
|---|---|---|
| Updating a status in a project tool after a deployment completes | Transcribing an event from one system into another | Emitting the event to both |
| Asking four people whether their part of the release is ready | Polling distributed state manually | Publishing readiness as observable state |
| Comparing configuration between staging and production by eye | Running a diff, badly, from memory | Continuously reporting drift |
| Collecting logs from three services to answer a support question | Performing a join across data stores using a human as the query engine | Exposing a per-tenant diagnostic view |
| Copying and anonymizing production-like data into a test environment | Executing an ETL pipeline by hand | Providing on-demand, compliant seeded environments |
| Checking each morning whether the nightly jobs completed | Being a monitoring system with a coffee habit | Alerting on failure, staying silent on success |
| Preparing the same weekly delivery report from three tools | Aggregating data that already exists | Generating the view directly from source systems |
| Relaying deployment status into a customer-facing channel | Being an API between engineering and the business | Publishing status where stakeholders can read it |
| Manually triggering an internal job when a condition is met | Being a scheduler with an inconsistent uptime record | Triggering on the condition |
Look at the middle column. Almost every entry describes a function that has a well-understood software implementation. The organization has not decided to solve these problems with people rather than software. It has never framed them as problems at all, because each one arrived as somebody's small responsibility rather than as an architectural gap.
Why this is worse than its cost
The direct cost of human middleware is the salary time it consumes, and that is the smallest part of the problem. The more serious consequences are structural.
Human middleware produces inconsistency. Two people performing the same reconciliation will do it slightly differently, and the same person will do it differently under time pressure. Where the work feeds a compliance or financial process, this inconsistency is a real exposure. It also produces poor auditability, because a manual step generally leaves no reliable record of what was checked, by whom, against what state.
It creates knowledge concentration, which is the same key-person risk described earlier, arriving through a different door. It slows onboarding, because a new engineer cannot become useful by reading the system; they have to be told things by people, and the people are busy. Onboarding time is one of the more sensitive indicators here, and one of the few that leaders can observe without instrumentation: if it takes a competent senior hire four months to become independently productive, a large share of the necessary knowledge is not in the system.
It fragments attention, because human middleware is almost always interrupt-driven by nature. The work arrives when a system reaches a state, not when the person is free. And it does not scale across time, which is a constraint that becomes painful the moment a company has customers or engineers in more than one timezone. A process that depends on a specific person being awake has a hard ceiling that no amount of hiring in the same office will raise.
There is a human cost too, and it is worth stating without melodrama. Engineers who spend a large fraction of their week on work that does not use their training tend to disengage, and disengagement precedes attrition. The developer experience research literature is consistent on this point: friction in daily workflows affects satisfaction, retention, and perceived productivity, not merely output (Greiler, Storey & Noda, IEEE Transactions on Software Engineering, 2022; Noda, Storey, Forsgren & Greiler, ACM Queue, 2023). The people most likely to leave over this are the ones with the most options, which is to say the ones you least want to lose.
The balancing argument
None of this justifies an unconditional preference for automation, and this is a good place to introduce the objection rather than saving it for later, because it applies to everything that follows.
Automated middleware fails too, and its failures have a different and sometimes worse shape. A human performing a reconciliation notices when something is obviously wrong; an automated one propagates the error at machine speed and does not notice anything. A human sees that the customer's name is in the price field; a pipeline writes it to the warehouse. Automated processes fail silently far more often than manual ones do, and they fail in ways that are harder to detect precisely because nobody is watching.
There is also a genuine argument that some human involvement in operational processes is valuable in itself. Engineers who never see how their software is deployed, or how it fails in production, or what customers say when it breaks, build worse software. On-call rotations exist partly for this reason. A team that has abstracted itself entirely away from operational reality tends to produce systems that are unpleasant to operate.
So the goal is not zero human involvement in operations. The goal is that human involvement should be chosen rather than inherited, and that when a human is in the loop it should be because their judgment is doing something, not because nobody ever built the connection between two systems.
Not Everything Should Be Automated
Any argument this far along runs a risk of becoming propaganda. It is worth stopping to make the case against itself seriously, because the failure modes on the automation side are real and expensive, and because an executive who reads only the first half of this argument will make bad decisions with it.
There are several categories where continuing to do the work manually is the correct engineering and business decision.
Genuinely rare work. A process performed twice a year, taking two hours, costs roughly four hours annually. Automating it might cost forty hours, plus ongoing maintenance, plus the risk that it silently breaks between uses and has to be debugged under pressure the one time it is needed. Rare automation also decays, because nobody exercises it. The payback period on rare processes is frequently longer than the lifetime of the process itself.
Work that is changing rapidly. Automating a workflow means committing to its current shape. If the process is being actively redesigned, or the underlying product area is in flux, automation built today will be rework tomorrow. There is a legitimate argument for waiting until a process has stabilized. The trap is that "it changes too often" becomes a permanent excuse for a process that has actually been stable for two years, which is why this claim should be tested against evidence rather than accepted as received wisdom. Look at how the process has actually changed over the last four quarters, not at how volatile it feels.
Work that is mostly judgment. If eighty percent of a task is deciding and twenty percent is executing, automating the twenty percent may add coordination overhead without meaningful relief. Automate the information gathering that supports the decision instead, and leave the decision alone.
High-consequence actions where a human checkpoint has independent value. Deleting customer data, moving money, executing a production database change with no clean rollback, granting elevated access to production systems. The value of the human is not that they perform the action better than a machine would. It is that a deliberate confirmation step forces a moment of attention proportional to the consequence. Automate the preparation, the verification, and the evidence collection around the action; keep a human on the trigger. Many mature organizations do exactly this and call it a deployment gate or a break-glass procedure.
Temporary work. Processes attached to a migration, a specific customer commitment, or an experiment will end. Automating a process with a known expiry date rarely pays back.
Work that is not standardized enough to automate. If five people perform the same task five different ways, and nobody can say which way is correct, automation will fail. Not because the tooling is inadequate, but because there is no specification. The prerequisite work here is standardization, not scripting.
Work that should be deleted. This is the most common and most overlooked case.
Ask first whether the process should exist
Before asking how to automate something, ask whether it should be happening at all. A remarkable share of recurring operational work exists because of a condition that no longer holds. A report is produced weekly because an executive asked for it two years ago and nobody has checked whether anyone reads it. A verification step exists because of an incident in 2022 that was subsequently fixed at the root. An approval exists because of an organizational structure that has since been reorganized away. A manual sync exists between two systems, one of which is now a system of record and one of which is a legacy view nobody trusts.
Automating a process that should not exist is the most expensive possible outcome. It converts a temporary waste into a permanent, maintained, institutionalized waste, and it makes the waste harder to remove because now there is a system with an owner and a runbook defending it.
This suggests a sequence, and it is the most useful framework in this piece. Work it in order, and do not skip forward.
| Stage | The question | What it looks like in practice | Common failure |
|---|---|---|---|
| Eliminate | Does this need to happen at all? | Turn the process off for a month and see who complains. Trace each step back to the reason it was introduced and check whether that reason still holds. | Assuming everything currently done is necessary because someone once decided it was |
| Simplify | If it must happen, can it happen with fewer steps, fewer people, and fewer handoffs? | Remove approvals that never reject anything. Collapse three tools into one. Cut verification steps whose failure mode is already covered elsewhere. | Automating the complexity instead of removing it |
| Standardize | Can it be made to happen the same way every time? | Write the actual procedure down, in full, including the exceptions. Reconcile the differences between how different people do it. Decide what "correct" means. | Skipping straight to automation and encoding one person's idiosyncratic version |
| Automate | Which deterministic parts can a machine do reliably? | Build the sequence. Keep humans on the decisions and the high-consequence triggers. Make it self-service where possible. | Automating the whole workflow including the parts that need judgment |
| Observe | How will we know it is still working? | Instrument the automation itself. Alert when it fails, when it is bypassed, and when its usage patterns change. Assign an owner. | Building automation with no monitoring, then discovering months later that everyone stopped using it |
Most organizations that talk about automation start at stage four. The value in stages one through three is usually larger, always cheaper, and almost never captured, because those stages require organizational decisions rather than engineering work, and organizational decisions are harder to assign.
Stage five deserves particular emphasis, and Act IX returns to it. Automation without observation is how a company ends up believing a process is handled while people quietly work around a system that stopped functioning in March.
Finding Your Automation Debt
Technical debt is a familiar concept to every engineering leader, and to most founders who have worked with one. It is understood, roughly, as the accumulated cost of implementation shortcuts that make future change more expensive.
Automation debt is a useful sibling concept: the accumulated cost of repeatable manual work that persists because the organization postponed, fragmented, or under-invested in removing it. It is not a standardized industry metric, and it should not be presented internally as though it were. It is a management lens, and its value is that it turns a diffuse condition into something that can be inventoried, prioritized, and funded.
Like technical debt, it accrues interest. Each new customer, environment, integration, compliance requirement, and team member increases the volume of manual work generated by the same unautomated process. Unlike technical debt, it is largely invisible in the codebase, which is why engineering leaders who are diligent about the first are often blind to the second.
Don't run another survey. Look at the work.
The instinct when leadership becomes concerned about productivity is to measure it. This usually produces a survey, and surveys of developer sentiment have real value; the SPACE framework makes a well-supported case that productivity is multidimensional and cannot be captured by any single metric, and that both perceptual and system data are needed (Forsgren, Storey, Maddila, Zimmermann, Houck & Butler, ACM Queue, 2021).
But a sentiment survey answers "how does it feel" when the question you need answered is "what specifically is consuming the week, how often, and what would it cost to remove." For that, you need an inventory of actual work.
The most direct method is a two-week engineering friction inventory. It is cheap, it requires no tooling purchase, and in most organizations it produces more actionable information than a quarter of dashboard work.
The mechanics matter, so a few notes on how to run it without poisoning the data. Announce clearly that this is an inventory of work, not an evaluation of people, and mean it; the moment engineers suspect the log will be used to assess individual output, the data becomes worthless. Keep the logging burden under two minutes per entry. Ask people to log anything that was repetitive, interrupt-driven, or performed on behalf of someone else, plus anything where they waited on another person to perform a small action. Include managers, QA, support engineers, and DevOps, not only developers. Two weeks is chosen deliberately: long enough to catch weekly and biweekly cycles, short enough that participation does not decay.
Here is a friction log structure that produces enough signal to prioritize against. It can live in a spreadsheet.
| Column | Why it matters |
|---|---|
| Task | The specific recurring thing, described concretely enough to recognize |
| Category | Release, environment, test data, access, support, reporting, incident, deployment, configuration, integration, monitoring, compliance, documentation sync |
| Who performs it | Reveals concentration; a task performed only by one person is a different problem than one shared by six |
| Frequency | Per day, week, month, or release |
| Execution time | Hands-on minutes |
| Waiting time | Elapsed time spent blocked on someone else, often larger than execution time |
| People involved | Every additional person multiplies coordination cost and adds a handoff |
| Trigger | What causes it to happen: a schedule, a system state, a person asking, a failure |
| Interrupt or planned | Interrupt-driven work carries a context-switching cost that planned work does not |
| Skill level required | Would a junior engineer, a non-engineer, or a machine be able to do this correctly? |
| Standardization | Does everyone do it the same way? Is it written down anywhere? |
| Judgment required | Honestly: is a decision being made, or is a known sequence being executed? |
| Failure mode | What happens when it goes wrong, how often does it go wrong, and what does the rework cost? |
| Current tooling | What exists already, and why is it not sufficient? |
The column most organizations omit, and the one that most often changes the conclusion, is waiting time. Consider an access request that takes an administrator four minutes of actual work and blocks an engineer for two days. Measured by execution time it is a trivial task not worth automating. Measured by its effect on flow, it is one of the highest-value things in the inventory. DORA's research on internal platforms found that the capability most associated with a positive developer experience was giving clear feedback on the outcome of tasks, which points at the same underlying issue: the cost is in the uncertainty and the wait, not in the work (DORA, Platform Engineering capability).
Frequency alone will mislead you
The naive way to read a friction inventory is to sort by total annual hours and start at the top. That is better than nothing and it is not sufficient, because it treats all hours as equivalent.
A five-minute task performed forty times a week by eight people is worth serious attention: roughly 1,400 hours a year, and every one of those occurrences is probably an interruption. A two-hour task performed twice a year is four hours and almost certainly should be left alone, unless it is a task that only one person can perform and it happens during quarter-end close, in which case its risk profile matters more than its duration.
Six dimensions beyond raw hours deserve to be scored:
Interrupt cost, because work that arrives unpredictably damages more than its duration. Waiting cost, because blocked time is often the largest hidden component. Handoff cost, because each transfer between people adds delay, ambiguity, and a chance of misunderstanding. Failure cost, meaning the frequency and expense of the task being done wrong, including the rework and the incidents it causes. Context-switching cost, which is related to interrupt cost but distinct: pulling a senior engineer out of deep architectural work is more expensive than pulling them out of email. And knowledge concentration, because a task that only one person can perform is an organizational risk whose cost is not measured in hours at all.
The Automation Payback Question
At some point the inventory has to meet a budget. What follows is a model for that conversation. It is deliberately directional rather than precise, and presenting it as precise would be a mistake, because most of its inputs are estimates and several of its most important terms are not quantifiable at all.
The purpose of the model is not to produce a number that survives audit. It is to force a structured comparison that most organizations currently make on instinct, and to make the annualized view visible next to the per-occurrence view.
Define the variables:
- F = frequency (occurrences per year)
- T = hands-on execution time per occurrence, in hours
- W = waiting or blocked time per occurrence, in hours, weighted by how much of it is genuinely lost
- P = number of people involved per occurrence
- C = loaded hourly cost of the people involved (use different values for different roles)
- I = interruption and context-switching premium, applied only to interrupt-driven work
- E = expected annual cost of errors and rework attributable to performing this manually
- A = estimated one-time cost to build the automation
- M = estimated annual cost to maintain, monitor, and support it
The annual manual cost of a process is approximately:
Annual manual cost ≈ F × (T + W) × P × C × (1 + I) + E
And the payback horizon is approximately:
Payback (years) ≈ A ÷ (Annual manual cost avoided − M)
Three cautions about using this.
First, the term you will be most tempted to inflate is I, the interruption premium. There is no defensible universal multiplier. The research on task resumption establishes that recovering context after an interruption is real and non-trivial, but it does not license a claim that every interruption costs twenty-three minutes, a figure that circulates widely in business writing with a much weaker evidentiary chain than its popularity suggests. A sensible approach is to use a modest premium for interrupt-driven work, apply it consistently, and be transparent that it is a judgment parameter rather than a measurement. If a proposal only clears the bar because of an aggressive interruption multiplier, it does not clear the bar.
Second, avoidable manual cost is not the same as total manual cost. Automation rarely removes one hundred percent of a process. If a release checklist has thirty-one items and nine require judgment, the realistic target is the twenty-two, minus whatever residual human verification remains. Model the residual explicitly.
Third, M is the term most commonly set to zero and most commonly wrong. Internal automation requires maintenance when APIs change, when the workflow changes, when it breaks, and when someone new needs to understand it. A reasonable planning heuristic used by many platform teams is to budget ongoing maintenance at a meaningful fraction of the original build cost per year. Set it from your own history if you have any; set it conservatively if you do not.
What the equation cannot hold
Several of the strongest reasons to automate resist quantification, and a purely numerical process will systematically underweight them. These belong in the decision explicitly, as named qualitative factors rather than as fudge factors inside the arithmetic.
Reliability and consistency, meaning the process produces the same result regardless of who runs it or how tired they are. Auditability, meaning there is a record, which matters enormously the first time a customer, an auditor, or a regulator asks what happened. Reduced key-person dependency, meaning the process survives a resignation or a vacation. Onboarding speed, meaning a new hire becomes productive by using a system rather than by accumulating oral history. Cycle time, meaning the process stops being a scheduling constraint on releases. Scalability across time zones, meaning the process does not require a specific person to be awake. Developer experience, meaning your best engineers spend their days on work that resembles the job they were hired to do.
A decision matrix worth arguing over
The two dimensions that matter most for a first-pass triage are how often the work happens and how much judgment it genuinely requires. Adding a third dimension, workflow stability, separates the good candidates from the traps.
| Low judgment | High judgment | |
|---|---|---|
| High frequency, stable workflow | The strongest candidates in your inventory. Environment provisioning, deployment execution, test data seeding, status reporting, routine access grants, diagnostic collection. Automate the workflow end to end and make it self-service. | Automate the inputs to the decision, never the decision. Assemble the evidence, present it well, and let a human decide. Release go/no-go, incident severity calls, risk-based test scoping. |
| High frequency, unstable workflow | Standardize before automating. If five people do it five ways, scripting one of those ways will produce an automation that two people trust and three route around. | The hardest quadrant. Usually a sign that the underlying system or organizational design needs attention more than the workflow does. Reduce the frequency rather than automating the response. |
| Low frequency, stable workflow | Weak candidates on economics alone. Reconsider only if the task is high-consequence, only one person can perform it, or it happens at moments of peak organizational stress such as quarter close or an audit. | Leave it alone. Document it well enough that a second person could do it, and move on. |
That top-right cell is the one leaders most often get wrong, in both directions. Some organizations refuse to touch anything involving judgment and therefore leave enormous value on the table, because the preparation for a judgment call is usually mechanical. Others attempt to automate the judgment itself, encode a policy that was never quite right, and spend the next two years handling exceptions.
The most valuable form of automation for high-judgment work is almost always the same: collect and present the evidence, fast and completely, and let the expensive human do the expensive thing.
Where to Look First
An inventory produces a long list. Experience suggests that in most growing software companies the significant items cluster in a small number of operational zones. What follows is a tour of those zones, framed around the judgment-versus-execution split rather than around specific tools, because the right tool depends on your stack and the right question does not.
Releases
Release processes are where operational waste is densest in most organizations, because releases touch every team and therefore have no single owner.
The mechanical portion typically includes: verifying preconditions across services, confirming flag and configuration state, assembling release notes from merged work, chasing approvals from people who will approve, executing the deployment sequence, running post-deploy smoke verification, and confirming rollback readiness. All of this is state that systems already possess and that humans are currently querying by hand and by conversation.
The judgment portion is small and genuinely important: given what is in this release, what has already shipped this week, what is happening in the business, and where the risk concentrates, should this go out now? That decision benefits from a human who understands context. It does not benefit from that human having spent the previous ninety minutes assembling a status picture.
A practical test for release friction: how long between "the code is ready" and "the code is in production," and how much of that gap is machine time versus human coordination time? If the answer is mostly the latter, you have found a large item. DORA's work on deployment automation and continuous delivery is the best-validated body of evidence on this class of improvement (DORA capabilities).
Note also that the checklist-as-document pattern deserves specific attention. A checklist in a wiki is a specification that has not been compiled. It is the clearest possible signal that a process has been fully described and then implemented in human beings.
Test data
This is one of the most under-appreciated consumers of engineering and QA time in companies with any data complexity, and it is worth explaining in terms a non-technical founder can act on.
To test a system meaningfully, you need data that resembles reality: an account with three years of history, a customer in an unusual subscription state, a dataset large enough that performance problems appear. Production has this data. You usually cannot use production data directly, for good reasons involving privacy law and customer trust. So someone has to construct realistic data that is not real, or take real data and remove everything sensitive from it, correctly, every time.
When this is not solved systematically, three things happen. Engineers and QA staff spend hours per cycle building data by hand. Tests run against unrepresentative data, so entire classes of defect are found in production instead. And engineers copy production data into lower environments informally to get their work done, which is a compliance incident waiting to be discovered.
DORA treats test data management as a distinct capability precisely because it gates so much else, and its guidance emphasizes making adequate test data readily available to automated tests and reducing reliance on data defined outside test scope (DORA, Test data management). For a founder, the translation is simple: if your engineers regularly spend hours creating the conditions under which they can begin testing, you are paying senior salaries for data entry, and you are probably also carrying a privacy risk nobody has written down.
Development and test environments
Environments are the second-order version of the same problem. Consider the questions that consume time here: how long does it take to get a working environment, who can create one, what happens when configuration drifts between staging and production, who resolves dependency conflicts, and what happens to environments nobody has cleaned up.
The most expensive pattern in this zone is the shared environment with contention. When six teams share a staging environment, a meaningful amount of engineering time goes into coordination about who is using it, into diagnosing failures caused by someone else's changes, and into waiting. That waiting is invisible in every system of record you have.
The second most expensive is configuration drift, which produces a specific and recurring form of waste: bugs that only appear in one environment, investigated repeatedly, by senior people, because environment-specific failures are hard to reason about. Automated drift detection converts a recurring investigation into a report.
Support escalations
For a business-focused reader, this is often the most consequential zone, because it sits at the boundary between engineering cost and customer experience.
Engineering becomes an extension of support whenever answering a customer question requires access or knowledge that support does not have. The escalation path then routes through the people with the deepest system knowledge, which is to say your most expensive engineers, and it routes to them unpredictably.
The important insight here is about what to automate. Companies tend to think about this problem as "how do we reduce escalations," which quickly becomes a debate about triage rules and support training. The higher-leverage move is usually to automate the information gathering rather than the problem solving. If a support engineer can see, for a specific customer, the recent job history, the ingestion timestamps, the configuration, the recent errors, and the account's feature flags, a large share of escalations resolve without engineering. The ones that do reach engineering arrive with the evidence already assembled, which turns a forty-minute investigation into a five-minute diagnosis.
Four capabilities do most of the work: customer-scoped diagnostic views, structured logging that supports per-tenant queries, self-service status information for customers, and a written record of previously diagnosed issue classes that support can search. None of these is exotic. All of them tend to be underfunded because they are internal-facing and therefore compete poorly against customer-facing roadmap items, despite being the thing that determines how much senior engineering time gets consumed by support each week.
Incident response
Nobody sensible proposes automating incident response. What is worth automating is everything that surrounds the thinking.
When an incident begins, a predictable sequence happens: someone assembles the state of the system, checks what changed recently, pulls the relevant dashboards and logs, determines blast radius, opens a channel, and notifies stakeholders. Almost none of that requires judgment, and all of it delays the moment when an experienced engineer can begin forming hypotheses. An automated diagnostic snapshot triggered at incident declaration, containing recent deploys, config changes, error rates by service, and the standard dashboard set, can compress that phase substantially.
The same applies to the tail end. Communication drafting, timeline reconstruction, and postmortem scaffolding are all largely mechanical. And for well-understood failure modes with known remediations, runbook automation is appropriate, with the caveat that automated remediation for a failure mode you do not fully understand is how a small incident becomes a large one.
What stays human: ambiguous incidents, novel failures, anything where the remediation itself carries risk, and every decision about what tradeoff to accept in order to restore service.
Reporting and status collection
This zone has the highest ratio of effort to value in most organizations, because the information almost always already exists in a system. When a person assembles a weekly report by opening four tools and copying numbers into a fifth, the organization is using an engineer or a manager as a query engine.
The questions worth asking about any recurring report: does anyone act on it, could it be generated directly from the source systems, and if it were simply not produced for a month, who would notice? The elimination stage of the framework earns its keep here more than anywhere else.
Access and provisioning
Access requests are the clearest example of the waiting-time problem. The work itself is minutes. The delay is days. The consequence is that engineers develop workarounds, which is how you end up with the situation in the opening narrative, where a senior engineer with standing permissions becomes an informal access broker because the official path is too slow to use.
At a business level the goal is straightforward: define roles and policies once, make the common cases self-service within policy, ensure grants expire automatically, and keep the audit trail as a byproduct rather than as a separate task. The unusual cases still deserve human review. The routine ones should not consume two days and a senior engineer's discretion.
Quality and regression work
Quality engineering deserves care here, because it is easy to collapse this section into a generic argument for test automation, which is not the point.
Split quality work the same way as everything else. Deciding what to test, where risk concentrates in a given change, what depth of coverage a feature warrants, and whether the evidence supports shipping: those are judgment, they are skilled, and they are exactly what a good quality engineer is for. Executing predictable verification, regenerating test data, collecting evidence of test results, checking that known regression paths still behave, and assembling the coverage picture for a release: those are mechanical.
The failure mode in most organizations is not that they have too little test automation. It is that skilled quality engineers spend their weeks on execution and evidence collection, and therefore have no time left for the risk analysis that would have told them what to execute. Automating execution without redirecting the recovered time into judgment produces a faster version of the same coverage decisions, which is a much smaller win than it appears.
Why Automation Projects Fail
An honest treatment of this subject has to spend real time here, because a great many organizations reading an argument like this one will respond by funding automation work that makes things worse. The objection "we tried automation and it became another maintenance problem" is not a symptom of an immature team. It is frequently an accurate report of what happened.
The failure patterns are consistent enough to list.
Automating an unstable process. The workflow changed three times during the build, and the automation now encodes a version of the process that no longer exists. Nobody has the appetite to rewrite it, so it is used partially, with manual steps filling the gaps.
Automating exceptions instead of fixing the workflow. A team notices that the deployment process fails in a particular way and builds handling for that failure, then another, then another. Two years later the automation is mostly exception handling, and the underlying workflow has never been simplified. Each individual addition was reasonable. The result is a system that is more complex than the problem.
Brittleness without observability. The automation works until an upstream API changes its response format, at which point it fails in a way that is difficult to diagnose because nobody instrumented it. Internal tooling is routinely built to a lower observability standard than production software, and then relied upon as though it were production software.
No owner. An engineer builds something genuinely useful, then moves to another team. The tool continues working for eleven months and then breaks. Nobody knows the codebase. The team reverts to the manual process, which everyone remembers, and the automation is quietly abandoned. This is the single most common ending for internal tooling.
Built for elegance rather than pain. An engineer chooses the technically interesting problem over the organizationally painful one. The result is a well-architected solution to something that was consuming ninety minutes a month, while the thing consuming eleven hours a week remains untouched because it was boring.
Platforms nobody adopts. A team builds an internal platform based on assumptions about what developers need, without treating developers as users whose behavior must be observed. Adoption stalls. The platform becomes a second path that some teams use and others avoid, which is worse than either alternative because now the organization maintains two workflows.
ROI never measured. The automation ships and nobody checks whether the manual work actually stopped. Frequently it did not; it moved. This is common enough to deserve its own name.
Bad automation creates a second manual job
The specific failure worth understanding in detail is displacement rather than elimination.
A team automates a nightly reconciliation process that previously took an engineer forty minutes a day. The script works. The manual process stops. Six weeks later, an engineer is spending thirty minutes most mornings checking whether the job ran, investigating failures that turn out to be transient, rerunning it, and maintaining a growing list of edge cases the script mishandles. Total time saved: ten minutes a day. Total complexity added: one more system to understand, maintain, and eventually migrate.
The organization believes this process is automated. The dashboard says so. In practice the work has changed shape, from a predictable forty-minute task that any of four people could perform to an unpredictable thirty-minute task that only the script's author fully understands. The second version may well be worse.
The mechanism connecting this to the flaky-test dynamic described earlier is the same: automation that produces unreliable signals generates human verification work, and human verification work is what the automation existed to remove.
Internal automation is a product, and its users are your employees
The correction for most of these failures is a single change in framing. Internal tooling is a product. It has users, it has adoption metrics, it has a reliability requirement, it has support obligations, and it requires ongoing investment. Organizations that treat it as a side project get side-project outcomes.
Concretely, this means five things. Every meaningful piece of internal automation has a named owner and appears in someone's actual responsibilities. It has monitoring, including alerts for its own failures and, ideally, a signal when people bypass it. It has documentation adequate for a second engineer to modify it. It has a feedback mechanism, because the people using it will know before you do when it stops fitting the workflow. And its usage is reviewed periodically, because an internal tool that nobody uses is worse than no tool: it is an unmaintained dependency with an implicit promise attached.
DORA's 2025 research found that internal platform adoption is now close to universal among surveyed organizations, with roughly ninety percent reporting an internal platform and about three-quarters having a dedicated platform team, and that platform quality rather than platform existence was what correlated with organizational outcomes (DORA / Google Cloud, 2025 State of AI-assisted Software Development). As with any large industry survey, the sample is self-selected and skews toward organizations engaged enough to participate. The directional finding is nevertheless the one that matters for this discussion: having built the thing is not the achievement. Having built a thing people trust and use is.
AI Changes the Price of Automation, Not the Economics
Only now, deliberately late, does this argument reach the subject that dominates most current writing about engineering productivity.
The genuine change is a change in cost. Building a small internal tool, a diagnostic helper, a data transformation, a piece of workflow glue, or a script that assembles a release status view has become substantially cheaper and faster with capable coding assistants. That matters more than it might appear, because the reason most small automations never got built was never that they were hard. It was that they sat below the threshold where anyone could justify the engineering time. Lowering the build cost moves the threshold, and a large volume of previously uneconomic automation moves above the line.
This is a real and significant shift, and it is worth planning around. It is also frequently over-read into a conclusion that does not follow.
Every question that determined whether automation was a good idea before remains exactly as binding. Is the workflow stable enough to encode? Should this process exist at all? What happens when the automation fails, and who finds out? Does the task require judgment? Can the output be verified, and by whom? Who owns this in eighteen months? What permissions does it hold, and what is the blast radius if it behaves unexpectedly?
Notice which side of the ledger the cost reduction lands on. It reduces A, the build cost, in the payback equation. It does not reduce M, the maintenance cost, and in some respects it increases the risk associated with it, because code that was cheap to produce is often produced without the surrounding investment in tests, documentation, and observability that makes it maintainable. It does nothing about permissions, security review, or operational risk. A generated integration that writes to production systems requires exactly the same scrutiny as a hand-written one.
There is a second-order effect worth watching. When the marginal cost of creating a script approaches zero, organizations produce many scripts. The failure mode described in the previous section, unowned internal tooling that works until it doesn't, becomes easier to reach at scale. The constraint shifts from can we build this to can we maintain, govern, and trust what we have built, and that constraint is organizational rather than technical.
The practical consequence for an engineering leader is a re-run of the payback calculation rather than a new strategy. Go back to the friction inventory and look specifically at the items that were rejected on cost grounds: the three-hour-per-month reconciliation, the diagnostic query someone reconstructs from memory every time, the report that takes forty minutes to assemble, the small transformation between two systems that never justified a ticket. That band, between roughly two and twenty hours of annual manual cost, is where the threshold has moved most. Items that were correctly declined eighteen months ago may now clear the bar comfortably, and the inventory is the only way to find them systematically rather than by whoever happens to feel most annoyed this week.
The corresponding discipline is that ownership requirements do not scale down with build cost. A tool that took two hours to produce still needs someone accountable for it, still needs to fail loudly, and still needs to be discoverable by the next person. The organizations that will get the most out of cheaper automation are the ones that already had an answer to "who owns this," because that constraint is now the binding one.
It is also worth being empirically careful about how much AI-assisted development is currently helping, because the evidence is more mixed than the marketing. Stack Overflow's 2025 developer survey, based on more than 49,000 self-selected respondents, reported that favorable sentiment toward AI tools had declined relative to the previous two years, that more respondents distrusted output accuracy than trusted it, and that the most-cited frustration was output that is nearly correct, which shifts effort into verification and debugging (Stack Overflow, 2025 Developer Survey). DORA's 2025 findings pointed in a compatible direction: AI amplified existing organizational capability rather than substituting for it, and individual productivity gains were frequently absorbed downstream by bottlenecks in testing and deployment.
That last observation is the one to sit with, because it is this argument restated from the other end. If your engineers write code faster but your release process still consumes ninety minutes of human coordination per release, your environments still take two days to obtain, and your test data still has to be assembled by hand, you have accelerated the cheapest part of the pipeline and left the expensive part alone.
The Organizational Problem Underneath the Technical One
Everything described so far is technically tractable. Most companies have engineers entirely capable of building the automation in question. The reason it does not happen is organizational, and it comes down to incentives and boundaries.
Look at how a typical software organization is measured. Feature teams are evaluated on shipping product. Support is evaluated on ticket resolution. Operations is evaluated on stability. Quality is evaluated on defects found and releases verified. Every one of these is a reasonable objective, and not one of them creates an incentive to remove repetitive work that spans several of these groups at once.
Recall the environment provisioning example: five teams, five steps, no owner of the outcome. That structure is the norm rather than the exception, and it is why cross-boundary friction survives in organizations full of individually excellent engineers who can all see the problem clearly.
Several ownership models address this, and they suit different company sizes.
Platform engineering creates a team whose product is the internal developer experience. This works well above roughly a hundred engineers, and it fails predictably when the team is measured on output rather than adoption.
Developer experience or engineering productivity teams are similar but usually smaller and more focused on measurement and targeted intervention than on building a platform.
DevOps or infrastructure ownership already exists in most companies. The question is whether their mandate includes removing friction from other teams' workflows or is limited to keeping infrastructure running.
Quality engineering frequently has the best cross-cutting view of where operational waste sits, because quality work touches releases, environments, data, and support. This vantage point is routinely underused; quality organizations are asked what is broken, not what is repetitive.
Rotating ownership assigns friction reduction to a different team each quarter. It is cheap, it builds shared understanding, and it works best when paired with a shared inventory so the rotation does not restart from zero each time.
Smaller companies do not need a department, and creating one prematurely is a good way to add coordination cost without adding capability. What a smaller company needs is an explicit, protected, recurring allocation of engineering capacity to friction removal, with a named owner and a visible inventory.
The percentage is a real decision and there is no universal answer. Google's fifty percent operational cap for SRE teams is a well-known reference point but it applies to a specific role in a specific operating model, not to product engineering broadly. The right allocation for your organization depends on how much friction your inventory actually revealed, how fast you are growing, and how much of your senior engineering time is currently being consumed by work that does not need seniority. What matters more than the number is that the allocation is explicit, protected from being reassigned the moment a customer commitment slips, and measured against whether the friction actually went away.
One structural suggestion that costs almost nothing: make the friction inventory a standing artifact rather than a one-time exercise, reviewed quarterly alongside the roadmap. The reason manual processes survive is that nothing in the operating rhythm ever forces a re-examination. A standing review is that forcing function.
Should You Hire Another Engineer?
Back to the question that opened this.
A CTO comes to a founder or a board with a request for four additional engineers. The roadmap is slipping, the team is at capacity, and the request is supported by everything visible: sprint velocity, missed commitments, a backlog growing faster than it is being consumed, and engineers who are visibly working hard.
There are two failure modes available here, and most organizations pick one by temperament rather than by analysis. The first is approving the headcount reflexively, which risks scaling the friction along with the capability, since each new engineer inherits the same manual processes and adds their own coordination load. The second is refusing it reflexively on the theory that the team should be more efficient, which is how you burn out a good engineering organization and lose the people you most need.
The way out is a set of questions asked before the decision rather than instead of it.
| Question | What a useful answer looks like |
|---|---|
| What is actually consuming the current team's week? | An inventory of real work, not a sentiment survey or a velocity chart |
| How much of that work requires engineering judgment? | A split between judgment and mechanical execution, done at the task-component level |
| Which work exists because we grew? | Work that scales with customers, environments, integrations, or releases, and will keep growing |
| Which work exists because tooling didn't keep up? | Processes correct at 5 engineers and now performed 40 times more often |
| Which work would disappear entirely if we questioned it? | Reports nobody reads, approvals that never reject, checks whose root cause was fixed |
| Which work could become self-service? | Anything where an engineer is currently a gatekeeper for a routine, in-policy request |
| Which work should stay human? | High-consequence decisions, novel diagnosis, genuine risk judgment |
| What would a new hire inherit on day one? | The honest version, including the undocumented parts |
This is not an argument against hiring, and it would be a poor reading of everything above to treat it as one. Plenty of organizations are genuinely under-resourced. Growing companies need more engineers, and there are problems that no amount of automation will make smaller. Sometimes the friction analysis confirms the headcount request and strengthens it, because it demonstrates that the team is already operating efficiently and the constraint is real.
What the analysis reveals is which of two things you are about to buy: additional capability, or additional capacity to absorb inefficiency. Those cost the same and produce very different returns.
The conversation worth having
There is a version of this discussion between a founder and a CTO that is more productive than the budget negotiation it usually becomes. It has three questions.
The founder asks: what happens if we don't hire? This is not a rhetorical challenge. It is a request for the specific consequences, so that the cost of not hiring can be compared against the cost of hiring rather than assumed to be catastrophic.
The CTO asks: what is consuming the current team? This is the question that requires the inventory, and it is the CTO's responsibility to be able to answer it with something better than an impression.
And then, together, they ask the question that changes the shape of the conversation: which of this work should the next engineer never have to inherit?
That question does something the others don't. It converts an argument about cost into a design decision about what kind of organization is being built. Every recurring manual process you have today will be taught to every person you hire, and each of them will spend a portion of their career performing it. Hiring does not dilute operational waste. It replicates it.
A Thirty-Day Friction Reduction Experiment
Frameworks are cheap. Here is a bounded experiment that produces evidence rather than opinions, sized so that a team can run it without pausing the roadmap.
Week one: observe. Run the friction inventory described earlier. Everyone in the engineering organization, including QA, DevOps, engineering managers, and support engineers, logs repetitive tasks, interruptions, work performed on behalf of others, and time spent blocked. Two minutes per entry, no evaluation of individuals, no exceptions for senior people. The senior engineers' logs will be the most revealing.
Week two: classify. Sort every entry through the eliminate/simplify/standardize/automate/leave-manual filter. Do this as a group, because disagreement is informative: when one engineer says a process requires judgment and another says it is mechanical, the discussion that follows usually surfaces the real constraint. Expect to find that a meaningful share of the list can be eliminated or simplified without writing any code. That is the cheapest value available and it is routinely skipped.
Week three: choose two or three targets. The selection criteria matter more than the execution. Choose high frequency, stable workflow, low judgment, and clear ownership. Deliberately do not choose the most technically interesting item; the correlation between "interesting to engineers" and "expensive to the organization" is weak, and the temptation is strong. Prefer targets where the same person can own the workflow end to end, because cross-boundary work is where good intentions go to expire. If a candidate requires four teams to agree before anything can ship, it is the right target eventually and the wrong target for a thirty-day experiment.
Week four: build, measure, and record the baseline honestly. Before-and-after measurement is the entire point, and it needs the baseline captured before anything changes. Measure execution time, frequency, waiting time, number of people involved, failure and rework rate, interruption count, and adoption. Adoption is the one most often forgotten and the most predictive: if the automation exists and people still do it manually, you have learned something important about whether you solved the actual problem.
Then, and this is the part almost nobody does: review it again after thirty to sixty days. The questions are uncomfortable and worth asking directly. Did the manual work stop, or did it change shape into monitoring and exception handling? Is anyone maintaining this? Has it broken, and did anyone notice? Are people bypassing it? What did the recovered time actually get used for, and can you point to it?
That last question is the one that turns this from an efficiency exercise into a strategic one. Recovered engineering time does not automatically become architectural work. Without a deliberate decision, it fills with more operational work, because operational work expands into available capacity. Deciding in advance where the recovered hours go is as important as recovering them.
Organizations that want an outside view of this sometimes bring in help specifically because internal teams have difficulty seeing processes they perform every day; this is the shape of work QAtronic does under the heading of engineering workflow assessment, and it is deliberately diagnostic before it is prescriptive. The method matters more than who runs it. What matters is that somebody looks at the actual work with fresh eyes and no attachment to how it came to be that way.
Give Engineers Their Engineering Back
Return to Dana's Monday, and imagine it eighteen months later in a company that took some of this seriously. Be careful with the imagining, because the honest version is not dramatic.
The staging deployment still fails occasionally, but the failure now produces a message identifying the known cause and a one-click remediation that the seven-month engineer runs himself, without messaging anyone. The support escalation still arrives, but it arrives with the tenant's job history, ingestion timestamps, and configuration already attached, because support has a diagnostic view; Dana spends eight minutes confirming a diagnosis instead of forty minutes assembling the evidence for one. The release checklist has twenty-two of its thirty-one items executed and reported by the pipeline, and the release captain spends fifteen minutes on the nine that require a person to think, which is the part that was always worth her time. The QA engineer generates the enterprise-tenant dataset herself, from a self-service tool, in four minutes. The platform engineer requests analytics replica access and receives a time-bounded credential in ninety seconds, within policy, logged.
Some things are unchanged. There is still an incident that week, and it is genuinely novel, and Dana spends most of Wednesday on it, which is exactly what she is for. The payments sandbox is still unreliable, though now the pipeline distinguishes vendor unavailability from a real failure. There is still a difficult conversation about whether to ship on Thursday. Someone still has to decide. Engineering remains messy, because building software is a fundamentally uncertain activity and no amount of tooling makes it otherwise.
The difference is not that the work disappeared. It is that fewer highly paid, deeply contextual people spend their days as connective tissue between systems that could have been connected directly. Dana gets three days on tenant isolation instead of ninety minutes, and the enterprise deals stop stalling in security review, and nobody writes a case study about it because the improvement is invisible by design.
The reason to pursue any of this is not efficiency in the abstract, and certainly not a smaller headcount. Automation, done well, is a decision about what your organization asks human beings to spend their attention on. Every hour a skilled engineer spends executing a sequence whose outcome was known in advance is an hour not spent on the problems where the outcome is genuinely uncertain, which are the only problems where their expertise makes a difference. Those problems are also, not coincidentally, the ones your competitors find hard.
So before the next requisition is approved, it is worth knowing precisely what the current team is doing with its weeks. Not how it feels. What it is. The answer usually reshapes the question, and occasionally it justifies the hire outright, which is a good outcome too. What it should never do is leave a new engineer walking into their first Monday and quietly inheriting a decade of accumulated chores that nobody ever got around to deleting.
Engineering Workflow Audit
Before adding headcount, it is worth understanding what your current engineers actually spend their week on.
QAtronic runs an Engineering Workflow Audit to identify repetitive engineering work, manual quality processes, release friction, environment and test data bottlenecks, and the automation opportunities with the clearest payback. The output is an inventory of your actual operational load, a judgment-versus-execution split across it, and a prioritized view of what to eliminate, what to simplify, and what to automate first.
If you are weighing a hiring decision this quarter, that inventory is a useful thing to have before the decision rather than after it. [Talk to us about an Engineering Workflow Audit.]