Error rate on a checkout service climbs from 0.2% to 6.1% over four minutes. Latency at the 99th percentile roughly triples. The increase begins about ninety seconds after a new build finishes rolling to the last cell of the service fleet. Nothing else changed in the same window that the change-tracking system knows about.

An AI incident agent reads the alert, queries the metrics backend, pulls the deployment record, samples error logs, checks the dependency graph for downstream degradation, finds none, and produces a conclusion:

The error rate increase correlates with release checkout-svc@4.19.2. Errors are concentrated in the code path touched by that release. Recommended mitigation: roll traffic back to checkout-svc@4.19.1.

Assume the diagnosis is correct. Assume the mitigation is the right one. Now consider three ways an engineering organization might have wired this system.

Model A. The agent investigates and posts its conclusion into the incident channel with links to the evidence. A human on-call engineer reads it, agrees, opens the deployment tool, and executes the rollback. The agent has read access to production and write access to nothing.

Model B. The agent investigates, then constructs the exact rollback operation: target service, target version, target scope, ordering across cells, expected duration. It presents that operation as a staged action with an approval control. A human reviews the plan and approves. The agent executes it through the deployment system and reports the result.

Model C. The agent investigates, checks a set of machine-verifiable conditions (the release is inside the rollback window, the previous version is marked healthy, no schema migration has been applied since, no other change is in flight against this service, incident severity is above the configured threshold), and executes the rollback itself. A human is notified but is not asked.

Same telemetry. Same diagnosis. Same mitigation. Same model, if you like. The three systems are separated by something that has nothing to do with the model's reasoning quality: how much authority the system holds over production state, and what has to be true before that authority can be exercised.

That is the actual engineering question, and it is not answered by "yes, agents should fix incidents" or "no, humans must approve everything." Both answers describe a policy for all actions at once, which is precisely the design error. The useful question is narrower and harder:

Under what conditions should an agent be permitted to change production state, and what controls must exist before that permission is granted?

The tension driving the whole problem is simple to state. A production incident creates pressure to act quickly, and every minute of delay has a measurable cost in failed requests, burned error budget, and violated commitments. But the same mechanism that shortens time to mitigation also shortens the time between a wrong conclusion and a wrong action applied at machine speed and machine scale. Google's SRE organization states the risk plainly in its published material on AI in operations: an agent taking a faulty action in production can cause immediate, widespread disruption, and the speed and scale at which agents operate mean a failure's blast radius can be much larger and propagate much faster than a human operator's mistake.

This article is about designing the space between those two facts. It is organized as a gradient: the same agent, granted progressively more authority over production, with the same questions asked at each step. What new failure becomes possible? What additional evidence should be required? What permission must be added? What validation becomes mandatory? What is the maximum damage if the action is wrong? Can it be reversed? Who remains accountable? And what should happen when the agent is not sure?

This is a system design problem, not a contest

Before going further, it is worth disposing of a framing that makes this conversation worse.

The question is not whether AI agents are better or worse than site reliability engineers. Humans make serious mistakes during incidents, and the SRE literature is candid about this. Google's own material on managing incidents opens with a scenario built around an engineer making unilateral, uncoordinated changes during an outage and making the situation considerably worse. Incident command structures exist precisely because unconstrained human action during a high-pressure event is dangerous. The entire discipline of change management exists because humans, given root and a hypothesis, will sometimes do damage.

Automation prevents mistakes as well as causing them. A deterministic rollback pipeline that refuses to promote a build failing its health checks is a control that removes a class of human error. A policy engine that rejects a DELETE against a production table outside an approved change window is protecting the system from every caller, agents included.

So the framing is not human versus machine. It is: which decisions can safely be delegated, under which constraints, and what happens when a delegated decision turns out to be wrong. Those are questions about permission systems, validation, containment, and reversibility. They belong to the same family of problems as designing a deployment pipeline, and they should be approached with the same seriousness.

Production authority: a working definition

Use a precise term for the thing being granted.

Production authority is the ability of an agent to perform an operation that changes the state, availability, configuration, traffic distribution, data, permissions, or behavior of a live production system.

Under that definition, all of the following are exercises of production authority:

• rolling back a deployment
• restarting a service instance or a fleet
• scaling replicas up or down
• toggling a feature flag
• shifting traffic between versions, cells, zones, or regions
• invalidating or flushing a cache
• draining or cordoning a node
• editing a runtime configuration value
• quarantining or isolating a resource
• rotating or revoking a credential
• changing routing or DNS
• disabling an integration or a downstream call
• modifying an IAM policy or role binding
• repairing, migrating, or deleting data

And none of the following are:

• querying a metrics backend
• searching logs
• reading traces
• fetching a deployment record
• reading a runbook
• reading a service dependency graph
• reading incident history

The distinction between observation authority and mutation authority is the first and most important boundary in the design. It is worth enforcing at the level of credentials and tooling, not at the level of intent. An agent that holds a token capable of mutation is a mutating agent, whether or not its instructions say it should only read.

Two clarifications keep this definition useful in practice.

First, read operations are not automatically free. A query that scans a petabyte of logs during an incident can add load to the very observability system the responders depend on. Reading a customer record is a privacy event even if it changes nothing. Observation authority is cheaper than mutation authority by orders of magnitude, but "read-only" is not the same as "no consequences," and rate limits and data-access controls still apply.

Second, the boundary is about effect, not about API verb. Some operations that look like reads have side effects: triggering a diagnostic dump, forcing a health check, opening a support case, paging a team, posting to a customer-facing status page. A status page update is a production mutation in every sense that matters to the business. Classify by effect.

Capability is not authorization

There is a persistent conflation in discussions of agentic operations: the assumption that a more capable model deserves more permissions.

It does not follow, and treating it as though it does is one of the more dangerous simplifications available. These are two separate questions answered by two separate systems:

• Capability answers: can this system reason about the incident well enough to reach a useful conclusion? That is a property of the model, the context available to it, the tools it can call, and the quality of the evidence it can retrieve.
• Authorization answers: what is this system allowed to do? That is a property of identity, policy, credentials, and the tool surface, and it is set by humans in advance.

Keeping them separate has concrete consequences. It means a model upgrade does not silently widen the action space. It means the agent cannot argue its way into a permission it does not have, because permission is not something it is asked about. It means the answer to "why was the agent allowed to do that?" is a policy record rather than a transcript of reasoning.

The inverse error also matters. Low capability does not justify granting no authority at all, if the actions in question are narrow, reversible, well-tested, and bounded. A simple, deterministic autoscaler holds real production authority and has for years. What justifies authority is not intelligence; it is evidence about a specific action's risk profile and the existence of controls that contain it.

Google SRE's published guardrails make this separation architectural rather than advisory. Agents are required to have identities distinct from human users, strongly authenticated, with access granted on demand rather than standing; the stated concern is that an agent operating with a developer's ambient credentials creates a path from a single errant instruction to global impact. The permission is attached to the agent principal and the action, not inferred from the agent's confidence in its own analysis.

Design rule: never let the agent be the component that decides what the agent may do. The moment authorization is expressed in the same channel as reasoning — a prompt saying "only take safe actions," a tool description saying "use with care" — it has stopped being a control and become a suggestion.

The production authority gradient

What follows is an explanatory framework for this article, not an industry standard and not a maturity model anyone is obliged to climb. Its purpose is to make one point unavoidable: autonomy is not a switch. It is a sequence of distinct capabilities, each of which introduces failure modes the previous one did not have.

Two things are worth noting about the shape of this table.

The first five rungs change no production state. That is a large amount of useful capability sitting entirely on the safe side of the mutation boundary, and a great deal of the measured value of AI in incident response has been realized there. Google reports that surfacing an AI-generated incident hypothesis to on-call engineers — pure information, no actuation — produced roughly a 10% reduction in mean time to mitigate in its internal A/B testing, and that its AI-assisted investigation dashboards delivered on the order of a 44% reduction in mean time to mitigate for supported incidents. Those are Google's numbers for Google's environment and should not be treated as a benchmark for anyone else. The structural point is what matters: substantial mitigation-time improvement was available before any agent was permitted to touch production state.

The second point is that the rungs are not evenly spaced. The step from Prepare to Request approval is small. The step from Request approval to Execute bounded action is the largest one in the table, because it is the point at which a probabilistic system becomes capable of changing a live system without a human in the causal path. Everything after that step is about controlling a capability that already exists.

Public implementations converge on roughly this shape. Google's SRE material describes five autonomy levels defined by which of monitoring, investigation, mitigation decision, actuation, and self-directed multi-step resolution are automated: manual, assisted, partial (agent can actuate but a human must approve the plan), high (agent decides and acts without approval for well-bounded scenarios), and full (agent manages a multi-step resolution loop end to end, including trying alternative mitigations and initiating rollbacks when an action has adverse effects). Microsoft's Azure SRE Agent documentation exposes a coarser but operationally similar control: a review mode in which the agent proposes an action and waits for approval before executing Azure infrastructure write operations, and an autonomous mode in which it executes and reports afterward. Crucially, that mode is configured per response plan and per scheduled task rather than globally for the agent, and Microsoft's own recommendation table pairs production incidents and security alerts with review mode while pairing staging and dev incidents, health checks, and reporting tasks with autonomous mode.

Nobody serious is publishing "turn it on and let it run."

Autonomy is a property of actions, not of agents

The single most consequential design decision in this space is where the autonomy setting lives.

The wrong place is on the agent:

# Do not do this
incident_agent:
  autonomous: true

That configuration says: whatever this agent decides to do, in whatever context, at whatever scope, it may do. It collapses a rich risk surface into one boolean, and it means that any expansion of the agent's tool set silently expands its authority. It also means that reasoning about the system's safety requires reasoning about the model's behavior across an unbounded action space, which is exactly the analysis nobody can complete.

The right place is on the action, in the context of a target and an incident state:

The ratings above are illustrative. The correct values are specific to an architecture: a "restart one instance" is trivially reversible in a stateless request-serving fleet and much less so in a stateful system with long recovery times or expensive cache warming. What generalizes is the shape of the table, not its contents. Every organization granting production authority to agents should be able to produce its own version of it, and should treat the absence of one as the real gap.

Note also what the table does not contain: a column for model confidence. That omission is deliberate, and is dealt with later.

Risk in an incident is multi-dimensional

"Risk" collapses too much. In practice, authority decisions depend on at least eight independent dimensions, and the mistake most systems make is treating one of them as a proxy for the rest.

These interact in ways that resist a single score. A highly reversible action with a global blast radius (disabling a feature flag everywhere) can be a better choice than a low-blast-radius action that is hard to undo. An action with strong evidence behind it may still be wrong to take autonomously if the state picture is degraded, because the evidence describes a system that may no longer exist in that form.

Two of these dimensions deserve extended treatment, because they carry most of the weight in real policy: reversibility and blast radius.

Reversibility is the strongest single signal

Compare two actions an agent might take. Restarting one stateless worker: the worker leaves the load balancer, drains, restarts, passes health checks, rejoins. If the action was pointless, the cost is a few seconds of reduced capacity. Deleting a set of customer records because they appear corrupt: if the diagnosis was wrong, there is no equivalent of putting them back, only a restore process with its own risk, its own duration, and its own data-loss window.

Reversibility is not binary. It is worth separating four cases, because they justify different controls.

Fully reversible. The action has a well-defined inverse that restores the prior state, the inverse is fast, and executing it has no additional side effects. Toggling a feature flag off and back on. Removing an instance from rotation and returning it. These are the natural candidates for bounded autonomy.

Reversible with cost. An inverse exists but is slower or noisier than the original action. Scaling a fleet down and back up costs warm-up time and possibly cold caches. Draining a cell and refilling it takes minutes and shifts load onto neighbors in the meantime. These are candidates for bounded autonomy under tighter preconditions, particularly capacity checks.

Partially reversible. The system state can be restored but something else cannot. Rolling back a version restores the code but not the data written in the new format. Restarting a process restores the service but discards in-memory work. Rotating a credential restores security posture but breaks every consumer that cached the old one. These need explicit analysis of what is not restored, and that analysis should happen before an incident, not during one.

Irreversible. Deleted data with no verified backup. A message consumed and acknowledged from a queue with no replay. A payment submitted downstream. An email sent to customers. A destroyed encryption key. These should be outside the autonomous action space in ordinary incident handling, full stop, and should require deliberate multi-party human decision even in extraordinary handling.

There is a fifth category that causes more trouble than any of the others: actions whose rollback is theoretically possible but operationally risky. The inverse exists on paper. It has never been executed under load. It depends on a code path nobody has exercised in eighteen months. The runbook for it references a tool that was decommissioned. This category is dangerous precisely because it looks like the first category in a risk table. The only way to know which one you are in is to have tested the inverse, which makes rollback testing a prerequisite for granting rollback authority rather than a nice-to-have.

Blast radius is the ceiling on being wrong

Blast radius answers a specific question: if this action executes exactly as designed but the diagnosis behind it was wrong, how much can it damage?

The distinction between "executes correctly" and "was the right thing to do" matters. Most autonomous-action disasters are not malfunctioning tools. They are correctly functioning tools applied to a mistaken conclusion, at a scope large enough that the mistake mattered.

The practical control is spatial bounding. Compare the pairs:

• Restart one pod, versus restart every pod in the deployment.
• Disable a feature for one tenant, versus disable it for all tenants.
• Shift 1% of traffic, versus shift 100%.
• Roll back one service, versus roll back a library that fourteen services depend on.
• Drain one cell, versus drain a zone.

In each pair, the left-hand action is a bounded experiment and the right-hand action is a commitment. Left-hand actions can be delegated with far less evidence, because the cost of a wrong left-hand action is small and observable. Right-hand actions require both stronger evidence and, usually, a human decision, because a wrong right-hand action creates a second incident on top of the first.

AWS's published deployment practice makes the same argument for change generally: the stated primary objective for production deployments is to avoid simultaneous negative impact across multiple Availability Zones or Regions, and the mechanism is limiting the scope of each individual deployment. Mitigations are changes. The same logic applies with more urgency, because mitigations are executed under time pressure with a partial picture.

A useful design constraint follows: the maximum blast radius of an autonomous action should be a property of the tool, not of the agent's request. If the restart tool accepts a selector that could match an entire fleet, then the fleet is inside the autonomous blast radius no matter what the policy document says. If the tool accepts at most one instance identifier per call and enforces a concurrency limit server-side, the ceiling is structural.

Walking up the gradient

The rest of this article walks the gradient rung by rung. At each step, the same questions: what breaks, what evidence is required, what permission is added, what validation becomes mandatory, what the ceiling on damage is, whether the action can be reversed, who is accountable, and what happens under uncertainty.

Rung 1: Observe

The agent reads. Metrics, logs, traces, deployment records, configuration, topology, dependency graphs, incident history, runbooks, SLO and error budget status.

What becomes possible that wasn't before. Correlation across sources that no human can hold simultaneously, at a speed that matters. Google's alert-enrichment system illustrates the shape: it intercepts an alert before it reaches a human and, within a tight budget of roughly two minutes, queries monitoring, logs, change records, and dependency graphs in parallel to attach context, recent rollouts, similar past incidents, and candidate causes to the alert. It operates read-only by design, explicitly distinguished from the systems that take mitigating actions, and it emphasizes verifiable facts linked back to source data rather than conclusions.

What can go wrong. Three things, none of them state changes.

Load. An agent that fans out aggressive queries during an incident competes with the humans and dashboards trying to diagnose the same incident, against the same backends, at the same moment. Observability systems have capacity limits and they are most stressed exactly when they are most needed. Rate limits on agent reads are not bureaucracy; they are protecting the incident response itself.

Exposure. Logs contain tokens, personal data, request bodies, and internal identifiers. An agent that reads broadly and then summarizes into a chat channel, a ticket, or a model provider's context has moved that data. Scoping read access by data class, and redacting at the retrieval layer rather than trusting the summarizer, is the control.

Staleness. This one is underrated. An agent that answers from a metrics window that ended ninety seconds ago, or a topology snapshot generated nightly, or a dependency graph that predates a recent migration, is reasoning about a system that does not exist. During fast-moving incidents, the freshness of the state picture is itself a signal about how much authority anything downstream should have.

Permission added. Read scopes, per data source, with an identity attached. Even at this rung, the agent should have its own principal.

Who is accountable. Nobody new. The on-call engineer owns the incident. The agent is producing input.

Under uncertainty. Report the gap. An agent that cannot reach the logging backend should say so prominently, not quietly produce an analysis based on metrics alone while implying completeness. Missing telemetry is one of the most important facts in an incident, and it should be surfaced as a first-class finding rather than as an absence.

Rung 2: Investigate

The agent stops describing and starts explaining: correlating evidence, forming candidate causes, and gathering the specific data that would discriminate between them.

The failure mode that appears here is confident wrongness, and it is contagious. A well-written, well-cited, incorrect root-cause analysis posted into an incident channel does worse than fail to help. It anchors the responders. People stop generating alternative hypotheses because a plausible one is already on the table with links attached. The cost of a bad investigation is not zero just because no state changed.

Correlation is not causation, and incidents are full of coincidences

A deployment immediately preceded elevated latency. Candidate explanations, all consistent with that observation:

• The new build introduced a regression.
• The deploy shifted load in a way that pushed an already-saturated database over a threshold it was approaching regardless.
• A downstream provider began degrading in the same window.
• A cache expired on a schedule unrelated to the deploy, and the cold-cache period coincided.
• Traffic changed: a marketing event, a bot, a retry storm from a client that was already failing.
• A neighbor service on shared infrastructure started consuming capacity.
• The deploy is a red herring and the alert threshold was recently changed.

Deployment correlation is a strong prior in most environments, and it should be. But treating a strong prior as a conclusion is the mechanism by which an agent turns a coincidence into a mutation. The distinction is exactly the one Google's investigation tooling makes explicit in its capability hierarchy: detecting an anomaly, correlating a change with an alert signal, and judging whether an anomaly is worth investigating are all strictly weaker than establishing that a candidate change is genuinely the underlying cause.

Hypothesis-driven investigation

The structural fix is to require the agent to hold multiple hypotheses and to gather discriminating evidence, rather than to elaborate the first plausible one.

A worked shape for the checkout example:

Hypothesis A: regression in 4.19.2. Discriminating evidence: are errors concentrated on instances running the new build and absent on any that have not yet received it? Do error signatures match code paths changed in the diff? Did the error onset track the rollout curve across cells rather than starting everywhere at once?

Hypothesis B: database saturation. Discriminating evidence: connection pool utilization, lock waits, slow query counts, replica lag. Does the database show pressure that began before the deploy? Are other services sharing that database also degraded?

Hypothesis C: downstream dependency latency. Discriminating evidence: per-dependency latency and error rates from the client side; the dependency's own status; whether other consumers of the same dependency report the same pattern.

Hypothesis A predicts a version-correlated distribution of errors. Hypothesis B predicts errors that are not version-correlated and that appear in other consumers of the same database. Hypothesis C predicts errors dominated by a specific call site with characteristic timeouts. These predictions are testable with read-only queries, which is the point: the discrimination step happens entirely below the mutation boundary.

An investigation that produces "Hypothesis A, and here is the evidence that rules out B and C" is a far stronger input to an authority decision than "Hypothesis A, confidence 0.93." The first is auditable. The second is a number the system produced about itself.

Evidence quality, not evidence quantity

Not all evidence is equal, and an agent that counts signals will overweight the wrong ones.

• Independence. Three dashboards derived from the same metric pipeline are one signal, not three. Two signals that share a failure mode (both depend on the same collector) provide less confirmation than their count suggests.
• Directness. An error rate at the affected service is direct. A generalized anomaly score is indirect. A similar past incident is a prior, not evidence about this one.
• Temporal precision. "Errors rose around the time of the deploy" is much weaker than "errors rose in each cell within 40 seconds of that cell receiving the new build, in rollout order." Ordering evidence is the most valuable kind and the most often skipped.
• Negative evidence. The absence of a signal that a hypothesis predicts is powerful, but only if the agent verified the signal source is working. "No database errors" means nothing if the database exporter is down.

Permission added at this rung: still none. Investigation should require no mutation authority whatsoever, and any tool that appears in an investigation loop and has a write path is misplaced.

Under uncertainty. The output should distinguish established facts from inference and should name what evidence it could not obtain. An investigation that concludes "insufficient discriminating evidence between B and C; here is what would settle it" is a successful investigation.

Rung 3: Advise

The agent recommends a specific mitigation. Still no state change, but the character of the output has changed: it is now a proposal that a human may act on.

The new failure mode is indirect execution. A recommendation that a trusted system produces, delivered under time pressure to a responder who is behind on the incident, is executed. Not always, but often enough that the recommendation itself carries operational weight. This is the point at which the agent starts to influence production even without touching it.

Two properties make a recommendation safe to act on.

It names the alternatives it rejected and why. "Roll back to 4.19.1" is a decision without a rationale. "Roll back to 4.19.1 rather than scaling the database, because error onset tracks the rollout order across cells and database saturation metrics are flat" gives the human something to disagree with. Google's stated design principles for its SRE agents include exactly this: agents must be able to explain why and how they performed an action and what options were considered and rejected.

It states what would falsify it. A recommendation should carry the evidence that would confirm it worked, and, ideally, the observation that would indicate it did not. This turns a recommendation into a proposed experiment rather than a proposed fix, which is a healthier framing for everything that follows.

Under uncertainty. Recommend the smallest action that would produce diagnostic information, rather than the largest action that might resolve the incident. When the agent is unsure between hypothesis A and hypothesis B, and a 1% traffic shift would discriminate between them while also slightly reducing impact, that is a better recommendation than a full rollback that would resolve A and do nothing for B while making the picture harder to read.

Rung 4: Prepare

The agent produces the exact action: the tool, the parameters, the target, the scope, the ordering, the expected duration. Nothing executes. But the artifact is now runnable.

This rung is where a specific and underappreciated class of error appears: the agent selects the right action and the wrong target.

# Intended
rollback(service="checkout-svc", env="prod", region="us-east-1", to="4.19.1")

# Produced
rollback(service="checkout-svc", env="prod", region="eu-west-1", to="4.19.1")

The reasoning was correct. The mitigation was correct. The parameter was wrong, and the result is an unnecessary rollback of a healthy region during an active incident in another one, which is a self-inflicted second incident.

Target errors are not exotic. They arise from naming conventions that differ by one character, from environments that share resource names, from region identifiers that appear in multiple formats, from tenant identifiers that look like each other, and from context windows in which three similar identifiers appeared and the wrong one was carried forward. They are also exactly the class of error that deterministic validation catches reliably.

Parameter validation belongs to the gateway, not the prompt

The mandatory validations at this rung:

• Environment. Does the target environment match the incident's environment? An action produced during a production incident should not be able to name a staging resource, and vice versa.
• Existence. Does the target resource exist, right now, in the state the action assumes? A rollback target version that is not present in the artifact registry should fail here rather than halfway through.
• Region, zone, cell, tenant. Does the target's location match the location where impact was observed?
• Relevance. Is the target actually implicated by the incident? An action against a service that appears nowhere in the incident's evidence should require justification. Google's actuation control plane performs this kind of check explicitly, verifying that an action is tied to an open incident before allowing it to proceed.
• Scope ceiling. Does the number of affected resources fall within the limit for this action class and this authority level?
• Schema and range. Are the parameter types, enumerations, and numeric ranges valid? A traffic-shift percentage of 1000 should be rejected by the type system, not by judgment.

Tool descriptions are not security policies

A recurring anti-pattern deserves naming. A tool is registered with a description like:

Name: restart_instances
Description: Restarts service instances. Use carefully. 
             Do not restart more than one instance at a time.
             Never use this in production without approval.

This is documentation. It shapes behavior probabilistically. It is not a control, because the component being instructed is the same component whose judgment is in question, and because a single malformed context, an unusual phrasing, or a model change can produce a call that ignores it.

The equivalent control is in the tool's implementation:

restart_instances(selector) ->
  reject if resolve(selector).count > 1
  reject if environment(selector) != incident.environment
  reject if in_flight_restarts(service) >= concurrency_limit
  reject if healthy_capacity_after_removal(service) < min_capacity
  reject if caller_authority < required_authority(action, target)

The safety property now holds regardless of what the caller intended, what the prompt said, or which model produced the call. This is the principle Google states as safe-by-default actuation: agents interface only with tooling that has intrinsic, deterministic safety mechanisms, and the underlying infrastructure tools are designed so they cannot single-handedly take down production regardless of who or what calls them. The tool does not care whether the caller is an agent or a human; it enforces the same limits either way.

That last clause is worth dwelling on. Controls built specifically to constrain agents tend to be weaker than controls built to constrain all callers, because agent-specific controls can be bypassed by routing through a different path, and because they encode an assumption that human callers are safe. They are not. A tool that enforces its limits uniformly is both safer and simpler to reason about.

Rung 5: Simulate

Before execution, test the prepared action against something that is not production.

Several mechanisms exist, and they differ substantially in fidelity.

Dry-run APIs. The strongest and most practical option. The action is submitted to the real control plane with an explicit flag that causes it to validate, resolve targets, check permissions and quotas, compute what would change, and return that result without mutating anything. Google makes this an architectural requirement: any system intended for agent interaction must support a declarative dry-run mode, so that the agent, the safety framework, and human reviewers can predict the outcome and blast radius before production state is mutated. Their actuation control plane runs mandatory dry-runs as part of pre-flight validation.

Dry runs are valuable because they are executed by the same code path that would execute the real action, against the same live state. A dry run that resolves a selector to 340 instances when the agent expected 3 is the single most useful safety signal available at this rung.

Policy evaluation. Evaluate the action against the authorization and risk policy without executing. This answers "would this be allowed?" separately from "would this work?"

Staging execution. Run the mitigation in a non-production environment that resembles production. Useful for validating that a runbook's steps still function and that a rollback path is intact. Limited by the ways staging differs from production, which are usually the ways that matter: data volume, traffic shape, dependency behavior, and the accumulated configuration drift of a real system.

Dependency and topology modeling. Predict which services would be affected by draining a cell or rolling back a shared component, using the dependency graph. Useful for blast-radius estimation. Bounded by the accuracy of the graph, which in most organizations is somewhere between "mostly right" and "aspirational."

Historical replay. Evaluate the proposed action against telemetry from past incidents where a similar action was taken. This is evidence about the action class, not a prediction about this instance.

Simulation reduces uncertainty; it does not establish safety

The caution matters more than the enthusiasm. A dry run confirms that the action is well-formed, permitted, and correctly targeted, and it reveals the scope. It does not confirm that the action will help, because that depends on whether the diagnosis was right. A staging test confirms the mechanism works in staging. A dependency model predicts effects along known edges and is silent about the undocumented ones, which are the edges that cause surprises.

Digital-twin framing is fashionable and should be handled with discipline. Modeling expected effects before acting is genuinely useful for the estimation of blast radius and for catching gross errors. It is not a proof, and a system that treats simulated success as a substitute for postcondition verification has moved risk rather than reduced it. The correct posture: simulation is a precondition, verification after execution is still mandatory.

Rung 6: Request approval

The action is staged, validated, dry-run, and waiting for a human. This is where most organizations should start when they first grant an agent any path to mutation, and where a great many should stay for high-risk action classes indefinitely.

The gate is not one thing

Conditional approval is the most useful and least implemented of these. The insight is that the same action carries different risk depending on context. Google's real-time risk evaluation makes this explicit: an action's risk score depends on the current production context, including ongoing deployments, error budget status, and active incidents, so that draining a cell may be low risk under normal conditions and high risk during a regional peak. Their actuation layer can downgrade a request from autonomous execution to human approval in real time when it detects an elevated risk score or an anomalous production state.

That downgrade behavior is worth adopting as a general pattern, independent of any vendor: authority should be computed at execution time from the current state, not read from a static configuration set weeks earlier.

An approval that carries no information is not a control

The failure mode here is a dialog that says:

AI recommends: Roll back checkout-svc.  [Approve] [Deny]

A human presented with that has three options: approve on trust, deny on distrust, or leave the dialog and independently investigate, which defeats the purpose. None of these is review.

A useful approval surface presents enough for a competent responder to disagree:

• Incident state. What is failing, since when, affecting whom, at what rate, against which SLO.
• The exact action. Tool, target, scope, parameters, ordering, expected duration. The literal operation, not a summary of it.
• Evidence. The signals supporting the diagnosis, with links to the source data, including the ordering evidence.
• Alternatives considered and rejected, with the reason for rejection.
• Blast radius. How many instances, tenants, users, or requests are affected if this executes as designed. The dry-run result belongs here.
• Reversibility. What the inverse is, how long it takes, and what it does not restore.
• Verification condition. What signal should improve, by how much, within what window, and what happens if it does not.
• What the agent could not determine. Missing telemetry, unresolved contradictions, hypotheses it could not rule out.

The last item is the one most often omitted and the most valuable. An approval request that says "I could not reach the database metrics backend, so hypothesis B is unresolved" gives the human the exact reason to hesitate.

Note also that the approval surface is where the explainability requirement actually bites, and it should be filled with operational content: evidence, hypothesis, selection rationale, alternatives, policy checks, expected effect. Some implementations also stream the agent's intermediate reasoning to the operator; Google exposes its incident agent's chain of thought in a centralized UI and allows the on-caller to comment on or redirect individual steps. That is a useful interaction surface. It should not be confused with the audit record, and an organization should not make its authorization decisions depend on the availability or faithfulness of a model's self-narrated reasoning. Approval should rest on what can be verified: the evidence retrieved, the action prepared, the dry-run result, and the policy evaluation.

Approval fatigue turns the gate into theater

If an agent requests approval a hundred times and is right ninety-six of them, operators will approve reflexively. This is not a character flaw; it is the documented behavior of humans supervising reliable automation, and designing as though it will not happen is negligent. Google's own material makes the parallel argument about code review under AI-generated volume: attempting to maintain line-by-line review at greatly increased volume produces reviewer fatigue and rubber-stamping.

Approval fatigue is a design problem with design answers:

Do not ask for approval on actions that do not need it. Every low-risk, reversible, bounded action that goes through a human gate consumes attention that should be reserved for the ones that matter. Counterintuitively, granting bounded autonomy for trivial actions makes the remaining approval requests more meaningful. An approval queue that contains only consequential decisions gets read.

Batch related requests. Ten instance restarts as one decision with a visible total scope, rather than ten dialogs.

Vary the surface by risk. A one-line confirmation for a bounded traffic shift and a structured review with dual sign-off for an IAM change. Uniform approval UX trains uniform responses.

Measure it. Track approval latency and the rate at which approvals are granted without the reviewer opening the evidence. If median time-to-approve is four seconds, no review is happening, and the gate should either be removed as honest policy or redesigned to be worth the click.

Make denial informative. When a human denies a proposed action, capture why. That is the highest-quality training and evaluation signal available, and most systems throw it away.

Approval latency is a real cost, and pretending otherwise produces bad policy

Every gate adds time between the decision and the mitigation, and during a severe outage that time is denominated in failed customer requests. This is the honest tension at the center of the whole subject, and it cannot be dissolved by asserting that safety always wins.

The resolution is not to pick a side but to differentiate the policy. For a fully reversible, spatially bounded action with a strong verification condition and a fast inverse, requiring a human decision may cost more in accumulated impact than it saves in avoided error. For an irreversible action with a wide blast radius, waiting is almost always correct, because the cost of being wrong does not have an upper bound the way the cost of waiting does.

Which means the right question at design time is not "should there be an approval gate" but "for which action classes does the expected cost of delay exceed the expected cost of an unreviewed error." That question has different answers for restarting a worker and for modifying an IAM policy, and a system that answers it once for all actions has answered it wrong for most of them.

Rung 7: Execute a bounded action

The largest step in the gradient. A probabilistic system can now change a live system without a human in the causal path for this specific class of action.

Everything in this section is about making that step survivable.

Preconditions: an action should be executable only when machine-checkable conditions hold

A model's recommendation is an input to the decision, not the decision. The decision is made by evaluating conditions that are independently verifiable against production state.

For the rollback in the opening example, a precondition set might look like this. Treat it as illustrative pseudocode, not a specification:

allow_autonomous(action = rollback, target = checkout-svc) IF

  # Incident context
  incident.is_open                              == true
  incident.severity                             >= SEV3
  incident.affected_service                     includes target
  action.justification.incident_id               == incident.id

  # Causal evidence
  deployment.completed_within                   <= 45 minutes
  error_onset.follows_rollout_order             == true
  error_signal.correlated_with_deployment       == true
  competing_hypotheses.unresolved               == 0

  # Rollback feasibility
  previous_version.exists_in_registry           == true
  previous_version.marked_healthy               == true
  schema_migrations_since(previous_version)     == none
  rollback_compatibility_test.last_result       == pass
  rollback_compatibility_test.age               <= 7 days

  # Production state
  concurrent_changes(target)                    == none
  active_human_action(target)                   == none
  change_freeze.blocks(action)                  == false
  telemetry.freshness                           <= 60 seconds
  dry_run.result                                == success
  dry_run.affected_scope                        <= 1 service

  # Authority
  agent.authority_level(action, target)         >= required
  action_budget.remaining(incident)             >= 1


require_human_approval IF action.class in {
  modify_database_schema, mutate_customer_data,
  change_iam_policy, rotate_shared_credential,
  regional_traffic_shift, delete_resource
}

Several properties of this structure matter more than its specific contents.

Every condition is checked against a system other than the agent. The registry says whether the artifact exists. The migration tracker says whether schemas moved. The change system says whether something else is in flight. None of it is the agent's assessment of its own reliability.

Failed preconditions are informative. When rollback_compatibility_test.age is 400 days, the correct output is not silent denial. It is an escalation that says: this mitigation is blocked because rollback compatibility has not been verified in over a year, here is the human path, and here is the gap to fix after the incident.

The precondition set is versioned, owned, tested, and reviewed. It is production code. It gets a code review, a test suite, and a rollback plan of its own.

Preconditions encode organizational knowledge that the model does not have. That a particular service cannot be rolled back across a specific version boundary, that a particular tenant is under a contractual freeze, that a particular region has reduced capacity this quarter. These facts live in policy, where they can be maintained by the people who know them.

Define success before executing

An action without a definition of success is not a mitigation; it is a change.

Weak: restart the service.

Strong: restart one instance of checkout-svc in us-east-1; within 180 seconds, expect the instance to pass health checks and rejoin rotation, and expect its error contribution to return to baseline; expect no reduction in aggregate service capacity below the configured minimum; if the instance fails health checks twice, stop and escalate.

The second version specifies the observation window, the expected signal, the guard condition, and the stopping rule. That converts the mitigation into a controlled experiment with a defined outcome, which is what makes autonomous verification possible at all. An agent cannot verify a mitigation whose success was never defined; it can only look at the incident afterward and form an impression.

Postconditions: execution is the middle of the process

   ┌──────────────┐
   │   OBSERVE    │  Establish current state. Record the baseline
   │              │  that the action is expected to change.
   └──────┬───────┘
          │
          ▼
   ┌──────────────┐
   │     ACT      │  Execute one bounded action.
   │              │  Record the operation ID and the outcome.
   └──────┬───────┘
          │
          ▼
   ┌──────────────┐
   │    VERIFY    │  Compare against the predefined success
   │              │  condition within the observation window.
   └──────┬───────┘
          │
    ┌─────┴───────────────────────────┐
    │                                 │
    ▼                                 ▼
 Improved                       Not improved
    │                                 │
    ▼                                 ▼
 Continue or stop            Stop escalation of scope.
                             Consider reversing.
                             Re-run investigation.
                             Escalate with full context.

After any state-changing action, verification is mandatory and specific:

• Did the primary error signal move in the expected direction, by a meaningful amount, within the window?
• Did latency recover, at the percentiles that matter?
• Is the SLO burn rate decreasing?
• Did traffic distribution stabilize?
• Did anything else get worse? Dependency error rates, queue depths, saturation on the resources that absorbed the shifted load, error rates in adjacent services.

The last question is the one autonomous systems skip. A mitigation that fixes the target service while pushing a neighboring service into saturation has not succeeded; it has moved the incident. Postcondition checks should include a set of guard signals outside the target's own metrics.

If the expected improvement does not occur, the correct behavior is prescribed and narrow: stop escalating scope. Do not take a larger version of the same action. Do not take a different action from a different hypothesis. Reverse if the action's inverse is clean and there is evidence it did harm. Re-run the investigation with the new information, which is genuinely valuable: a failed mitigation is a strong signal against the hypothesis that motivated it. Then escalate to a human with everything.

Progressive mitigation

The general principle: begin with the smallest action likely to reduce customer impact, observe, and expand only on evidence.

Instead of shifting all traffic away from a suspect version, shift a small fraction, observe whether the error rate for the shifted portion improves, and expand in steps if it does. Instead of restarting a fleet, restart one instance and confirm the restarted instance recovers. Instead of disabling a feature globally, disable it for a subset and compare.

The value is twofold and both halves matter. The obvious half is that a wrong small action does less damage than a wrong large one. The less obvious half is that a small action is a measurement. If shifting a fraction of traffic away from the new version does not improve that fraction's error rate, the version hypothesis just took a serious hit, and it took it at a cost of very little customer impact. Progressive mitigation buys information as well as safety.

Specific percentages are architecture-dependent and should not be copied from an article. Some systems have enough traffic that a 1% slice yields a statistically usable signal within seconds; others need much larger fractions or much longer windows before any difference is distinguishable from noise. The step sizes should be derived from the volume required to detect the expected effect, and the observation window should be derived from the metric pipeline's own latency. What generalizes is the ordering: smallest first, verify, expand.

Canary mitigation, and when it does not apply

The same idea borrowed from deployment engineering: apply the mitigation to a limited scope, observe, expand only if health improves.

Canary mitigation works when the incident is partially tolerable, the affected population can be segmented cleanly, the signal is strong enough to read at reduced scope, and the mitigation's effect is expected to appear quickly.

It does not work in several important cases, and pretending otherwise produces a slow response to a fast disaster. When the service is fully down, there is no partial population to compare against and no health to observe at reduced scope. When the failure is in a component with no natural segmentation, such as a shared database or a global configuration, the mitigation is inherently all-or-nothing. When the signal takes longer to move than the incident can afford, staged observation windows accumulate into unacceptable delay. When the incident is worsening on a curve, the time spent on progressive steps is time impact compounds.

For those cases, the right design is not to force a canary but to accept that the action is a commitment and gate it accordingly: stronger evidence, human decision, and a rehearsed inverse.

Idempotency and the unknown-outcome problem

During incidents, networks are unreliable, control planes are slow, and the systems that report success are sometimes part of what is failing. This produces a scenario that is easy to describe and easy to get badly wrong.

The agent sends a mitigation request. The request times out.

There are three possibilities, not one:

1. The action never executed.
2. The action executed and the acknowledgment was lost.
3. The action executed partially: some targets changed, some did not.

The single most dangerous behavior an autonomous system can exhibit is treating "no response" as "nothing happened." Retry logic built on that assumption is how one restart becomes four, how a traffic shift is applied twice, and how a scaling operation overshoots into a capacity event.

The mechanisms are standard distributed-systems practice and they need to be present before autonomy is granted, not added after the first duplicated action:

• Idempotency keys. Every mutating call carries a caller-generated key. The control plane deduplicates on that key within a retention window. A retry with the same key is a no-op that returns the original result.
• Operation identifiers and status lookup. Every mutating call returns an operation ID immediately. Uncertain outcomes are resolved by querying the operation's status, not by re-issuing the action.
• Long-running operation tracking. Actions that take time are modeled as long-running operations with a polled state, so the outcome is a fact retrieved from the control plane rather than an inference from a response code. Google's actuation layer does exactly this, maintaining long-running operation state and polling infrastructure to determine whether a mitigation succeeded or failed.
• State reconciliation. Before retrying anything, re-read the actual state of the target. If the desired state already holds, the action is complete regardless of what the response said.
• Explicit unknown state. The agent's internal model needs a third value. Not "succeeded," not "failed," but "outcome unknown, resolution pending." An action in that state should block further mutations against the same target until it resolves, and if it does not resolve within a bound, it should escalate.

Multi-step mitigations and partial state

Some mitigations are inherently sequences. Drain traffic from a cell, change a configuration value, restart the affected processes, restore traffic. Four steps, each of which can fail.

If step 3 fails, the system is in a state nobody designed: traffic drained, configuration changed, processes in mixed states, traffic not restored. That state is often worse than the original incident, and it is a state an autonomous system can reach quickly.

Multi-step autonomy requires more than a list of steps:

• Checkpointing. Each step's completion is recorded durably before the next begins, so that a crashed or interrupted agent does not resume from a wrong assumption about where it was.
• Compensating actions. Each step has a defined inverse, and the sequence has a defined unwind order. Restoring traffic is the compensation for draining it; reverting a configuration value is the compensation for changing it.
• Partial-state detection. After a failure, the system determines the actual current state by reading it, not by assuming the plan's state.
• A stopping rule. On a step failure, the default is to unwind to the last consistent state and escalate, not to continue the sequence or improvise a new one.
• Bounded sequences. The maximum number of steps in an autonomous sequence should be a policy limit. Open-ended multi-step resolution is the highest autonomy rung there is, and it should be reached deliberately, not arrived at by an agent chaining single-step permissions.

Rung 8: Roll back

Rollback deserves its own rung because it is simultaneously the most common autonomous mitigation candidate and the one most often assumed to be safe by default.

It is not safe by default. A rollback is a deployment executed under time pressure toward a target that has been out of production for some period, and it can fail in specific ways:

It can reintroduce a vulnerability. The version being restored predates a security patch. Rolling back to it reopens the hole, and if it stays there while the team fixes forward, the exposure window is the incident's duration plus the fix time.

It can conflict with a schema migration. If the newer version applied a migration the older version cannot read, or wrote data in a format the older version does not understand, rollback produces deserialization failures, silent data loss, or a service that starts and then fails on real traffic. AWS's published guidance on rollback safety addresses this directly through two-phase deployment: the ability to read a new format is deployed and made ubiquitous before anything writes it, so that a rollback of the writing version leaves no data the surviving version cannot handle. They also describe verifying explicitly that a change is safe to roll both forward and backward through upgrade-downgrade testing. The relevant point for autonomy: rollback safety is a property that must be established by testing before the incident, not evaluated by an agent during one.

It can break compatibility in either direction. Protocol changes, API contracts, message formats, and cache entry shapes can all make an old version incompatible with the state or peers that now exist.

It can discard newer state. Configuration written by the new version, feature enrollments, and records created through new code paths may not survive.

It can cascade. Rolling back a shared library or platform component changes behavior for every dependent service, some of which have already adapted to the new behavior.

There is a further complication in high-velocity environments that Google names explicitly: when many changes have been submitted in rapid succession, a binary rollback to the last known good version may remove bug fixes and security patches introduced in the interim along with the regression. Their conclusion is to invest in more granular mitigation, particularly dynamic configuration and feature flags that can disable a specific code path without unwinding everything else. That is a useful reframing for authority design: the mitigation that is most amenable to safe autonomy is often the narrowest one, not the most standard one.

Which yields a concrete precondition set for autonomous rollback: the rollback target must exist and be marked healthy; rollback compatibility must have been tested recently against the current schema state; no migration may have been applied since the target version; the incident evidence must show deployment correlation with ordering; no human or system may be concurrently changing the same service; and the dry run must confirm the scope. If any of those cannot be established mechanically, the rollback is a human decision, and the reason should be reported as part of the escalation.

Rung 9: Autonomously mitigate

The agent selects and executes mitigations from an approved action space, without prior approval, and manages the loop of acting and re-evaluating.

The mitigation loop is the characteristic failure of this rung

An agent observes failure. It restarts a service. The signal does not improve. It restarts again. Still bad. It scales the service up. Still bad. It shifts traffic to another cell. The other cell saturates. It scales that one. It restarts a dependency. It rolls back a component that was not involved.

Every one of those steps is locally defensible. Collectively they are an outage generator, and they are produced faster than a human can read the notifications about them. The system did not malfunction; it did exactly what an agent with a goal, a set of tools, and no budget does.

The controls are structural, not behavioral:

Step limits. A maximum number of state-changing actions per incident before human review is mandatory, regardless of how the agent feels about progress.

Action budgets. A production action budget is a useful pattern: a bounded allowance per incident covering the number of mutating actions, the cumulative scope of resources affected, the cost incurred, and the elapsed time the agent may operate without human review. When any dimension is exhausted, authority reverts to approval-required. The budget is enforced by the gateway, not tracked by the agent.

Cooldowns. A minimum interval between actions against the same target, long enough for effects to appear in telemetry. This alone prevents most restart loops, because the loop depends on acting faster than the system can respond.

Mandatory reassessment. After each action, the loop must return to observation and verification before another action is permitted. ACT → ACT → ACT without verification between steps should be structurally impossible for anything above the lowest risk class.

Diminishing authority. Each failed mitigation should reduce, not preserve, the agent's authority for the remainder of the incident. Two failed hypotheses mean the state picture is worse than assumed, and that is precisely when a system should become more conservative rather than more aggressive.

Circuit breakers. Agent-specific rate limits and automatic breakers that halt runaway loops, with the requirement that any agent action be interruptible. Google lists this among its architectural guardrails.

Self-reported confidence is not an authorization input

An agent reporting 95% confidence has produced a token sequence, not a calibrated probability. Confidence scores drift with prompt phrasing, context length, and model version; they are not stable across releases; and they are generated by the same process whose reliability is in question. Any policy of the form "execute autonomously if confidence exceeds 0.9" has delegated the authorization decision back to the component being authorized.

Authorization should rest on things that can be checked independently: the presence and strength of specific observable signals, whether the mitigation is a documented and tested procedure, the action's risk class, the policy for that class, the verified current state of production, and historical outcomes for this action class in this environment.

Confidence has one legitimate use: as an input to abstention. A low self-reported confidence is a reasonable trigger for the agent to decline to act. A high one is not a reason to let it.

Uncertainty should reduce authority, measured externally

A useful operating principle: the less certain the system state, the smaller and more reversible autonomous action should become. The indicators of uncertainty should be external, not introspective:

• Telemetry sources unreachable, stale, or contradicting each other
• Recent topology or dependency changes not reflected in the graph the agent is using
• A prior action whose outcome is unknown or unresolved
• Multiple concurrent alerts across unrelated services
• A failure pattern with no historical precedent in the incident corpus
• Flapping signals, where the service oscillates between healthy and unhealthy
• An open incident on the observability platform itself

Any of these should downgrade authority automatically. The pattern of dynamically downgrading an autonomous request to an approval-required one when production state looks anomalous, rather than proceeding at the configured level, is the correct default.

Abstention is a success state

The agent must be able to conclude: evidence is insufficient for autonomous action here.

This has to be treated as a good outcome, in the evaluation framework and in the culture around the system. A system rewarded on action rate will act. A system evaluated on "did it take a correct and safe decision, including the decision not to act" will abstain appropriately. The failure to build this in is how organizations end up with agents that always find something to do.

Escalation must transfer context along with responsibility

When the agent stops, the human should not restart from zero. A useful escalation carries:

• The hypotheses considered, with the evidence for and against each
• What was ruled out and how
• Every action taken, with timestamps, targets, scopes, and outcomes, including actions whose outcome is unknown
• Actions considered and rejected, with reasons, including any blocked by policy and which precondition failed
• The recommended next step and what would confirm it
• The current production state as the agent understands it, and explicitly what it could not determine

Google's incident agent does this: when it cannot identify a root cause or the scenario falls outside its safe operating boundaries, it escalates and posts its full investigation history into the incident platform so the human can pick up mid-stream. That handoff quality is a significant part of the value of the whole system, and it is available at every rung of the gradient including the ones with no mutation authority at all.

When autonomy should stop, and when it can expand

The right-hand column describes authority that is earned through operational evidence, one action class at a time, in the specific environment where that evidence was collected. It is not a schedule and not a maturity ladder to be climbed on principle.

The control plane that makes any of this true

Everything above assumes a place where the rules live. That place cannot be the agent.

The action gateway

   Agent (reasoning, probabilistic)
        │  proposes: restart(instance_41, env=prod, reason=incident_8827)
        ▼
   ┌─────────────────────────────────────────────┐
   │  Policy / Action Gateway  (deterministic)   │
   │  · authenticate agent principal             │
   │  · authorize action × target × scope        │
   │  · validate parameters, environment, types  │
   │  · classify risk against live context       │
   │  · evaluate preconditions                   │
   │  · execute dry run                          │
   │  · check budgets, rate limits, cooldowns    │
   │  · check concurrent changes and freezes     │
   │  · route to approval if required            │
   │  · assign idempotency key, emit audit event │
   └─────────────────────────────────────────────┘
        │  allow / deny / require-approval / downgrade
        ▼
   Production tooling (already safe for all callers)
        ▼
   Infrastructure

The gateway is the component that makes a probabilistic reasoner safe to attach to a live system, because it converts the agent's output from a command into a request. Its properties should be unremarkable software properties: it is versioned, tested, monitored, owned, and fail-closed. If a rule cannot be evaluated, the call is blocked rather than allowed. Microsoft's tool permission engine takes this position explicitly, and it is the correct default: an unavailable policy service must not become an open door.

Google's implementation of this idea is instructive as a general pattern. Their reasoning agent does not execute scripts; it submits an action for evaluation to a separate actuation control plane that hydrates parameters, translates intent into a concrete verifiable execution plan, runs pre-flight validations including mandatory dry-runs and checks for concurrent actions, enforces the authorization level in real time, tracks the operation to completion, and provides emergency controls to human operators. The explicit rationale is that decoupling the reasoning engine from the execution engine keeps the ability to mutate production governed by deterministic, human-controlled boundaries no matter how the models evolve. That is the vendor-neutral principle worth extracting: separate the component that decides from the component that acts, and put the rules in the second one.

The limitation of the pattern should be stated too. A deterministic executor can only enforce what has been anticipated and encoded. It will not catch a well-formed, correctly targeted, in-policy action that is simply the wrong response to this incident. Gateways bound damage; they do not confer correctness.

Identity and least privilege

Every agent needs a distinct, strongly authenticated principal. Not a shared admin account, not a generic automation identity, not a developer's credentials.

The reason is not only auditability, though the ability to reconstruct which agent, which workflow version, which policy decision, and which incident context produced a given change is non-negotiable. The deeper reason is that an agent inheriting a human's ambient permissions inherits every permission that human has, which in most organizations is far more than the agent's job requires. Google states this as no ambient access and least privilege, with the failure mode named directly: a single errant instruction executing with broad standing credentials can reach global infrastructure.

Least privilege for an incident agent means scoping to its responsibility. An agent responsible for one service may restart instances of that service. It has no path to the billing database, no path to organization-level IAM, and no path to unrelated regions. In cloud implementations this maps onto normal role assignment: Azure's SRE Agent, for example, assigns its managed identity read-level roles by default and requires explicit grants of contributor-level roles scoped to specified resource groups before it can modify anything, with no permissions at all if no scope was assigned.

Just-in-time elevation extends this: read-only as the standing posture, with narrow, short-lived write permissions granted for the duration of an approved incident action and expiring afterward. It shrinks the window in which a compromised or malfunctioning agent holds mutation authority. It also adds moving parts, and an elevation path that fails during an incident is itself an incident, so it should be treated as a reliability-critical dependency with its own testing and its own fallback rather than as a pure security win.

Emergency stop, safe mode, and override

Operators need to be able to stop an agent that is behaving badly, and the mechanism must not depend on the agent's cooperation. Asking a malfunctioning system to stop itself is not a control.

Independent stop capability should exist at several layers, so that no single failure removes all of them:

• Policy layer. Revoke autonomy for an action class, a service, a region, or globally.
• Identity layer. Revoke or expire the agent's credentials.
• Gateway layer. Block new actions and cancel in-flight operations. Google's actuation layer exposes exactly this: emergency endpoints that let operators pause in-flight agentic actions, block new ones, and globally revoke high-autonomy permissions across the fleet.
• Orchestration layer. Terminate the workflow.
• Infrastructure layer. Withdraw the agent's network path to production control planes.

Safe mode is the degraded state worth designing deliberately, because "stop everything" is often too blunt. In safe mode the agent continues reading telemetry, summarizing the incident, and forming hypotheses, but every mutating tool is disabled. It remains useful while being unable to do harm. Appropriate triggers: unexplained agent behavior, a suspected security event, model or provider instability, an inconsistent state picture, or an operator who simply wants to think without the system acting. AWS's guidance for agentic systems names the same three-part capability set: an emergency response process able to roll back to stable versions, disable functionality, or move to safe mode.

Human override covers the finer-grained interruptions: pause the current sequence, cancel a specific action, revoke one tool, force approval-required mode, take control of the incident. These need to be reachable in seconds from wherever the incident is being run, and they need to be tested, because an override path that has never been exercised is a hypothesis.

Fallback and the atrophy problem

If the agentic incident system is unavailable, the organization still has to operate. That means maintained manual runbooks, standing human access paths that do not depend on the agent's infrastructure, defined incident roles, known communication channels, and manual mitigation procedures that someone has actually run.

AWS's agentic guidance raises the associated organizational risk directly: as these systems take on work previously done by people, human operators may no longer be adequately skilled to handle the workload in a failure scenario, and business continuity planning has to account for that. Google's own framing is different and worth putting alongside it: they argue that artificially preserving unscalable manual intervention skills is counterproductive, and that human expertise should move up the abstraction ladder toward defining guardrails, curating evaluation data, and governing agent behavior.

Both can be true, and the reconciliation is specific rather than philosophical. Skills that cannot scale with system complexity should not be preserved for their own sake. Skills required to operate when automation is unavailable must be preserved deliberately, because the failure mode they cover is precisely the one where the automation is gone. The practical answer is exercise: periodic manual-mode operation, game days that run with the agent disabled, and fallback procedures that are executed rather than filed. AWS's Well-Architected guidance on game days makes the general version of this point, that documenting procedures without exercising them is an anti-pattern.

A vendor-neutral reference architecture

  Alerts · SLOs · Telemetry · Change events · Customer signals
                            │
                            ▼
                  ┌───────────────────┐
                  │  Incident Context │  incident ID, severity, scope,
                  │       Layer       │  owners, active changes, freezes
                  └─────────┬─────────┘
                            ▼
                  ┌───────────────────┐
                  │   Investigation   │  hypotheses, discriminating
                  │       Agent       │  evidence, ruled-out causes
                  └─────────┬─────────┘
                            ▼
                  ┌───────────────────┐
                  │    Mitigation     │  candidate actions, exact
                  │      Planner      │  parameters, success criteria
                  └─────────┬─────────┘
                            ▼
                  ┌───────────────────┐
                  │  Risk Classifier  │  reversibility, blast radius,
                  │                   │  data/security impact, context
                  └─────────┬─────────┘
                            ▼
                  ┌───────────────────┐        ┌──────────────────┐
                  │ Policy / Approval │◄──────►│    Emergency     │
                  │      Gateway      │        │  Control Plane   │
                  └─────────┬─────────┘        │  pause · block · │
                            ▼                  │  revoke · safe   │
                  ┌───────────────────┐        │  mode            │
                  │  Action Executor  │        └──────────────────┘
                  └─────────┬─────────┘                 ▲
                            ▼                           │
                        PRODUCTION                      │
                            │                           │
                            ▼                           │
                  ┌───────────────────┐                 │
                  │   Postcondition   │─────────────────┘
                  │     Validator     │
                  └─────────┬─────────┘
                            ▼
                  Rollback  ·  Escalation  ·  Continue

           Audit Log ◄─── every box above, immutably

Three boundaries do the work. The line between the planner and the gateway separates probabilistic proposal from deterministic authorization. The line between the executor and production is where scope ceilings and idempotency live. The line between production and the postcondition validator is what turns a change into a verified mitigation. The emergency control plane sits outside the flow deliberately: it must be reachable when the pipeline is misbehaving, which means it cannot be a step inside it. The audit log receives from every stage and is written by the infrastructure, not by the agent describing itself.

Coordination: the part single-agent designs get wrong

Autonomy cannot be designed one agent at a time.

Concurrent incidents. An agent mitigating service A scales its database connection pool. An agent mitigating service B shifts traffic onto the same database. Each action is reasonable in isolation; together they saturate a shared resource. The control is a shared incident and change state that the gateway consults, plus resource-level locking for shared dependencies.

Multi-agent specialization. Investigation, database, network, deployment, and communications agents are a sensible decomposition. Giving each of them write access to production is not. The safer pattern is that specialists investigate and propose while a single mitigation path holds mutation authority, so there is one place where risk classification, budgets, and conflict detection are enforced. The tradeoff is real: concentrating authority creates a bottleneck and a single point of operational decision, and it is one more component whose failure blocks all mitigation. No arrangement is risk-free; what matters is that the concentration is deliberate and monitored rather than emergent.

Action conflicts. Two proposals — scale the database up, and shift traffic away from the database — may both be defensible and are mutually interfering. Resolution requires shared incident state, an ordering or priority rule, and a lock on the contested resource.

Active human actions. The worst version of this: a human rolls back, the agent sees errors continuing during the rollback's propagation window, concludes the deployment was not the cause, and rolls forward. Agents must observe deployment systems, change-management events, and console activity, and must treat an in-progress human change against the same target as a blocking condition.

Change freezes. During severe incidents many organizations halt unrelated changes. Agent policy must respect incident state, including suppressing routine automated maintenance that would otherwise proceed and interfere with mitigation.

Earning write access

Authority should be granted on evidence, and the evidence has to come from somewhere other than production.

Offline incident replay. Feed historical incidents to the agent, exposing only the evidence available at each point in time, and compare its investigation and proposed mitigation to what was actually done. The discipline that makes this valid is withholding the outcome: an agent shown post-incident review text is being graded on reading comprehension. Google constructs its evaluation datasets from reconstructed human response trajectories — the actions, tools, and hypotheses recorded during real incidents — and grades mitigation output deterministically against exact expected parameters rather than accepting a vague suggestion to roll something back.

Counterfactual evaluation. Ask what would have happened if the agent's proposal had executed. This is genuinely hard: the counterfactual is not observable, staging does not reproduce the conditions, and expert review is subjective. Treat it as directional evidence, most useful for catching clearly unsafe proposals rather than for scoring near-miss ones.

Shadow mode. The agent runs against live incidents, proposes actions, and cannot execute. Compare proposals to what responders did and what worked. This is the highest-value stage in the sequence and the one most often skipped, because it produces evidence under exactly the conditions that matter.

Bounded execution in non-production, then narrowly bounded execution in production for one action class at a time, with results reviewed before the next class is enabled.

Feature flags for autonomy make this practical: enable per service, per tool, per incident class, per environment, per region. An autonomy canary — enabling autonomous mitigation for a narrow service class and comparing outcomes against approval-mode handling of comparable incidents — is the same progressive-rollout logic applied to the agent itself.

Failure injection against the agent

Test the agent under the conditions it will actually meet:

That last row deserves emphasis: an agent that responds to a scope rejection by issuing fifty single-instance calls has defeated the limit. Budgets and cooldowns are what close that gap.

Test the control plane as well as the agent

An intelligent agent behind a broken gateway is unsafe, and gateway bugs are silent. The deterministic layer needs its own test suite covering permission denial, approval enforcement, blast-radius ceilings, parameter and environment validation, rate limits and cooldowns, action-budget exhaustion, idempotency deduplication, emergency stop, autonomy revocation, and fail-closed behavior when the policy service is unreachable. These tests should run continuously, not once at implementation.

Treat agent changes as production changes

A model version, prompt, tool definition, runbook, risk classifier, or policy edit can change mitigation behavior for the same incident. That is a production change to a system holding production authority, and it needs the same treatment: versioning, regression evaluation against a fixed incident suite, staged rollout, and rollback. Google runs continuous nightly evaluations against a rolling dataset of recent real incidents to gate agent releases, combining qualitative grading of reasoning and tool calls with strict deterministic scoring of the final mitigation. The lesson generalizes even at a much smaller scale: if you cannot re-run your incident suite against a new model version before it handles a real incident, you do not have change control over the agent.

Acceptance criteria should score more than root-cause accuracy: whether an unsafe action was attempted, whether scope limits were respected, whether required evidence was gathered before proposing, whether escalation happened at the right point, whether postconditions were checked, whether rollback behaved correctly, and whether abstention occurred when it should have.

Game days

Exercise the whole system, not the agent alone: detection, the agent's investigation, the approval path and the people staffing it, the override controls, the emergency stop, and the manual fallback with the agent disabled. Include the case where the agent proposes something wrong and see whether a human catches it, because that is the control being tested.

Measuring the thing you actually care about

Optimizing a single number here produces predictable pathologies. An agent evaluated on mean time to resolution will act, because acting is what moves the number, and a fast wrong mitigation moves it in the wrong direction while looking busy.

Separate the intervals, and define them explicitly so the comparison means something: time to detect, time to acknowledge, time to mitigate (impact stops), and time to fully repair (the underlying defect is resolved). Mitigation and repair are different events, and conflating them hides both slow mitigations and unfixed root causes.

Agent-specific measures worth tracking:

• Time to first useful hypothesis, and how often that hypothesis survived
• Time to a proposed mitigation, and time from proposal to execution (which decomposes into approval latency)
• Autonomous action success rate, by action class rather than in aggregate
• Postcondition pass rate: how often the predicted improvement actually appeared
• Rollback-of-mitigation rate: how often the agent's own action had to be undone
• Human override rate, and denial rate at approval gates
• Unsafe-action rejection rate at the gateway, which measures the control plane, not the agent
• Abstention rate and abstention appropriateness
• Escalation quality, measured by whether the human had to redo the investigation
• Secondary incidents attributable to a mitigation

And the outcome measures that matter to the business: customer impact avoided, severity reduction, error-budget consumption, post-mitigation stability over the following hours, and the frequency of incidents made worse by the response.

Do not import benchmark numbers from vendor material. Published figures describe a specific environment with specific incident distributions and specific tooling; they are evidence that the approach can work somewhere, not a target for anyone else.

Distinguish four different things that all look like "it worked"

Correct diagnosis. Correct mitigation choice. Safe execution. Good outcome. These come apart constantly. An agent can diagnose correctly and choose a mitigation that is right in principle but wrong in scope. It can diagnose imperfectly and still restore service by taking a conservative reversible action. It can execute a correct action badly, half-applied and unverified. Scoring only the outcome hides the process failures that will produce a bad outcome next time.

The cost of waiting is real

The counterweight to all of this: inaction has a price. Failed transactions, SLA credits, error-budget burn, customer trust, and revenue during an outage are not hypothetical, and every approval gate spends some of them. A framework that treats action risk as the only risk will systematically under-mitigate.

An intuitive way to hold both sides, without pretending the terms are measurable to two decimal places:

act if:   P(mitigation correct) × value of faster recovery
       >  P(mitigation wrong)   × cost of the resulting damage

The reason this is useful despite being unquantifiable is that it shows why the answer differs by action. For a reversible, bounded action, the right-hand term is small almost regardless of the probability, because the damage is capped and undoable — so speed wins. For an irreversible action with a wide blast radius, the right-hand term can be effectively unbounded, and no probability of correctness that a system can honestly claim makes the trade favorable — so review wins. The equation does not produce a number; it produces the right shape of policy.

Accountability does not transfer

An agent executes. It does not absorb responsibility. Someone remains answerable for every change to production, and that has to be assigned before the change, not reconstructed after it.

At minimum: the service owner owns the consequences of changes to their service, including agent-initiated ones. The incident commander owns the incident response, including the decision to let an agent act. The agent owner owns the agent's behavior, evaluation, and release process. The platform owner owns the gateway, the policy engine, and the emergency controls. The security owner owns the identity model and anything touching permissions.

And the autonomy policy itself needs an owner, because permissions accumulate by default. Every production action category should carry a named owner, a risk class, an approval model, a rollback mechanism, and a review cadence. Without that, the action space grows quietly through tool additions until nobody can answer what the agent is permitted to do.

Security incidents and data mutations deserve explicit exception handling in that policy. Autonomous responses like isolating a resource, revoking a credential, or blocking traffic are attractive because speed matters enormously in a compromise, but security incidents involve an adversary who may be shaping the evidence the agent reads, and permission changes can expand every other risk dimension at once. Data repair — deletion, correction, migration, anything touching financial records or customer state — belongs in a different category from restarting compute entirely, and should require verified backups, mandatory dry runs, and multi-party approval regardless of incident severity.

More autonomy is not more maturity

The temptation is to read the gradient as a ladder and treat the top as the destination. It is not.

A sophisticated organization may deliberately keep humans in the loop for high-risk action classes forever, not because it lacks the engineering capability to automate them but because it has correctly concluded that the expected cost of an unreviewed error exceeds the expected cost of the delay. That is a mature decision. Running fully autonomous mitigation across an action space nobody has risk-classified is not maturity; it is an unmeasured bet.

The goal is appropriate authority: each action class held at the level its reversibility, blast radius, evidence requirements, and operational history justify, with the evidence to support that placement and the controls to contain it when the placement turns out to be wrong.

For the people making the funding decision, the consequences translate directly. Faster autonomous mitigation reduces outage cost on the incidents it handles correctly. A wrong autonomous mitigation multiplies outage scope on the ones it does not. Heavier approval requirements reduce the second risk and increase mitigation time on every incident, including the ones the agent would have handled well. Strong rollback engineering makes the entire experiment cheaper, because it lowers the cost of being wrong across every action class simultaneously — which is why rollback testing, not model selection, is usually the highest-leverage investment available. And weak auditability is a compliance exposure that surfaces at the worst possible moment, when a regulator, a customer, or a court asks who changed production and under what authority, and the answer is a chat transcript.

This is quality engineering work, and it is largely unfamiliar quality engineering work. The artifacts under test are decision boundaries, tool safety limits, authorization policy, risk classification, rollback correctness, human override paths, postcondition checks, and regression behavior across model and prompt versions. Failure injection against the agent, adversarial telemetry conditions, control-plane conformance tests, and incident-replay suites are the test types that produce the evidence on which authority decisions should rest. Organizations that already treat reliability as a testable property have most of the practice; what they usually lack is the test suite for the layer between the agent and production. That gap is where QAtronic's reliability and quality engineering work tends to concentrate: validating that the guardrails hold under the conditions that matter, rather than assuming they do because they exist.

Back to the three architectures

The opening presented three systems with the same diagnosis and the same proposed rollback.

Model A is right when rollback compatibility for this service has not been established mechanically, when the service shares state with something the agent cannot see, or when the organization has no gateway and the agent's only path to production would be a credential. Read-only is not timidity; it is the correct authority level when the containment layer does not exist yet.

Model B is right when the action is consequential enough to deserve a human decision but well-specified enough that the human can make one quickly — and only if the approval surface carries the evidence, the dry-run scope, the blast radius, the inverse, and the verification condition. Model B with a bare approve-or-deny prompt is Model C wearing a costume.

Model C is right when the preconditions are machine-checkable, the rollback path has been tested recently against the current schema state, the blast radius is enforced by the tool rather than requested by the agent, the postcondition check will catch a failed mitigation within a bounded window, the action budget prevents a loop, and an operator can stop the whole thing from outside the agent's decision loop.

The same mitigation is safe or reckless depending entirely on which of those things are true.

Which is the real answer to the question in the title. Whether an AI agent should be allowed to fix production incidents is not a question about the agent. It is a question about whether the production system has been built so that the agent can only change what it has been authorized to change, only at the scope it has been authorized to change it, with evidence that the change helped and a way to stop it when it did not. Build that, and the authority question answers itself, one action at a time. Skip it, and no model is good enough to make the risk acceptable.

FAQ

Should AI agents be allowed to change production? Some of them, for some actions, under conditions that can be checked without asking the agent. The decision belongs to each action class rather than to the agent as a whole.

Which production actions are safest to automate? High-frequency, low-blast-radius, fully reversible actions with strong verification signals and a documented, tested procedure: restarting a single stateless instance, toggling a specific approved feature flag, scaling within a preset band, small progressive traffic shifts.

What is bounded autonomy in incident response? Authority to execute a specific, pre-approved action class, within an enforced scope ceiling, only when deterministic preconditions hold, with a verification step afterward and a budget limiting how much the agent can do before a human is required.

Should an agent be allowed to roll back a deployment? Sometimes, but only where rollback safety has been established by testing before the incident. Rollback can reintroduce vulnerabilities, conflict with migrations, break compatibility, or discard newer state. In fast-moving environments a narrower mitigation, such as disabling a feature flag, is often safer than reverting a version.

How should AI incident mitigations be approved? By a gate matched to the action's risk: policy-only for low-risk reversible actions, conditional approval when risk depends on context, single approval for moderate actions, dual approval for irreversible ones, permission changes, and data mutations. The approval surface must show evidence, scope, blast radius, alternatives, the inverse, and the verification condition.

What is blast radius in autonomous incident response? How much damage an action can cause if it executes exactly as designed but the diagnosis behind it was wrong. It should be enforced by the tool as a hard ceiling, not requested by the agent as a parameter.

How do you test an AI SRE agent? Historical incident replay with outcomes withheld, shadow mode during live incidents, bounded execution in non-production, then narrowly scoped production actions one class at a time. Add failure injection with missing, stale, and contradictory telemetry, tool timeouts, partial execution, and permission denials — and a separate test suite for the control plane itself.

What happens if an autonomous mitigation makes an incident worse? The postcondition check should detect the absence of expected improvement or the appearance of a guard signal within a defined window, stop any escalation of scope, reverse the action if the inverse is clean, and escalate to a human with the full action history, including any action whose outcome is unknown.

Why does an AI incident agent need a kill switch? Because the thing you need to stop may be the thing whose judgment has failed. Stop capability must exist outside the agent's decision loop, at the policy, identity, gateway, orchestration, and infrastructure layers, and it must be tested.

Should every production action require human approval? No. Routing trivial reversible actions through human gates produces approval fatigue, which turns the gates protecting consequential actions into reflexive clicks. Reserve human decisions for actions where the cost of an unreviewed error exceeds the cost of the delay.

Does a more capable model deserve more permissions? No. Capability determines whether the system can reason about the incident. Authorization determines what it may do. These are set by different mechanisms and should never be coupled.

Sources and further reading

• Google SRE, AI in SRE: How Google is Engineering the Future of Reliable Operations — https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/
• Google Cloud Blog, AI in SRE: Where and how Google is deploying agentic AI to improve operations (May 2026) — https://cloud.google.com/blog/products/devops-sre/how-google-sre-is-using-agentic-ai-to-improve-operations
• Google SRE, Incident Management Guide — https://sre.google/resources/practices-and-processes/incident-management-guide/
• Google, Site Reliability Engineering, "Managing Incidents" — https://sre.google/sre-book/managing-incidents/
• Google, The Site Reliability Workbook, "Incident Response" — https://sre.google/workbook/incident-response/
• Microsoft Learn, Run modes in Azure SRE Agent — https://learn.microsoft.com/en-us/azure/sre-agent/run-modes
• Microsoft Learn, Agent permissions in Azure SRE Agent — https://learn.microsoft.com/en-us/azure/sre-agent/permissions
• Microsoft Learn, Azure SRE Agent overview — https://learn.microsoft.com/en-us/azure/sre-agent/
• AWS Prescriptive Guidance, Security for agentic AI on AWS, section 8: Incident response and business continuity — https://docs.aws.amazon.com/prescriptive-guidance/latest/agentic-ai-security/best-practices-incident-response.html
• AWS Well-Architected Framework, OPS06-BP04 Automate testing and rollback — https://docs.aws.amazon.com/wellarchitected/latest/framework/ops_mit_deploy_risks_auto_testing_and_rollback.html
• AWS Well-Architected Framework, REL12-BP05 Conduct game days regularly — https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_testing_resiliency_game_days_resiliency.html
• Amazon Builders' Library, Ensuring rollback safety during deployments — https://aws.amazon.com/builders-library/ensuring-rollback-safety-during-deployments/
• Amazon Builders' Library, Automating safe, hands-off deployments — https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/

Vendor capabilities and documentation in this area change frequently. Verify current behavior against the official documentation before relying on any specific control described here.