From Runbook to Governed API

Most incidents never need anyone to reach into production. The telemetry an engineer already has is enough, the dashboards tell the story, and the fix ships through the normal path. Every so often one does not go that way. A SEV1 opens on a single service, the shared views are not enough to see what is happening, and whoever is on call needs to look at the production logs, traces, and recent deploys for that one service to find the cause.

At 14:09 the page goes out. Alice, on call, picks up a SEV1 on checkout-api: 5xx errors climbing after a latency spike, and nothing in the shared dashboards explains why. She needs the last half hour of logs and traces for that service, and the deploys that landed just before it started. A human in her seat knows this is a reasonable thing to look at during an incident, and knows not to go wandering into payments or customer records while she is there. An agent helping her does not know any of that on its own. That knowledge is the runbook, and it is the part that kept the work manual.

Open the runbook and the imbalance is the same as it always is. The action at the bottom is a single query: pull the last thirty minutes of logs and traces for the service, and the deploys that came just before.

obsctl query checkout-api --signals logs,traces --since 30m --env prod

Everything above that line is the part that kept a human in the loop:

• Only pull production data while a real incident is open.
• Stay inside the incident: the failing service, the right environment, a recent window.
• Read-only observability only, like logs, traces, metrics, and recent deploys. Nothing that writes.
• Never surface secrets, payment data, or raw personal data, even by accident.
• A database session, or anything that can change state, waits for a second person.

Those conditions are policy, even if nobody calls them that. A human on-call engineer carries them in their head and applies them without thinking. An agent has none of that judgment, so the tempting shortcut is to hand it a read-only key to the logging stack and let it query whatever it decides it needs. That moves the judgment into the one place that cannot enforce it. A model can misjudge the scope, act on an incident that was resolved an hour ago, or be talked into a service and a time range that quietly include data it should never see, and the query runs anyway. The agent does not need to own production. It needs enough context to help the human find the cause faster.

In Agent-Ready Infrastructure we called this pattern Runbooks as Governed APIs, and the idea underneath it is almost boring in its simplicity: let the agent ask for what it needs, and put a separate piece of software in charge of deciding what it is actually allowed to see. What follows is that piece of software, built for the one observability request above.

Make it concrete. The SEV1 on checkout-api is open, Alice is on call for it, and the logs and traces she needs live behind a boundary she does not normally cross. An agent watching the incident prepares a request on her behalf: logs, traces, and recent deploys for checkout-api, in production, for the last thirty minutes, because 5xx errors are climbing after a latency spike.

There are two ways to build this, and the difference matters. The API could check policy and then hand the agent a short-lived token to the logging stack; that works, but now the agent holds a credential to production, even briefly. The cleaner design never gives it one. The agent does not get access at all. It sends a request to a governed observability API, and the API does the work behind the boundary: it confirms the incident is real and active, confirms checkout-api is actually in scope, checks that read-only logs and traces for thirty minutes are inside policy, runs the query with its own credentials, redacts anything classed as secret or personal, and returns a filtered view. If any of that fails, the request is refused. Either way, the call is written into the audit log on the way out.

That is the whole move, and it rests on one line you can trace in the diagram: the query runs inside the boundary, and only a filtered view crosses back out. The agent makes the request; it never holds a credential to production. Look at the dashed verify arrow: rather than trust what the agent reported, the API confirms the incident and the scope against the systems that own them. The agent does not get a view because it asked nicely. It gets one because the governed API verified the incident, the service, the policy, and the time window.

None of this makes the prompt pointless; it just gives it a narrow job. Before the agent sends a single request, the prompt is what teaches it to read the incident and decide what context would actually help, and it fences in what the agent may ask for: incident-scoped, read-only, the least it needs, and never the kinds of data that should not leave the boundary at all. One small job, written in plain language:

You are incident-investigator-agent.

During an open incident, you help the on-call engineer find the
cause faster. You have no access to production. You cannot read
logs, traces, or metrics directly. You request incident-scoped
observability context, and a governed API returns a filtered view
if policy allows it.

Rules:
- Only act while there is a real, active incident.
- Ask only for context tied to this incident: the failing
  service, the right environment, a recent time window.
- Request the least you need. Prefer a narrow window and a short
  list of context types.
- Never ask for secrets, payment data, or raw personal data. The
  API will refuse, but do not ask.
- Anything beyond read-only observability, like a database
  session, goes to a human.
- Give one short sentence explaining what you are trying to find.

Tool:
request_observability_context(incident_id, service, environment,
                              requested_context, time_window,
                              reason)

The loop that carries this prompt is just as modest. It reads the incident, asks the model what context would help and how little of it is enough, sends the request under its own credential, and gets back a filtered view, not a key.

import os
import httpx

OBSERVABILITY_API = "https://observability.internal"
AGENT_TOKEN = os.environ["INCIDENT_INVESTIGATOR_AGENT_TOKEN"]


def investigate(incident_id: str) -> dict | None:
    incident = read_incident(incident_id)
    if not incident["active"]:
        return None

    decision = ask_model(
        goal="help the on-call engineer find the cause",
        observation=incident,
        available_actions=["request_observability_context"],
    )
    if decision.action != "request_observability_context":
        return None

    # The agent never holds credentials for the observability
    # stack. It asks the governed API for a view; the API runs the
    # query and returns a filtered result. A model can write any
    # JSON it wants, so the API verifies the incident and scope
    # against the systems that own them, not against the body.
    response = httpx.post(
        f"{OBSERVABILITY_API}/actions/request-observability-context",
        headers={"authorization": f"Bearer {AGENT_TOKEN}"},
        json={
            "incident_id": incident_id,             # "SEV1-2026-0412"
            "service": decision.service,            # "checkout-api"
            "environment": "production",
            "requested_context": decision.context,  # ["logs", "traces"]
            "time_window": decision.time_window,    # "last_30_minutes"
            "reason": decision.reason,
        },
    )
    return response.json()        # a filtered view, or a denial

The detail that matters most is the one easiest to miss: the agent never holds credentials for the observability stack. It authenticates to the governed API with its own identity, and the API queries the backend on its behalf. The body names the incident, the service, and the context it wants, but the API treats every field as a claim to be checked, not a fact to trust. The model writes arguments, the runtime turns them into an authenticated call, and the API confirms the incident is open and the service is in scope before it runs anything. In production that agent credential should be a real workload identity, a service account or a short-lived token or an mTLS certificate or an OIDC identity. If the model goes off the rails and asks for too much, the API is still the thing that refuses, and the data it would have exposed never moves.

A Governed Action API is a narrow endpoint for one operational action, even when that action is only reading. It checks who is asking and whether the request matches the live state of the world, runs it through policy, acts only inside the bounds it was given, and returns only what policy allows. Every decision is written down, answered or refused. Use it whenever an agent needs to reach production state and getting it wrong would cost something.

The route is where the runbook's checks finally turn into something that runs, and the first one is blunt: check the world, not the model's claim about the world. This is the part the agent cannot argue with.

from fastapi import Depends, FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel

app = FastAPI()


class ContextRequest(BaseModel):
    incident_id: str
    service: str
    environment: str
    requested_context: list[str]
    time_window: str
    reason: str


@app.post("/actions/request-observability-context")
async def request_observability_context(
    req: ContextRequest,
    agent: dict = Depends(authenticate),
):
    # Check the world, not the model's claim about the world.
    incident = await get_incident(req.incident_id)
    if incident is None or not incident["active"]:
        return JSONResponse(
            status_code=409,
            content={"outcome": "denied", "reason": "no_active_incident"},
        )

    if not incident.get("on_call"):
        await audit(agent, req, incident,
                    outcome="denied", reason="no_on_call_responder")
        return JSONResponse(
            status_code=403,
            content={"outcome": "denied", "reason": "no_on_call_responder"},
        )

    if req.service not in incident["affected_services"]:
        await audit(agent, req, incident,
                    outcome="denied", reason="service_out_of_scope")
        return JSONResponse(
            status_code=403,
            content={"outcome": "denied", "reason": "service_out_of_scope"},
        )

    decision = evaluate_policy(
        agent=agent,
        action="request-observability-context",
        req=req,
    )
    await audit(agent, req, incident, decision=decision)

    if decision["outcome"] == "deny":
        return JSONResponse(status_code=403, content=decision)

    if decision["outcome"] == "requires_approval":
        approval = await queue_approval(agent, req, decision)
        return JSONResponse(status_code=202, content=approval)

    # The agent never touches the backend. The API runs the query
    # with its own credentials and redacts the result before it
    # leaves the boundary.
    raw = await query_observability(
        service=req.service,
        environment=req.environment,
        context=req.requested_context,
        window=req.time_window,
    )
    view = redact(raw, blocked=decision["blocked_data"])
    await audit(agent, req, incident,
                decision=decision, returned=view["summary"], outcome="executed")
    return {"outcome": "executed", "context": view}

There is nothing exotic in this code, and it would be a mistake to add any. The safety does not come from cleverness; it comes from where the checks are placed. A request cannot reach the query until it has cleared the incident check, the scope check, and the policy bounds, and the raw result is redacted before it leaves the function. No branch quietly skips a step, and the agent never sees the unfiltered data.

The policy does not hide inside the service. It sits in an ordinary configuration file the API reads, and the test of whether it is written well is whether someone can open it and see what the system will and will not return without first reading the code around it.

agents:
  incident-investigator-agent:
    owner: platform-team
    actions:
      request-observability-context:
        autonomy: autonomous
        environments: [production]
        allowed_context: [logs, traces, metrics, recent_deployments]
        max_time_window_minutes: 60
        required_context:
          - active_incident
          - service_in_incident_scope
          - requester_on_call
        blocked_data: [secrets, payment_data, raw_personal_data]

      request-database-access:
        autonomy: requires_approval
        approvers: [sre-lead, security-oncall]

defaults:
  unknown_agent: deny
  unknown_action: deny
  audit: all

Read it once and the rules are plain. During an active incident, the agent can pull logs, traces, metrics, and recent deploys for a service in scope, for up to an hour. Three kinds of data, secrets, payment data, and raw personal data, are blocked outright and stripped from anything that comes back. Anything that can change state, like a database session, gets none of that latitude and waits for an SRE lead or security on-call. And anything the file does not recognize, an agent it has never heard of or an action no one declared, is denied by default. The prompt can suggest whatever it likes; this file is what the system actually returns.

Early on, the thing that reads this file can be plain code living inside the service.

ALLOWED_WINDOWS = {
    "last_15_minutes": 15,
    "last_30_minutes": 30,
    "last_60_minutes": 60,
}


def evaluate_policy(agent, action, req):
    agent_policy = POLICY["agents"].get(agent["id"])
    if agent_policy is None:
        return deny("unknown_agent")

    rule = agent_policy["actions"].get(action)
    if rule is None:
        return deny("unknown_action")

    if req.environment not in rule["environments"]:
        return deny("environment_not_allowed")

    # A kind of context nobody allowed is a human's call, not ours.
    extra = set(req.requested_context) - set(rule["allowed_context"])
    if extra:
        return {"outcome": "requires_approval",
                "approvers": ["sre-lead", "security-oncall"]}

    minutes = ALLOWED_WINDOWS.get(req.time_window)
    if minutes is None or minutes > rule["max_time_window_minutes"]:
        return deny("time_window_out_of_bounds")

    return {"outcome": "allow",
            "rule": f'{agent["id"]}:{action}',
            "blocked_data": rule["blocked_data"]}


def deny(reason):
    return {"outcome": "deny", "reason": reason}

There comes a point, once policy starts spanning many teams and actions and ownership models, where this should graduate to a real policy engine like OPA or Cedar. That point tends to arrive later than people expect, and until it does, a short evaluator you can read top to bottom is easier to trust, and easier to test, than a policy platform you adopted before you needed one.

A log line reading request-observability-context returned 200 tells you almost nothing worth knowing. The moment that matters comes weeks later, in a security review or a post-incident write-up, when someone asks two questions at once: what production data did an agent see, and why was that allowed. A record earns its place only if it can answer both, which means it has to hold far more than the outcome.

{
  "id": "evt_01k2_obs_context_001",
  "created_at": "2026-06-20T14:09:52Z",
  "caller": {
    "type": "agent",
    "id": "incident-investigator-agent",
    "owner": "platform-team"
  },
  "action": "request-observability-context",
  "input": {
    "incident_id": "SEV1-2026-0412",
    "service": "checkout-api",
    "environment": "production",
    "requested_context": ["logs", "traces", "recent_deployments"],
    "time_window": "last_30_minutes",
    "reason": "Investigating elevated 5xx errors after a checkout latency spike"
  },
  "observed_context": {
    "incident_active": true,
    "service_in_incident_scope": true,
    "requester_on_call": true,
    "context_within_policy": true,
    "redaction_applied": ["payment_data", "raw_personal_data"]
  },
  "policy_decision": {
    "outcome": "allow",
    "matched_rule": "request-observability-context:logs,traces,recent_deployments:window=30m"
  },
  "decision_explanation": "Active SEV1 on checkout-api; in-scope, read-only observability for the last 30 minutes; payment and personal data redacted.",
  "result": {
    "status": "executed",
    "returned": "412 log lines, 38 traces, 2 recent deploys",
    "credentials_issued": false
  }
}

The part to trust is observed_context, because it is what the API itself confirmed at the moment of the query, not what the agent claimed, down to which data classes it redacted. The decision_explanation field is more awkward: useful in a review, and noisy if you let it sprawl, so it is kept to one sentence. Do not store the raw result beside it; the whole point was to keep that data behind the boundary. And because refusals matter as much as answers in a security review, the log records the denials too, with the reason attached.

None of this saves you from bad data underneath it. If the on-call schedule is stale, the API will treat the wrong incident as legitimate. If service ownership is wrong, the scope check passes for a service that should have been out of bounds. If the data classification is incomplete, the redaction step strips nothing, and personal data rides out in a filtered view that was never filtered. If an incident can be opened with a self-declared SEV1, the whole chain rests on a fact anyone can fabricate. The policy file can read flawlessly the entire time while everything beneath it rests on sand, which is why Agent-Ready Infrastructure never mentions governed APIs without operational ground truth in the same breath.

Review does not disappear, either. A database session, or anything that touches data rather than reads observability, still calls for a person, and it should. What changes is only what that person is handed: the same structured request the agent prepared, with the incident, the scope, and the policy decision already attached, instead of a Slack message they have to chase down and reconstruct.

Approval itself can be done badly. Send every request to a queue and the approvers stop reading it; send none and you have trusted the policy more than it has earned. The line worth drawing follows the risk, so read-only, redacted observability during a live incident runs on its own inside tight, time-boxed bounds while anything that can change production, or expose sensitive data, holds on to its context and waits for a human.

And the audit log, which begins life as a convenience, ends up as infrastructure. A JSON file is fine while you are proving the idea works. A system people lean on will want retention and search over that log, real access control, some resistance to tampering, and a way to line its entries up against incidents, deploys, and approvals when an investigation needs to see all of them at once.

If we were building this for real, we would not begin with a general operations agent that does a hundred things. We would begin with one action whose judgment is already written down somewhere, which is to say the exact observability request in this article, and we would build it as a query broker rather than a credential vending machine, so the agent never holds a key to production. The runtime around it stays deliberately dull: one worker watching incidents and calling one governed API, with a single credential, a single policy file, and a single audit log behind it. On GCP that is a Cloud Run service in front of your logging stack; on AWS it is a Lambda in front of CloudWatch or OpenSearch, behind the same boundary. None of the plumbing wants to be interesting.

Then we would go through the runbook one warning at a time and ask what each one wants to become. Some are preconditions the API should check for itself, like whether the incident is really open and the service is really in scope. Some are bounds the policy should hold, like read-only observability only and an hour at most. Some are redactions, like never letting payment or personal data cross the boundary. Some are approvals that belong to a person, like a database session. And some are simply things the audit log should remember, like what an agent was shown and why. Most of the warnings settle cleanly into one of those.

The interesting ones are the warnings that refuse to settle anywhere. When a line in the runbook cannot become a precondition, a bound, a redaction, an approval, or an audit field, it is quietly telling you it still leans on context the system does not yet have, the on-call schedule it cannot read, the data classification nobody finished. That is not a failure. It is a map of exactly where the infrastructure is not ready to be trusted on its own, and it is worth more than any of the parts that fit.

Pulling the observability request behind an API turns out to be more than dressing a query in HTTP. The judgment a human on-call engineer applies without thinking, what is fair to look at during an incident and what is not, gets spread into the places software can actually hold it, the preconditions and the policy and the identity and the redaction and the audit trail, where it stays put whether or not the right person is awake. The prompt asks for the behavior; the backend enforces it. The day those two roles blur together is the day a helpful agent becomes a standing way to read production.

Once that boundary is real, the relationship changes. The agent is no longer trusted with production; it is only allowed to ask a system that already knows how to refuse, and that hands back a filtered view instead of a key. That is the whole distance between an agent that looks helpful in a demo and one you would let investigate a SEV1 at 14:09, knowing the worst it can do is ask.

Does the agent get access to the logs?

Not directly. In the safer design it never holds credentials at all. It sends a request to a governed observability API, and the API runs the query behind the boundary, redacts what policy blocks, and returns only a filtered view. The prompt may explain intent; the API decides what the agent is allowed to see.

Is the policy file part of the prompt?

No. The prompt tells the agent what to try. The policy file is configuration read by the API. The API enforces it even when the model asks for the wrong thing.

Does the agent have an identity?

Yes. The identity should come from a credential owned by the agent runtime, such as a service account, short-lived token, OIDC identity, mTLS certificate, or workload identity. Do not trust the incident, service, or context fields in the request body as facts; verify the caller from the token and the request against systems of record.

Where does this connect to Agent-Ready Infrastructure?

It is the concrete version of the first pattern in Agent-Ready Infrastructure: Runbooks as Governed APIs. It also depends on Operational Ground Truth, because policy can only check facts the platform can read.