Your OpenAPI spec stopped matching production months ago

Someone needs to integrate with an internal API, so they do the responsible thing and read its spec. The spec for GET /users/{id} says the response has a phone field, and that email is required. They build against exactly that: they render the phone number, and they never check email for null because the contract says it is always there. It works in every test they write, because they wrote the tests from the same spec.

Then it meets production. The phone field was removed six months ago. email has been nullable since a migration in the spring, for the accounts that predate email capture. The integration renders undefined where a phone number should be, and throws the first time it meets a real user without an email. None of this is exotic, and none of it is in the spec, because the spec stopped describing the API two quarters ago and nobody noticed, because nothing made them notice.

The spec was authoritative and wrong, which is the most expensive combination a document can be.

Nothing in the loop makes the spec tell the truth

The structural problem is that the spec and the running API are two separate artefacts, and only one of them is under any pressure to be correct.

The code is what runs. It has tests, it has a deploy that can fail, it has users who complain when it misbehaves. Every one of those is a forcing function that drags the code back toward correctness. The spec has none of them. It is a description that sits beside the code, and when the code changes, nothing fails if the spec is not updated to match. There is no test that goes red because the document now lies. Updating it is an act of discipline, and discipline is not a mechanism.

Worse, drift is invisible while it happens. When the spec and the implementation diverge, the API keeps working perfectly. Traffic flows, dashboards stay green, nobody pages anyone. The only thing that is broken is the description, and a broken description does not announce itself. It waits, silently, until someone trusts it. Drift is monotonic for the same reason: with no force pulling the spec back toward reality and nothing signalling the gap, it only ever grows.

sequenceDiagram
    participant S as Published spec
    participant P as Production API
    participant C as New integrator (human or agent)
    S->>S: written once: phone exists, email is required
    P->>P: phone removed, email made nullable (two quarters of change)
    Note over S,P: nothing fails when the spec and prod disagree
    C->>S: read the spec, build against it
    C->>P: call production
    Note over C,P: passes every test, breaks on the first real record

The tax on anyone who trusts the document

Day to day, the cost is that you learn not to trust the spec, which quietly destroys the reason it exists.

Once you have been burned, the spec stops being the contract and becomes a hypothesis. You read it, then you go and confirm it against the code, or you ask the person who owns the service, or you fire a real request and inspect what comes back. The spec was supposed to save you exactly that work. Now it is a starting guess you have to verify, which is slower than having no spec at all, because at least with no spec you would not have spent an afternoon debugging on the assumption that it was right.

And the failure is quiet in the worst way. “The spec said so” is not a defence anyone accepts, but it is genuinely what happened: you did the diligent thing, read the documentation, and the documentation lied. The lesson people take from this is to stop reading the spec, which means the organisation slowly abandons the one artefact that was meant to let teams work without talking to each other.

The tax on everything built on top of the spec

Step back, and the spec is not just a document a few integrators read. It is the source for a whole chain of derived things, and every one of them inherits the drift.

Your published docs render from it, so the docs are wrong. Generated SDKs are built from it, so the client libraries encode fields that no longer exist. Mocks are generated from it, so teams build against a contract production abandoned. Governance checks run against it, so you are enforcing rules on a fiction. When the spec drifts, it does not fail loudly in one place. It degrades quietly across every surface downstream of it, and each of those surfaces looks fine on its own, which is what makes the rot hard to see and expensive to trace.

For a leader, this is the uncomfortable part: the spec is the coordination layer the org runs on, and its accuracy is nobody’s job in particular. There is no owner, no alert, no metric that moves when it slips. You find out it drifted the way everyone else does, one broken integration at a time.

Why agents turn a slow leak into an acute one

You do not need a statistic to feel the scale. A spec with a couple of hundred operations, each changing a handful of times a year, drifts continuously; without a forcing function, the share of it that is still true declines every quarter. That has always been the case. What is new is who reads it.

A veteran engineer routes around drift without noticing. They know, from tribal memory, that phone is gone and email can be null, so they read the spec loosely and correct it against what they know. That correction is invisible and it is doing a lot of work. An agent has no such memory. It takes the spec at its word, because the spec is precisely the kind of authoritative, structured source an agent is built to trust. Drift that a human silently absorbed becomes an answer the agent delivers with confidence, and it delivers it at machine speed, across every developer using it, before anyone has checked.

flowchart TD
    D["Spec claims phone exists; prod dropped it"] --> W{Who is reading?}
    W -->|Veteran engineer| H["Corrects it from memory, routes around the gap"]
    W -->|New hire or agent| M["Takes the spec at its word, ships the wrong assumption"]

The spec was always drifting. Agents just removed the human who used to quietly patch it on the way past.

Why the usual answers only cover a corner

There are real tools here, and each closes part of the gap.

Code-first generation, springdoc or FastAPI and their kind, produces the spec from the code, so the documented surface cannot drift from the code that generated it. That is a genuine fix for one slice of the problem. Where it stops is that it documents what the annotations and types say, which can still lag actual behaviour, the validation rules, the error responses, the endpoints nobody annotated, and it only helps where it is uniformly applied, which in a polyglot estate with hand-written specs it rarely is. It keeps the map matching one description of the territory, not the territory.

Contract testing verifies a provider against what a consumer expects, in CI, which catches drift on the paths a test exercises and the expectations a consumer bothered to declare. That is valuable and it is partial: it is as complete as your test coverage and your list of declared consumers, and it says nothing about the operations nobody wrote a pact for.

Gateways and observability see the territory directly. They know what production actually does, in real traffic. What they do not do is compare that to the spec, so they can tell you what is happening and never that it disagrees with what was promised.

The gap none of them fills is the continuous comparison of the published contract against the running API, at the level of the operation, with the divergence surfaced to the person who owns it.

What would have to be true

Argue it from the properties.

The spec needs a forcing function tied to change. Either it is generated from the source of truth so it cannot lag, or it is continuously checked against reality so that lag becomes visible. Discipline is not a forcing function, and anything that relies on someone remembering to update a document will drift, because the whole failure mode is that people do not see the gap.

Drift has to be made visible. Something has to alert, or ideally break, when the spec and the runtime disagree, because silence is why drift is permanent. A gap that nothing reports is a gap nobody closes.

The comparison has to be between the published contract and the actual behaviour of the running API, at operation granularity, so that “this endpoint’s response no longer matches its schema” is a specific, routable finding and not a vague sense that the docs are stale.

And it has to be honest that some drift is semantic, not structural. A response can match its schema perfectly and still behave differently than the contract implies. Shape checks will not catch that, and pretending they do is its own quiet lie.

graph LR
    SPEC["Published spec (the map)"]
    RUN["Running API behaviour (the territory)"]
    CHK{"Compare, per operation"}
    SPEC --> CHK
    RUN --> CHK
    CHK -->|agree| OK["Spec is trustworthy"]
    CHK -->|diverge| FLAG["Flag drift to the owner"]

How we come at it

Full disclosure: this is the spec0 blog, so read this as interested rather than neutral, and read it carefully, because this is a place I have to be precise about what exists.

What is real today is the foundation, not the finish. spec0 keeps your published specs in a versioned registry and detects breaking changes between published versions, so the intentional changes, the ones you ship as a new version, are compared, gated and recorded rather than lost. That makes the published contract a first-class, versioned artefact instead of a file that quietly rots in a repo.

What is not real today is the part this post is actually about. spec0 does not yet compare the published spec against the running API and detect drift between the map and the territory. That is direction, not a feature, and I would rather say so plainly than let a good problem statement imply a solution we have not shipped. The reason the registry matters here is that you cannot detect drift from a contract you do not have under version control in the first place, so the versioned registry is the ground that runtime drift detection would eventually stand on. It is groundwork, described as groundwork.

The part worth arguing about

The hard question is where the forcing function should live, because both honest answers have a failure mode. Generate the spec from the code and it will match the code, but the code is not the same as the behaviour, and not every service can generate. Compare the spec to real traffic and you catch behavioural drift, but now you need a faithful sample of production, and you have to distinguish an intended change that has not been published yet from an actual bug, or you will cry drift at every deploy and train everyone to ignore the alarm.

So the thing I keep turning over is the false-positive cost. A drift detector that is too eager is just a new source of noise, and noise is how good signals die. What is the threshold at which “the spec and production disagree” is worth interrupting a human over, and who gets to set it? If you have run something that watched for drift and stayed quiet enough to be trusted, I would like to know what you compared, and what you decided not to.