A postmortem in February exposed something we had not previously audited across our estate. A contract review pipeline — not one we had built, but one we had taken on for production management seven months prior — had been returning stable precision and recall metrics for the entire period. Then, across a five-week window beginning in mid-January, precision on complex multi-clause indemnity provisions fell from 88.4 per cent to 74.8 per cent. There was no change in the system's prompt. No deployment event. No material shift in the input corpus. The engineering log was clean.
We found the cause in the fourth hour of the investigation. The pipeline was calling the API against a named model alias that the provider had updated twice in the relevant period — changes released as non-breaking, announced in a developer forum post that nobody in the client organisation had subscribed to. One of the updates had modified the model's instruction-following behaviour for complex multi-step extraction tasks. The system prompt contained exactly that pattern. The precision regression was a consequence of a change that had happened outside the system entirely, on a schedule the team had no visibility of, with a log entry that did not exist in any system they controlled.
We fixed it within seventy-two hours by pinning to a snapshot version the provider supported, running our evaluation against it, and confirming the behaviour had returned to baseline. What we could not fix, or rule out, was that the original system had been delivered without a model version management protocol. The incident was traceable to a provider update. It was also traceable to an architectural gap that nobody had named during delivery. This piece is about that gap.
#02What providers actually update when they update a named version
The versioning conventions of model providers are not the same as software versioning conventions for libraries and frameworks. A new model family — a major release in the provider's product line — ships as a distinct named product with its own evaluation profiles and announced capability changes. The stable aliases that most enterprise deployments call — the integration targets intended to give teams a consistent API surface without requiring updates on every model improvement — are updated on the provider's schedule, not the consumer's. The alias points to the provider's current best version of the capability. What constitutes 'current' is the provider's decision.
The changes that arrive under a stable alias are, from the provider's perspective, improvements. They typically are improvements on the benchmarks the provider tracks internally. They are not evaluated against the prompts in your production system, against your input distribution, or against your evaluation criteria. A shift in instruction-following sensitivity that improves the model's aggregate benchmark score may degrade a pipeline whose behaviour depends on the model's prior handling of ambiguous sequential instructions. The improvement and the regression are not in conflict — they are simultaneously true. Only one of them appears in the provider's release notes.
We have now audited version history across fourteen active production engagements. In eleven of the fourteen, the model version being called today is not the one called at launch — the underlying weights have been updated, under an alias that presented as unchanged. In six of the eleven, no engineer on the team responsible for the deployment has reviewed the update. Not because anyone was negligent: model version management has not been established as a named engineering discipline for most production AI teams. What is not named is not owned.
#03The four gaps that appear in production
The gaps follow a consistent pattern across the systems we manage and the ones we have inherited. The first is the absence of a pinning policy. Of the fourteen deployments: three have explicitly pinned to a provider-supported snapshot version, with a documented policy for when the pin will be reviewed and advanced. Eleven are calling aliases that can be updated by the provider without notice. None of the eleven have a stated policy for when they will evaluate a provider update, or who has authority to approve the update entering production. The evaluation decision is being made for them, on the provider's schedule, by a process they do not control.
The second gap is monitoring. Provider release notes are not a regulated document. The channel on which they appear — a changelog, a developer forum post, a product blog — varies by provider and is not guaranteed to be consistent or complete. In twelve of the fourteen deployments, nobody is subscribed to any provider release notification. In none of them has provider release monitoring been added to a standard operations checklist. The update that caused the February incident was announced eleven days before we traced it. Eleven days in which a subscription to the right channel would have triggered an evaluation run and, with reasonable probability, caught the regression before it was visible in production metrics.
The third gap is evaluation triggering. In mature software deployments, a dependency update triggers a regression test suite before the dependency enters production. In twelve of the fourteen AI systems we manage, a provider model update does not trigger any evaluation. The evaluation framework exists in all twelve — it runs on a fixed schedule against a fixed corpus. What does not exist is the trigger: the mechanism that connects a provider-side change to a re-run of the evaluation before the change affects production behaviour. The evaluation and the update lifecycle are not connected.
The fourth gap, and the hardest to retrofit, is rollback state. Software deployments have rollback procedures. A provider-side model update cannot typically be reversed by the consuming team — the alias points to the provider's current version. Where rollback is available, it means pinning to a historical snapshot the provider has agreed to support for a finite window. In three of our current deployments, the team is operating against model versions the provider has not formally committed to beyond an unspecified near-term horizon. In two of those three, the team is unaware of this. The deprecation event will not arrive as a warning — it will arrive as a breaking change if the team has not moved to a supported version before the sunset date. Enterprise AI teams are accustomed to thinking of model provider changes as improvements. They are not accustomed to thinking of them as a deprecation schedule that requires active management.
“The improvement and the regression coexist. A 'non-breaking' update that modifies instruction-following behaviour is a breaking change on a different axis — one the provider's definition of 'breaking' does not capture, because the provider is not in a position to evaluate it against your production configuration.”
#04The minimum viable practice
The practice that addresses these four gaps is not technically complex. It is organisationally complex, because it requires someone to own a responsibility that has not previously existed as a named role in most AI delivery teams.
The first requirement is a monitoring subscription — a named person, or an automated process, subscribed to the provider's release communications, responsible for reviewing each update against a documented list of the system's known sensitivities. The review does not need to be exhaustive. It needs to answer one question: does this update touch the model's instruction-following behaviour, refusal thresholds, or output format defaults in ways that could affect this system's production behaviour? If the answer is unknown, the update triggers an evaluation run. If the answer is clearly no, the change is logged and closed. In our experience this takes between fifteen minutes and an hour per update, depending on the update's scope and the team's familiarity with the system's evaluation profile.
The second requirement is a trigger mechanism connecting provider updates to evaluation runs. The evaluation that was run at launch needs to be runnable on demand, against any model version the provider currently supports. When a provider update is flagged as potentially relevant, the evaluation runs against the updated alias and the most recent approved version as a baseline. A regression above a threshold defined at design time holds the update — the team investigates before the updated version enters production. In four deployments where we have implemented this, the evaluation runs take between nine and thirty-one minutes. The trigger has been fired eleven times across the four systems. Three times, the evaluation identified a regression that would otherwise have been visible first in production metrics.
The third requirement is an explicit pinning decision. Every production AI system should have a documented statement of whether it is tracking the provider's latest version or pinned to a specific snapshot, and if pinned, what the policy is for advancing the pin. Both choices carry risk — tracking the latest version means absorbing changes as they arrive; pinning means accepting a version that may not receive future improvements, and running the risk of deprecation if the snapshot approaches end-of-life without review. The choice should reflect the system's risk profile and the provider's versioning commitments. What is not acceptable, as a production discipline, is the default: having made neither choice, calling a mutable alias, and discovering after an incident that the decision was made by the provider on the provider's schedule.
#05What this is not
This is not an argument for self-hosted models as a substitute for managed APIs. Self-hosted model management solves some of what I have described above and introduces a different set of operational concerns — hardware costs, upgrade cycles, security patching, inference infrastructure maintenance. For most enterprise deployments, managed APIs remain the right choice. The argument is narrower: that using a managed API responsibly requires treating the provider's model update cadence as an operational dependency, subject to the same monitoring and change management practices as any other external dependency in a production system.
Nor is it a criticism of provider versioning practices. The stable alias model is a reasonable design for the majority of API consumers, for whom receiving the provider's latest improvements automatically is a net benefit. The problem is that production enterprise AI systems are not typical API consumers. They have specific evaluation profiles, specific prompt designs, and specific output format dependencies that were assembled and tuned against a particular model behaviour. When that behaviour changes, even for the better in aggregate, the specific system may not benefit — and has no mechanism to know.
The operational practice I am describing is not a new concept. It is software dependency management, applied to a dependency that most teams have not recognised as one. The model has been treated as a service: reliable, improving, managed by the provider, expected to be stable without requiring the consumer to manage its version behaviour. For production enterprise AI systems whose correct operation depends on specific model behaviours, it is more accurate to treat the model as a versioned dependency with a changelog that requires active review and a deprecation schedule that requires active tracking. The February incident took four hours to diagnose and seventy-two hours to resolve. The monitoring subscription and trigger mechanism that would have caught it in evaluation take roughly two days to implement at delivery time. We have started implementing them that way.