An AI Audit Failed Because Nobody Could Show the Model Card: Five Years of Policy Drift
Archi4 min read
Contents
- The failure: a model card that nobody could find
- What the review looked like in 2020
- The specific gap: versioning stopped at the deployment file
- Why this is harder now than five years ago
- More parties, less ownership
- More rules, fewer formats
- Faster retraining, slower archiving
- What the audit failure actually cost
- The comparison in one line
The failure: a model card that nobody could find#
In late March, an independent auditor asked a mid-sized lending platform for the documentation behind its credit-scoring model. The company had a model card. It just could not produce the right version. The card on file described a gradient-boosted tree trained on 2019 applicant data. The model actually serving decisions was a fine-tuned transformer retrained twice in 2025. The audit stalled for 19 days while engineers reconstructed training logs from a decommissioned cloud bucket. The final report flagged the gap as a governance failure, not a technical one.
That sequence would have been unusual five years ago. In 2020, the same company would have been asked for far less. The shift is not that auditors got stricter. It is that the paperwork multiplied while the rules for keeping it coherent stayed vague.
What the review looked like in 2020#
Five years ago, AI audits in financial services were mostly model risk reviews. A validator checked inputs, outputs, and a stability report. Documentation ran to 20 or 30 pages, and the model card was a single slide. The core question was whether the model behaved as intended. Provenance, data lineage, and retraining history were secondary.
Three things changed. First, retraining cycles shortened. A model that once shipped annually now updates monthly. Second, vendors entered the stack. A lending platform in 2026 may run three foundation models from two providers alongside its own scorer. Third, regulators began asking for evidence that the system is understood, not just that it performs. The European Union's AI Act, phased in from 2024, pushed documentation requirements into law for high-risk uses. In the United States, agency guidance and state rules created a patchwork rather than a single standard.
The result is a documentation burden that grew faster than the practices for maintaining it.
The specific gap: versioning stopped at the deployment file#
The company in the audit had version control for code and for the deployment manifest. It did not have version control for the model card itself. When the transformer replaced the tree model, the card was edited in place. The edit overwrote the prior description. Nobody kept a snapshot.
This is a common pattern. Teams treat the model card as a static artifact produced once for sign-off, not as a living record tied to a release. A 2023 survey of AI practitioners by a research consortium found that fewer than half of respondents had a process for updating model documentation when a model was retrained. The number is likely similar today. The tooling exists, but it sits outside the deployment pipeline where it would actually be checked.
Why this is harder now than five years ago#
More parties, less ownership#
A single model in 2020 was built and documented by one team. In 2026, the training data may come from a vendor, the base model from a foundation provider, and the fine-tuning from an internal group that has since been reorganized. The auditor asks who owns the model card. The honest answer is often that nobody does.
More rules, fewer formats#
Regulators and standards bodies have issued overlapping templates. The EU wants technical documentation. A U.S. state may ask for an impact assessment. An internal risk committee wants a model risk report. Each has a different structure. Teams fill them out separately, and the versions drift. There is no accepted crosswalk that maps one to another.
Faster retraining, slower archiving#
When models retrained once a year, a stale card could survive until the next review. Monthly retraining means the card is stale within weeks. Archiving practices have not caught up. Object storage is cheap, but the habit of snapshotting documentation on every release is not widespread.
What the audit failure actually cost#
The direct cost was modest: 19 days of engineering time, a delayed product launch, and a remediation plan. The indirect cost was larger. The company's regulator asked for a follow-up review of all models in production. Two other models had similarly outdated cards. The remediation now includes a documentation pipeline that runs on every deployment and writes a signed snapshot to an audit log.
That fix is not exotic. It is the same pattern used for infrastructure changes: version the artifact, tie it to the release, keep the history. The failure was not a missing tool. It was a missing habit, and the habit was easier to skip when the rules were looser.
The comparison in one line#
Five years ago, an auditor asked whether the model worked. Today, the auditor asks whether you can prove what the model is. Many organizations can still answer the first question. Fewer can answer the second without a scramble.
More articles