Inside a Model Card Audit: What Regulators Actually Check and What They Will Demand Next
Archi4 min read
Contents
- What the Auditor Asked For First
- The Three Documents That Decide the Interview
- The Version Drift Problem
- How a Model Gets Out of Sync With Its Paperwork
- The Sign-Off Question Nobody Prepares For
- What the Next Round of Rules Will Demand
- From Paperwork to Evidence
- The Data Provenance Squeeze
- A Short Checklist From the Field
- What to Build Before a Reviewer Asks
- What Regulators Will Do With the Gaps
What the Auditor Asked For First#
The email arrived on a Tuesday with a subject line that read like a form letter. A mid-sized bank had hired an outside firm to review its AI vendors, and the reviewer wanted the model card, the data sheet, and the last version history. I agreed to sit in on the review as an observer, with the vendor's consent and names withheld.
The first ninety minutes were not about accuracy. The reviewer opened a spreadsheet and asked when the model was last retrained, who signed off, and whether the training data could be traced to specific contracts. The vendor's team had prepared a polished slide deck. The reviewer never looked at it.
That gap, between what vendors prepare and what reviewers request, is where the next two years of AI enforcement will be decided. Draft rules in the US, the EU AI Act's high risk obligations, and procurement checklists from banks and insurers all converge on the same handful of artifacts. Most vendors cannot produce them on demand.
The Three Documents That Decide the Interview#
The reviewer asked for three things in the first hour. The first was a model card with a version number and a date. The second was a data provenance record showing where training data came from and under what license. The third was a change log for the last three versions. The vendor had the first, sort of. The second existed in a legal folder. The third did not exist.
A change log is not a nice-to-have in an audit. It is how a reviewer establishes that the model in production matches the model that was approved. When the vendor admitted there was no formal record, the reviewer moved to a different set of questions, and the tone of the meeting changed.
The Version Drift Problem#
How a Model Gets Out of Sync With Its Paperwork#
We pulled the deployment history from the vendor's internal dashboard. The production model had been updated four times in eight months. The model card on the vendor's website described version 2. The buyer's contract referenced version 3. Production was running something the team called 3.2, which nobody had documented.
This is ordinary in machine learning operations. It is also the kind of fact that turns a routine audit into a regulatory finding. The reviewer marked three items as unresolved and asked for a written explanation within ten business days.
I have now watched four of these reviews. The pattern repeats. Vendors invest in evaluation benchmarks and red team reports. Reviewers spend their time on version control, data lineage, and sign-off authority. The two lists barely overlap.
The Sign-Off Question Nobody Prepares For#
The reviewer asked a question that stopped the room. Who has authority to approve a model for production, and where is that written down?
The vendor's head of engineering said the CTO approves. The head of compliance said the model risk committee approves. The contract said the vendor's board had delegated approval to a named individual who had left the company eleven months earlier. No document reconciled the three answers.
Regulators will care about this more than benchmark scores. A signed approval with a clear owner is the minimum evidence that a governance process exists. Without it, a reviewer has no way to distinguish a controlled deployment from an unsupervised one.
What the Next Round of Rules Will Demand#
From Paperwork to Evidence#
The direction of travel is clear. Early frameworks asked companies to describe their processes. The next generation asks them to produce evidence that the process ran. That means timestamps, named approvers, retained evaluation results, and a log of what changed between versions.
The EU AI Act's obligations for high risk systems already point this way. US sector regulators, particularly in banking and insurance, are following procurement practice rather than waiting for new statutes. When a bank's vendor risk team asks for a change log, that request has the same practical force as a rule.
The Data Provenance Squeeze#
The hardest artifact to produce is the data record. Most vendors can name their datasets. Fewer can show the license terms for each source, the date it was acquired, and whether it was later removed. When a model is retrained, that record has to update too.
I watched a reviewer ask for the license for one specific data source. The vendor produced a contract from a reseller. The reseller had not included the upstream terms. The reviewer wrote it down and moved on, but the item stayed open for the rest of the audit.
Expect regulators to formalize this. The practical requirement will not be perfect provenance. It will be a documented attempt, with gaps disclosed and dated.
A Short Checklist From the Field#
What to Build Before a Reviewer Asks#
, A model card with a version number that matches production, updated on the same schedule as deployments., A change log for every version, with dates, approvers, and a one-line description of what changed., A data provenance record that names each source, its license, and the date it entered or left the training set., A named approver with written authority, reviewed annually and reconciled with contracts., Retained evaluation results tied to the version they tested, not to the model family.
What Regulators Will Do With the Gaps#
The reviewer in my last session closed with a sentence I have heard in different forms all year. The documentation gaps were not treated as dishonesty. They were treated as evidence that the process did not exist. That distinction matters for how the next round of enforcement lands.
The vendors that fare best will be the ones that treat the model card as an operational record, not a marketing page. The ones that struggle will keep producing polished decks for questions nobody asked.
More articles