AI Audit Rules Compared: The 0.02% Error Rate That Reshapes Compliance
Archi4 min read
Contents
The number that decides who pays#
A 0.02% error rate sounds like nothing. In a test set of 500,000 decisions, it is 100 mistakes. In a credit scoring model or a resume screener, 100 mistakes can become 100 complaints, and complaints become the paper trail regulators read first.
That single threshold sits at the center of two competing AI audit proposals now circulating in Brussels and Washington. One would require outside audits for any model that touches housing, credit, employment, or health, with a hard ceiling on error rates as measured by a fixed test. The other would let companies run their own tests and publish a summary, with the regulator stepping in only after a pattern of complaints. The difference in cost is not 2%. It is closer to 20x.
Option A: The fixed threshold#
Under the fixed threshold approach, a company must show that its model stays below 0.02% on a standardized evaluation set before deployment, and again every six months. The test set is public. The scoring code is public. The audit is performed by an accredited third party, and the results go into a registry that regulators and, in redacted form, the public can read.
The appeal is comparability. A bank, a landlord, and a hiring platform all get measured with the same ruler. If a model drifts above the line, the company has 90 days to fix it or pull it.
The bill is real. Early estimates from two consultancies put first-year compliance at $400,000 to $1.2 million for a mid-sized model developer, mostly for test infrastructure, outside audit fees, and the staff time to rebuild pipelines so they can produce audit-ready logs. For a startup with 30 employees, that is a round of funding.
Option B: Self-assessment with disclosure#
The second proposal drops the fixed threshold. Companies test their own models, publish a summary of methodology and results, and keep full records for inspection. Regulators open an investigation when complaints cross a volume trigger, not when a single test set shows a bad number.
The cost is lower, perhaps $40,000 to $120,000 in the first year for the same mid-sized developer, according to the same estimates. The tradeoff is that the number stops meaning the same thing across companies. One firm's 0.01% may be measured on a narrow population. Another firm's 0.01% may hide a much worse rate for a subgroup that the summary does not break out.
What 0.02% actually buys#
Here is the surprising part. At 0.02%, a model making 10 million decisions a year produces 2,000 errors. If those errors cluster in one zip code or one age band, the aggregate number looks fine while the harm is concentrated. Both proposals acknowledge this. Only the fixed threshold forces companies to publish subgroup error rates alongside the total.
That disclosure is where the real fight is. Industry groups argue that publishing subgroup numbers creates a roadmap for litigation. Consumer advocates argue that without them, the headline number is theater.
A third path: tiered audits#
A smaller camp is pushing a tiered model. Companies under a revenue threshold self-assess. Companies above it get an outside audit every two years, and the 0.02% line applies only to models that make final decisions without human review. A human in the loop resets the clock.
The tiered approach costs less than Option A and more than Option B, roughly $150,000 to $300,000 in year one for a mid-sized developer. It also creates a new incentive: keep a human reviewer in the pipeline, and the strictest test does not apply. Whether that reviewer is doing meaningful work or rubber-stamping is the question regulators have not answered.
The timeline nobody budgeted for#
Both main proposals include a 24-month transition. The fixed threshold proposal compresses that to 12 months for models already in production. That is the clause that has compliance teams worried. Rebuilding a logging pipeline takes months. Retraining a model to pass a new test set takes longer. A 12-month clock on a model that has been running for three years is a deadline that will produce either rushed patches or quiet withdrawals from sensitive markets.
Companies that sell into multiple jurisdictions face a second problem. The US proposal does not recognize the EU audit, and the EU proposal does not recognize the US one. A firm selling credit models on both sides of the Atlantic could pay for two audits with different test sets and different thresholds.
What to watch#
The 0.02% number will move. It is a negotiating position as much as a technical standard. The more important question is whether subgroup disclosure survives the final text. If it does, the fixed threshold becomes a genuine tool. If it does not, companies will spend millions to prove a number that hides the thing regulators care about.
For now, the practical advice is dull. Start logging decisions at the subgroup level. Keep six months of records. Do not assume the self-assessment path will stay cheap, because the complaint trigger is undefined in both drafts, and undefined triggers have a way of expanding.
More articles