AI Regulators Shift From Model Rules to Deployment Duties
Archi4 min read
The Next Phase of AI Rules Hits Deployment, Not Training#
Two years of AI policy debate produced definitions, risk tiers, and model evaluations. The next two years will be different. Watch for regulators to shift their attention from what a model can do to what an organization actually does with it. That means procurement rules, logging requirements, and incident response plans will matter more than a model card with benchmark scores.
The first sign is in the EU AI Act. The Act spends many pages on foundation models and general-purpose systems, but enforcement will happen on the ground. The obligations that require human oversight, data governance, and post-market monitoring are the ones that apply to a hospital using a triage tool or a landlord using a credit-scoring system. These are not theoretical duties. They are conditions attached to real products and services.
A similar pattern appears in the United States. The Equal Employment Opportunity Commission and the Consumer Financial Protection Bureau have both signaled that they will apply existing anti-discrimination law to automated decisions. They do not need a new statute. They can issue a request for information, open an investigation, or bring an enforcement action based on the outcomes of an AI system in use. The Federal Trade Commission has already acted against companies that made misleading claims about their AI products. The next cases will be about the companies that deployed those products without a way to detect harm.
Procurement is the lever most people miss. Public agencies buy a lot of software. When a city uses an AI system for traffic enforcement or benefits eligibility, the contract becomes the regulatory tool. Regulators can require vendors to provide audit logs, document override procedures, and demonstrate that the system was tested on the local population. This is not hypothetical. The U.S. Office of Management and Budget has issued guidance on how federal agencies must govern AI acquisitions. State and local governments are following with their own checklists. Procurement officers are becoming the front line of AI policy, and they are stricter than most technology companies expect.
The emphasis on deployment changes the kind of evidence regulators want. Evaluations that report accuracy on a benchmark are useful during development, but they do not show how a system behaves in a specific clinic, school, or call center. Enforcers will ask for the things that show operational control: the threshold that triggers human review, the number of times a human overrode the system, the average time to respond to an incident, and the method for correcting a biased output. These are not new metrics in other industries. Aviation, food safety, and pharmaceutical manufacturing all rely on similar measures. AI regulation is catching up to that standard.
The companies that will feel the pressure first are not the tiny startups. They are large enterprises with existing compliance teams. A hospital network already has a clinical risk department. A bank already has a model risk management group. The new AI rules will fit into those existing structures, which means the compliance burden will be manageable for institutions that have hired for it. The mid-size companies that sell into those sectors will have a harder time. A vendor that supplies an AI module to a hospital will need to provide documentation, update logs, and accept liability clauses in ways that many software companies are not used to.
We expect regulators to converge on a few procedural requirements. The first is a documented chain of accountability: one named person responsible for each AI system in production. The second is a retention rule for logs that cover automated decisions and human interventions. The third is a clearinghouse or database for serious incidents, so that a pattern in one hospital or one city becomes visible to others. The EU Artificial Intelligence Act already includes a serious incident reporting mechanism for high-risk systems. Similar systems exist for medical devices and aviation. Adding AI to that list is a natural extension, and it will happen faster than most observers think, because the machinery for serious incident reporting is already in place.
There will be a split on liability. The EU will move toward producer liability, meaning the vendor who designs and sells the model bears the legal risk. The United States, with its state-by-state approach, will likely put more responsibility on the user and the deployer. A company that buys an AI system to screen job applicants cannot expect a liability waiver in the software contract to protect it from a lawsuit. Courts will look at the actual decision process, not at the indemnification clause. That is a change from how software has been sold for decades. Enterprise software has traditionally been sold with disclaimers: the vendor provides a tool, and the user takes responsibility for outcomes. AI systems that make or suggest decisions will not fit that mold cleanly.
Regulators will also test whether human oversight actually works. A human who clicks a button to approve every recommendation is not meaningful oversight. Enforcers will examine the context of those approvals: How much time did the reviewer have? Did the reviewer have access to the original data or only the system's summary? Was the reviewer trained to spot the failure mode that the system later exhibited? These questions are common in investigations of algorithmic harm. The answers will determine whether a company is found to have complied or to have engaged in what one enforcement action might call rubber-stamping.
The practical advice is simple. Start at the deployment boundary. Write down how the system is monitored, who can stop it, what the fallback is, and how incident reports flow to the people who can fix the root cause. A regulator's first request will be for that documentation. If it does not exist, the regulator will assume the worst. The next two years of AI policy will be defined by that assumption.
More articles