AI Policy Myths vs Facts: The Decision That Cost a Company 1,200 Users

Archi4 min read

ai-policycompliancehealth-tech

Contents
  1. The Decision
  2. Myth 1: The EU AI Act bans AI in health care
  3. Myth 2: Documentation is paperwork
  4. Myth 3: A disclaimer solves the transparency problem
  5. Myth 4: Only large companies face enforcement
  6. The Rebuild
  7. What the Decision Teaches

The Decision#

In March 2025, a mid-sized health technology company in Berlin had a choice: keep its AI triage tool running for 1,200 users across three hospitals, or shut it down for six weeks while engineers rebuilt the model's documentation and oversight process.

The tool analyzed symptoms and suggested appointment urgency for general practitioners. It was not a medical device. It did not diagnose. But it did sort patients, and that sorting used personal health data.

The company's legal team flagged a provision in the EU AI Act that classified certain health related uses as high risk. The product team argued the tool was low risk. They were both partly right, and that ambiguity is where the real decision lived.

Here is what happened, told as a myth versus fact breakdown, because the gap between what companies believe about AI policy and what regulators actually wrote is where the expensive mistakes hide.

Myth 1: The EU AI Act bans AI in health care#

Fact: The act does not ban AI in health care. It bans specific practices, such as social scoring and real time remote biometric identification in public spaces, with narrow exceptions. For health applications, the act creates obligations based on risk tier.

A symptom triage tool that influences clinical workflow can fall under Annex III as a safety component of a medical device, or as a system used for medical purposes. If the company's tool merely routed appointment requests without clinical judgment, it may have landed in the limited risk category, which requires transparency, not a full conformity assessment.

The company's mistake was treating the act as a binary: compliant or illegal. The actual text asks: what is the intended purpose, and what is the risk to health or fundamental rights?

Myth 2: Documentation is paperwork#

Fact: The act's high risk requirements are engineering requirements wearing legal language. Article 11 asks for technical documentation. Article 15 asks for accuracy, robustness, and cybersecurity. Those demands force decisions about data governance, logging, and human oversight that a product team cannot retroactively bolt on.

The firm discovered that its model's training data came from four hospital partners with four different consent forms. None of the forms explicitly covered AI driven triage. Rebuilding consent language took longer than rebuilding the model.

By the time legal and engineering aligned, the six week shutdown had grown to nine.

Myth 3: A disclaimer solves the transparency problem#

Fact: Transparency under the act is not a banner. For certain systems, including those that interact with people, Article 50 requires that users know they are dealing with an AI system. For high risk systems, instructions for use must cover intended purpose, accuracy metrics, and known limitations.

A generic footer saying "this tool is AI assisted" does not meet that bar. The company needed to tell clinicians what the model was trained on, how it handled edge cases such as pediatric patients, and what the false negative rate looked like in its validation set.

That last number was not available. The model had been validated on internal data, not on hospital data. The team had to run a fresh evaluation, which is why the downtime stretched.

Myth 4: Only large companies face enforcement#

Fact: The act applies to providers, deployers, importers, and distributors regardless of size. Small and mid sized firms get some relief on administrative fees and can use regulatory sandboxes, but the core obligations do not vanish.

This company employed 140 people. It assumed enforcement would target large platform companies first. That assumption was reasonable but risky: a hospital customer can be a deployer, and deployers have duties too. When one hospital's compliance officer asked for documentation in February, the company had 30 days to produce it or lose the contract.

The Rebuild#

The company chose to shut the tool down and rebuild. Engineers created a model card, a data provenance log, and a human oversight protocol that required a clinician to review every triage suggestion above a certain urgency threshold.

They also separated the model from the appointment scheduling system, so that a failure in one would not cascade into the other. That architectural change was not required by the text, but it made the conformity assessment simpler.

When the tool returned, 340 of the original 1,200 users had moved to a competitor. The company kept the hospital contracts, which were worth more than the lost users, but the episode left a mark.

What the Decision Teaches#

The myth versus fact gap is not about reading comprehension. It is about incentives. Product teams want to ship. Legal teams want to avoid liability. Policy text sits between them, and it rarely says what either side wants it to say.

The company's chief technology officer now runs a quarterly exercise: pick one AI feature, read the relevant annex, and write down the intended purpose in one sentence. If the sentence requires a comma splice, the feature probably needs a risk assessment.

That is cheaper than nine weeks of downtime. It is also more honest than a disclaimer nobody reads.

More articles