Who Is on the Hook When AI Gets It Wrong?

Picture a busy clinic. A doctor uses an AI tool to help with a patient. The tool gets it wrong. The patient is hurt. Who answers for that?

The FDA asked a version of this question in August 2026. It asked the public how to share the job of watching AI devices after they launch. More than 380 people and groups wrote back. This article looks at what they said about duty, blame, and trust.

What the FDA asked

The FDA said it is the manufacturer’s job to monitor its devices. But it also noted that many others play a part. They include doctors, patients, hospitals, insurers, and standards groups.

Then the FDA asked a tricky question. How can those groups help, without letting the manufacturer off the hook?

Here is a made-up example. A hospital uses an AI tool that drafts advice for patients. After an update, the tool starts to under-warn about chest pain. Each hospital sees only a few cases. The manufacturer sees the trend across ten hospitals within days. Who is best placed to catch it?

Most agreed: the manufacturer stays responsible

This was the clearest answer in the comments. Again and again, groups said the company that makes the device must stay accountable. That word means "answerable." If something goes wrong, this is who has to answer.

• The American Hospital Association asked the FDA to give manufacturers the main job of checking devices, both before and after launch.
• The Radiological Society of North America said manufacturers must remain accountable for safety and effectiveness.
• The American Psychiatric Association gave a reason: the manufacturer controls the design and the updates. It can see how the tool does at many sites. So it is best placed to spot and fix problems.
• The American College of Surgeons said the job of building the evidence rests with the manufacturer.
• The Permanente Federation said shared roles should not water down the manufacturer's accountability.

Why start with the manufacturer?

Think about who can see what. A doctor sees one case at a time. A hospital sees only its own patients. The manufacturer sees how the tool does across many sites. The manufacturer also builds the tool and ships the updates. When a problem shows up, only the manufacturer can fix the tool itself.

That does not mean no one else helps. The Coalition for Health AI said a device should also be checked in each local setting, by independent parties. And doctors can report what they see. But the manufacturer leads.

One health-system analyst warned that most hospitals today cannot watch AI tools well on their own. They lack the data, the staff, and the tools. That fits the view that the manufacturer should carry the responsibility.

But what about the AI model underneath?

Many devices are built on an AI model made by another company. That makes the question messier. Three views stood out.

The Chamber's view. The U.S. Chamber of Commerce said manufacturers may not be able to stop, or even fully understand, updates to a model they did not build. Still, manufacturers remain responsible for the finished device.

Google's view. Google said responsibility fits best with the company that sets the device's intended use. That is the app manufacturer. It is not the company that supplies the base model.

Microsoft's view. Microsoft said responsibility may be spread across many players. They include manufacturers, doctors, hospitals, platforms, and model providers. Each plays a different part.

These views differ. But they agree on one thing. Someone must be named.

People still matter

Doctors said they do not want to be left holding the bag.

The American Academy of Family Physicians said AI should help doctors, not replace them. Doctors should keep responsibility for care choices. Mayo Clinic said much the same. The College of American Pathologists said both things are needed: pathologist oversight and manufacturer accountability.

One pathologist put it plainly. Will doctors be asked to sign off on results they cannot inspect?

Having a person in the loop is not the same as having a safe tool. One AI developer said a human check only counts as a safety step if you measure how often the person catches mistakes. If reviewers miss most errors, the check is weaker than it looks.

Another worry runs the other way. Busy people may trust a smooth, sure-sounding answer too much. About one in three letters from medical societies, hospitals, and universities warned about this.

The American Medical Association raised a related worry. Watching devices after launch is useful. But it should not make patients and doctors the ones who must find hidden risks that were never well tested.

The gap that worries people

Johns Hopkins Medicine warned about a quiet shift. When the rules for manufacturers are unclear, the duty can slide to the hospitals that use the tools. Liability slides with it. That also makes hospitals less willing to adopt new AI.

The Coalition for Health AI passed on a question from its members. If a patient-facing AI tool causes harm, and no clinician was involved, who is responsible?

A law group that represents people it says were harmed by chatbots made a related point. A small warning label should not cancel how a tool is really used.

Some groups are not waiting. The Joint Commission, which accredits hospitals, now offers a voluntary certification for responsible use of AI in health care.

Different groups, same worry

Medical societies, hospitals, and universities raised accountability often. About seven in ten of their letters did. So did about two in three letters from industry groups. The two sides often name the same fix. Clear roles, written down, before anything goes wrong.

What this means for regulatory teams

Nothing here is a rule yet. This is our reading of the comments, not FDA guidance. But three habits seem wise:

• Write down who owns each decision. Name the manufacturer, the hospital, and the clinician for each step.
• Keep three things apart. What the source said. What the AI suggested. What a person decided.
• Say what the tool is for, in plain words. A clear intended use makes it clear who is answerable.

When it is unclear who answers, the person closest to the patient often ends up answering. A clear record protects everyone.

At Bevrik, we are building software to help medical-device teams organize evidence and trace each finding back to its source. It is designed to keep source facts, AI suggestions, and human decisions apart. People stay in charge of every decision.

Less paperwork. More confidence.

References

1. FDA Discussion Paper: Considerations for the Regulation of Generative AI-Enabled Medical Devices https://www.fda.gov/media/185836/download
2. FDA Announcement Requesting Public Feedback on Generative AI Medical Devices https://www.fda.gov/medical-devices/digital-health-center-excellence/fda-seeks-public-feedback-considerations-regulation-generative-ai-enabled-medical-devices
3. FDA Docket FDA-2026-N-7874 (Public Comments) https://www.regulations.gov/docket/FDA-2026-N-7874
4. FDA Artificial Intelligence-Enabled Medical Devices Resource Page https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices