AI Data Drift: When the Hospital Changes Scanners but the AI Stays the Same

1. The problem in practice

Imagine a hospital replaces one of its CT scanners. The installation goes smoothly. Images are flowing into PACS, radiologists are reviewing scans, and the connected AI software is processing images without a problem. Everyone is happy. The upgrade is complete.

But there's something nobody has noticed. The new CT scanner uses slightly different image reconstruction settings. Maybe the images have different noise characteristics, or the field of view has changed. Nothing has changed in the AI software itself. It's still running the same model that was tested and validated.

But the images going into that model are no longer quite the same.

This is what's known as data drift, or covariate shift. And here's what makes it particularly concerning. The AI might process every image successfully. No crashes. No error messages. No failed requests. Your software monitoring dashboard looks perfectly healthy. Yet the model could be getting worse at detecting the very things it was designed to find. And you might not discover that until much later, when enough clinical evidence becomes available to show that something has changed.

This is the part of AI post-market surveillance that deserves more attention.

We are typically using PMS/PMCF plans to monitoring complaints, software defects, adverse events, and device performance. But for AI medical devices, we also need to consider whether the data entering the model still represents the conditions under which it was validated. And this can become a much better early indicator.

That means thinking beyond traditional software monitoring and establishing controls to detect meaningful changes in real-world input data. Because a system that is running correctly isn't necessarily a system that is performing correctly. And I'd much rather investigate a change in input data today than discover a possible loss of clinical performance six months from now.

2. The regulatory thread

EU MDR Annex III requires manufacturers to collect and assess post-market information and establish suitable indicators and thresholds for continuously reviewing device safety, performance, and benefit-risk.

EN ISO 14971:2019, Clause 10 connects production and post-production information to the risk management process, ensuring that new hazards, changing risks, and emerging performance concerns are evaluated throughout the device lifecycle.

EN 18286:2026 introduces AI-focused quality management provisions, including data management and post-market monitoring. These provide a framework for identifying changes in real-world input data and assessing whether they may affect AI system performance.

For applicable high-risk AI systems, Article 17 of the EU AI Act allows the AI quality management requirements to be integrated into an existing sectoral QMS. Rather than creating separate AI procedures, medical device manufacturers can extend their existing PMS SOPs, risk management procedures, and software maintenance processes with AI-specific controls for data drift, performance monitoring, and escalation.

United States: 21 CFR § 820.10, under FDA's Quality Management System Regulation (QMSR), incorporates ISO 13485:2016 into the FDA's medical device quality requirements. In particular, ISO 13485 Clauses 8.2.1 (Feedback), 8.4 (Analysis of Data), and 8.5.2 (Corrective Action) provide a framework for collecting real-world performance information, identifying trends, and responding to emerging problems.

The FDA is also actively researching methods to detect changes in AI model inputs and outputs and evaluate real-world performance.

For manufacturers selling into both markets, the practical opportunity is to use one controlled, risk-based AI monitoring process that generates evidence applicable to both regulatory frameworks, while addressing their respective requirements.

3. A worked example

Now back to our example. How an upstream scanner change becomes a clinical concern:

1. Day 0: The hospital upgrades its scanner - A CT scanner receives a new image reconstruction algorithm. Image texture and noise change, but the AI model remains unchanged.
2. Day 7: Input monitoring detects a shift - The manufacturer detects that an unusually large proportion of images now differ from the validated image profile. This is evidence of data drift, not necessarily reduced accuracy.
3. Day 30: AI output trends become concerning - The AI is flagging fewer scans as suspicious. Is this because the patients are different, the images changed, or the AI is missing abnormalities?
4. Day 30–90: Clinical accuracy becomes measurable from assessment of AI data - With enough reviewed scans and reliable reference diagnoses, the manufacturer may estimate whether sensitivity and specificity have changed. The timing depends on the availability of reference data and adequate sample sizes.
5. Months or years later: Patient outcome data emerges from typical PMS activities - Some missed cancers may only become apparent during later imaging, diagnosis, treatment, or follow-up. By then, patients may already have been affected.

The important implication is that monitoring patient outcomes alone may provide a delayed safety signal. But there is another critical nuance: clinical diagnostic accuracy and long-term patient outcomes are not the same measurement.

A radiologist's confirmed finding or a pathology result might allow relatively prompt evaluation of AI accuracy. Waiting for survival, cancer recurrence, or treatment outcomes could take much longer.

That difference matters when designing a PMS or PMCF program. A rise in out-of-distribution images can trigger investigation before enough clinical outcomes are available. Monitoring and identifying this early is essential for improving overall health outcomes for patients.

4. What should appear in the documentation

Within your QMS, you should be able to trace the issue from the PMS Plan to the original validation dataset description and intended-use boundaries. Useful records include the model and scanner versions; the validated input profile; the method for measuring shift; the chosen alert threshold and rationale; the latest monitoring log; the clinical performance investigation; and any risk or change-control decision.

A vague sentence such as 'AI performance is monitored regularly' provides little help. It does not say:

• what is measured,
• where the evidence comes from,
• who can interpret it, or
• what happens after an alert.

Check whether the monitoring plan covers meaningful subgroups and sites, but avoid collecting patient information simply because more data is available. Privacy, representativeness, rights, and data access still matter.

Level 1: Input data drift --> Earliest signal

Has the information going into the AI changed?

Examples: CT image characteristics, scanner manufacturer, reconstruction software, patient demographics, or imaging protocols.

These changes can sometimes be detected immediately, without knowing the correct diagnosis.

Level 2: AI output monitoring --> Has the AI started producing different results?

Examples: fewer suspicious lesions detected, changes in confidence scores, more rejected images, or a shift in the percentage of scans flagged as abnormal.

These can also be detected quickly, but they don't necessarily reveal whether the AI is right or wrong.

Level 3: Clinical performance and outcomes

Is the AI still clinically accurate, and are patients receiving the intended benefit?

Examples: sensitivity against confirmed diagnoses, false negatives, missed cancers, delayed treatment, or longer-term patient outcomes.

These often require reliable reference diagnoses, clinical follow-up, adequate sample sizes, and sometimes months or years of observation.

5. Implement it in your existing MDR QMS

Suggestions for update the existing PMS SOP, not a brand-new AI department manual.

1. Create a Data Drift Log for monitoring. Add fields for expected input characteristics, data source, check frequency, alert rule, sample size, owner, and clinical review criteria. Link it to your risk procedure when the signal could affect hazardous situations, and to the software maintenance/change SOP if a configuration or model update is proposed.
2. Give Regulatory Affairs ownership of the PMS plan and periodic review; engineering supplies input-monitoring data; clinical affairs interprets potential patient impact; Quality owns any nonconformity or CAPA. The exact owners may vary, but someone must have clear accountability.
3. Keep two decisions distinct in the Data Drift log: 'input changed' and 'device performance affected.' Closing the first without investigating the second can leave the most important question unanswered.

6. Free resource: use this now

The accompanying Data Drift Log contains a fictional completed row and blank rows to adapt. Start by naming one input characteristic that was important during validation, such as scanner type or image protocol. Enter where you obtain that information, how often it is checked, and what would prompt clinical review.

Then test the process with one mock shift. Can you locate the validation basis, produce the monitoring calculation, open an investigation, document the patient-risk decision, and explain who approved closure? If that evidence trail takes hours to reconstruct, the procedure may need work before launch.

The template is an example, not a plug-and-play clinical acceptance criterion. Use device-specific statistical methods, qualified review, and documented risk justification.

https://bevrik.net/resources -> Data Drift Monitoring Template

References and review notes

1. European Commission - harmonized AI standards: https://single-market-economy.ec.europa.eu/single-market/goods/european-standards/harmonised-standards/artificial-intelligence-ai_en
2. Official Journal - Implementing Decision (EU) 2026/2274: https://eur-lex.europa.eu/eli/dec_impl/2026/2274/oj
3. EU AI Act - consolidated legal text: https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng
4. EU AI Act - Article 17 QMS: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-17
5. EU AI Act - Article 43 conformity assessment: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-43
6. EU MDR 2017/745 - current legal text: https://eur-lex.europa.eu/eli/reg/2017/745/oj
7. EN 18286:2026 catalogue/contents preview: https://www.iteh.eu/catalog/standards/cen/34ea911c-a980-4433-85ac-1344f93da01b/en-18286-2026
8. FDA QMSR - 21 CFR Part 820: https://www.ecfr.gov/current/title-21/chapter-I/subchapter-H/part-820
9. FDA QMSR - 21 CFR § 820.10, Requirements for a Quality Management System: https://www.ecfr.gov/current/title-21/chapter-I/subchapter-H/part-820/section-820.10
10. FDA - Quality Management System Regulation (QMSR), including ISO 13485:2016 incorporation: https://www.fda.gov/medical-devices/postmarket-requirements-devices/quality-management-system-regulation-qmsr
11. FDA - Methods and Tools for Effective Postmarket Monitoring of AI-Enabled Medical Devices: https://www.fda.gov/medical-devices/medical-device-regulatory-science-research-programs-conducted-osel/methods-and-tools-effective-postmarket-monitoring-artificial-intelligence-ai-enabled-medical-devices
12. FDA - Measuring and Evaluating AI-Enabled Medical Device Performance in the Real World (Request for Public Comment): https://www.fda.gov/medical-devices/digital-health-center-excellence/request-public-comment-measuring-and-evaluating-artificial-intelligence-enabled-medical-device