Building a Clinical AI Model: Data Governance, PHI Handling, and FDA SaMD Classification

A clinical AI model that works well in a demo is not the same as one that is legally allowed to influence patient care. The gap between the two is regulatory, and it is wider than most engineering teams expect.

The FDA has already cleared over 1,000 AI and machine learning enabled medical devices as of its most recent published list, most of them in radiology and cardiology. Every one of those clearances required proof of data governance, PHI handling controls, and a defined risk classification long before the model reached a hospital.

For medical device manufacturers, CROs, and life sciences teams building clinical AI, this guide breaks down what regulators and technical reviewers actually check, and where most teams create gaps without realizing it.

When Does an AI Model Become a Medical Device?

Not every clinical AI tool is regulated the same way. The FDA's Software as a Medical Device (SaMD) framework applies when software is intended to diagnose, treat, monitor, or manage a disease or condition, independent of any hardware.

A model crosses into SaMD territory when it:

• Analyzes clinical or diagnostic data to inform a treatment or diagnostic decision
• Is marketed or intended for use in patient care rather than research only
• Produces an output a clinician or patient could act on directly

A model built purely for internal research, with no clinical decision impact, generally sits outside SaMD scope. The line gets blurry fast though, especially with generative or predictive tools, which is why large language models used in clinical settings often need a formal regulatory review even when teams assume they are "just a support tool."

FDA SaMD Risk Classification

The FDA and the International Medical Device Regulators Forum (IMDRF) classify SaMD based on the severity of the medical condition and the significance of the information the software provides to a healthcare decision.

Higher classification means more rigorous premarket requirements, including clinical validation data and, in many cases, a full 510(k) or De Novo submission. Getting this classification wrong early, either overestimating or underestimating it, tends to surface exactly when investors run technical due diligence for Series A healthcare funding.

Data Governance Foundations for Clinical AI

Before a single model is trained, the data pipeline behind it needs structure. Regulators and hospital partners look for specific answers here, not general assurances.

1. Data provenance and lineage

Every dataset used for training or validation needs a documented source, collection method, and consent basis. If you cannot trace where a data point came from, you cannot defend the model's output.

2. Dataset representativeness

FDA guidance increasingly asks for demographic and clinical diversity in training data to avoid biased outcomes across patient populations.

3. Version control for datasets and models

Every model version needs to map back to the exact dataset version it was trained on. This matters for audits and for FDA's predetermined change control plan (PCCP) framework for adaptive AI.

4. Data quality and labeling standards

Clinical labels need to come from qualified reviewers with documented inter-rater agreement, not a single annotator's judgment.

PHI Handling in the AI Training Pipeline

This is where most teams underestimate the work involved. Training a clinical model on real patient data means PHI protection has to extend into the machine learning pipeline itself, not just the production application.

• De-identification before training: Remove or mask the 18 HIPAA identifiers using Safe Harbor or Expert Determination methods before data touches a training environment.
• Minimum necessary access: Data scientists should only access the fields needed for the specific modeling task, not the full patient record.
• Segregated training environments: Training infrastructure should sit in its own access-controlled environment, separate from production systems handling live PHI.
• Vendor and BAA coverage: If a third-party labeling service, cloud GPU provider, or annotation vendor touches PHI at any point, a signed BAA needs to be in place before data transfer, similar to how SOC 2 Type II compliance gets verified for infrastructure vendors.
• Audit trails on data access: Every query or export from a training dataset should be logged with who accessed it and why.

A 2024 study published by Johns Hopkins researchers found that improperly de-identified datasets remain one of the most common findings during HIPAA compliance reviews of AI research programs, often due to residual metadata in imaging files.

Clinical Validation Expectations

FDA's guidance on AI and machine learning based SaMD expects evidence across three areas before a model reaches patients.

For adaptive models that continue learning post-deployment, the FDA's predetermined change control plan lets manufacturers pre-specify how the model may change over time without triggering a new submission for every update, provided the boundaries are documented upfront.

Read More: LLMs in Clinical Settings: What the FDA, HIPAA, and Your Hospital Client Actually Require

Common Pitfalls That Delay FDA Review or Fail Diligence

Practical Checklist Before You Build Clinical AI Product

Conclusion

Building a clinical AI model is as much a data governance exercise as it is a machine learning one.

FDA reviewers, hospital security teams, and investors all check the same underlying things: where your data came from, how PHI was protected during training, and whether you understood your model's risk classification before you built it.

Teams that document data lineage, de-identification, and validation plans from the start move through review far faster than those trying to reconstruct that evidence after the model already exists. Getting this right early is what separates a research prototype from a fundable, deployable clinical product.