HIPAA-Compliant AI in Healthcare: Building Clinical AI Scribes, Decision Support, and Patient-Facing AI

In October 2023, a clinical AI startup in New York, Series A, $16M raised, shipped an ambient documentation feature. The product listened to provider-patient conversations during outpatient visits, transcribed the audio, and used an LLM to generate a structured SOAP note that pre-populated in the provider’s EHR.

Providers loved it. Documentation time dropped from 22 minutes per encounter to 6 minutes. The NPS from the first 40 provider beta users was 74.

In the sixth week of beta, a family medicine physician in their Pittsburgh pilot site caught something. The AI-generated note for a patient visit about knee pain included a medication in the Assessment and Plan section, ibuprofen 600mg TID, that the provider had not prescribed during the visit. The patient was allergic to NSAIDs. The allergy was documented in their chart.

The medication had not been discussed in the visit. The LLM had hallucinated it, generating a plausible but entirely fabricated treatment recommendation from context patterns in its training data. The physician caught it in review before signing the note. The note was corrected. The patient was not harmed.

But here is what the founder told me when he called two weeks later: his product had a one-click “Accept and Sign” button that a provider under time pressure, and every provider is under time pressure, could use to push the AI-generated note to the EHR without reading it in detail.

One click. A hallucinated NSAID prescription for an NSAID-allergic patient. No harm occurred because the physician caught it. Not because the product was designed to catch it.

They redesigned the note review workflow before the next beta cohort. The one-click sign was removed. A required field-by-field attestation was added. The time savings dropped from 16 minutes to 11 minutes per encounter. Three beta providers complained about the extra steps. The founder shipped anyway.

Eleven minutes is still eleven minutes. And no hallucinated NSAID prescription has reached a clinical record since.

This is what clinical AI guardrail design looks like when you build it from first principles rather than from conversion rate optimization. It costs you some of the efficiency gain. It protects your users’ patients. And it keeps your product from being the one that caused the harm that ended the company.

Build the guardrails first. Optimize the efficiency second.

Eight Things Clinical AI Founders Get Wrong Before They Build

• Wrong #1: “OpenAI/Anthropic can sign a BAA, we’re covered.”

OpenAI offers a BAA under an enterprise agreement. Anthropic offers a BAA through direct enterprise negotiation. But a BAA with the LLM provider covers only the API service under the BAA’s specific terms. It does not cover every service those companies offer, it does not cover your data pipeline from clinical source to LLM and back, and it does not cover the third-party services in your stack that also touch the clinical data. The BAA with the LLM provider is one piece of a multi-vendor compliance architecture, not a complete solution.

• Wrong #2: “The LLM won’t hallucinate on clinical data.”

All LLMs hallucinate. The frequency and severity vary by model, by prompt design, by the specificity of the clinical domain, and by how well the model’s training data covered the relevant clinical area. Building a clinical AI product on the assumption that the LLM will not hallucinate is not a product decision, it is a liability decision. Every clinical AI output that could influence a clinical decision requires architectural guardrails that catch, flag, or prevent hallucinated content from reaching clinical use.

• Wrong #3: “We’ll add clinical review later, let’s ship with auto-accept first.”

There is no “add clinical review later” in clinical AI. The moment your product produces an output that a clinician could act on, a medication in a SOAP note, a diagnosis in a clinical summary, a recommendation in a patient communication, that output can cause harm if it is wrong and the clinician does not review it. Ship with the clinical review workflow from Day 1 or do not ship.

• Wrong #4: “Fine-tuning on clinical data is just training on more data.”

Fine-tuning a foundation LLM on clinical data, patient records, clinical notes, medical literature, involves ePHI that is subject to HIPAA. The fine-tuning process itself, the training infrastructure, the model weights that embed clinical data patterns, and the storage of fine-tuning datasets all have HIPAA implications. Fine-tuning on ePHI without a compliant data governance framework is a HIPAA violation regardless of whether the fine-tuning produces a better model.

• Wrong #5: “Patient-facing AI just needs a disclaimer.”

A disclaimer that says “this is not medical advice” does not eliminate clinical liability if the AI provides information that a reasonable patient could interpret as clinical guidance and that information is incorrect. Patient-facing clinical AI requires clinical content governance, crisis escalation pathways, culturally appropriate health literacy design, and, for mental health contexts, specific safety protocols. A disclaimer is legal language. It is not a clinical safety architecture.

• Wrong #6: “We don’t need IRB approval for clinical AI development.”

If your clinical AI development involves collecting data from patients or using patient data in a way that constitutes human subjects research, which includes using identified patient data to train or validate a clinical AI model, IRB approval may be required. The determination depends on whether the activity meets the regulatory definition of human subjects research under 45 CFR Part 46. Get a research compliance attorney’s opinion before collecting or using patient data for model development.

• Wrong #7: “The FDA doesn’t regulate clinical AI unless it’s a diagnostic tool.”

The FDA’s regulatory reach for AI in healthcare is broader than most founders realize. Software that provides patient-specific recommendations that influence treatment decisions, even if positioned as decision support rather than diagnosis, may meet the SaMD definition. The FDA’s 2022 CDS guidance and the 2023 AI action plan clarify the boundaries. Do not assume your clinical AI product is outside FDA jurisdiction without a regulatory attorney’s written opinion.

• Wrong #8: “Model performance on the benchmark dataset means clinical performance.”

Academic AI benchmark performance, USMLE pass rates, clinical NLP benchmarks, diagnostic accuracy on published datasets, does not translate directly to clinical performance in your specific use case on your specific patient population. Clinical validation in your intended clinical environment, with your intended user population, on your intended patient population, is required to characterize real-world performance. Benchmark performance is a starting point. It is not clinical evidence.

The Clinical AI Landscape in 2026, Four Categories, Four Different Builds

Clinical AI in 2026 spans four meaningfully distinct product categories. Each has different regulatory requirements, different architectural patterns, different clinical risk profiles, and different go-to-market motions.

• Category 1: Clinical AI Scribes (Ambient Documentation)

What it is: AI that listens to provider-patient conversations, transcribes the audio, and generates structured clinical documentation, SOAP notes, visit summaries, after-visit summaries, that the provider reviews, edits, and signs.

Clinical risk profile: Moderate. The primary risk is hallucination, content in the generated note that was not in the conversation, or content that was discussed but is incorrectly captured. The provider’s review step is the primary safety control.

Regulatory profile: Generally outside FDA SaMD jurisdiction if the software does not independently diagnose or treat, it is generating documentation, not making clinical decisions. The FDA’s CDS guidance suggests ambient documentation software is not a medical device if it only documents what the provider said and does not add clinical interpretation. Confirm with a regulatory attorney.

HIPAA profile: Significant. Session audio is ePHI. Transcripts are ePHI. Generated notes are ePHI. The entire pipeline, audio capture, transcription, LLM processing, note storage, requires HIPAA BAA coverage for every service involved.

Market size and maturity: The most commercially mature category of clinical AI in 2026. Nuance DAX Copilot (Microsoft), Suki, Abridge, Ambience Healthcare, and DeepScribe are the established players. The market is competitive but not saturated, and health systems that have not yet deployed ambient documentation are actively evaluating solutions.

• Category 2: Clinical Decision Support AI

What it is: AI that analyzes patient clinical data, structured EHR data, lab results, imaging reports, clinical notes, and surfaces patient-specific recommendations, risk scores, care gaps, or clinical insights to providers or care managers.

Clinical risk profile: High. The primary risks are: incorrect recommendations that lead to inappropriate treatment decisions, missed recommendations that lead to delayed treatment, and algorithmic bias that produces disparate recommendations across patient subgroups.

Regulatory profile: Variable and FDA-sensitive. Clinical decision support that analyzes physiological signals, medical images, or in vitro diagnostic data is a medical device. CDS that surfaces evidence-based guideline recommendations based on structured clinical data may qualify for the CDS exemption, but the exemption analysis is complex and case-specific. Get a regulatory attorney’s written opinion.

HIPAA profile: Significant. The patient clinical data analyzed by the AI is ePHI. Every service in the pipeline, EHR data extraction, AI inference, results storage, requires HIPAA BAA coverage.

Market opportunity: High. Clinical decision support AI is the highest-value category for health system enterprise sales, risk stratification, care gap identification, sepsis prediction, readmission prevention, medication safety. The market is large, the buyer is the health system CMO and CMIO, and the contract values are significant ($200K–$2M/year per health system).

• Category 3: Patient-Facing AI

What it is: AI that interacts directly with patients, symptom checkers, medication adherence chatbots, mental health support tools, care navigation assistants, chronic disease management companions.

Clinical risk profile: Highest of the four categories. The patient is the direct user. The patient may be medically illiterate, may be in acute distress, may misinterpret AI outputs as clinical advice, and may take clinical action based on AI outputs without provider involvement. The consequences of incorrect AI outputs reach the patient directly, without a clinician as an intermediary safety layer.

Regulatory profile: FDA-sensitive. Patient-facing AI that provides patient-specific health information that could be used to make clinical decisions, “your symptoms are consistent with X”, may be a medical device. The FDA has been actively developing its regulatory posture for direct-to-patient AI. Get a regulatory attorney’s opinion before building patient-facing clinical AI.

HIPAA profile: Variable. Patient-facing AI in a consumer context (direct-to-consumer wellness app) may not be subject to HIPAA if the app has no Covered Entity relationship. Patient-facing AI embedded in a health plan’s member portal or a provider’s patient portal is subject to HIPAA. Understand your HIPAA applicability before building.

Market opportunity: Large but structurally challenging. Patients are cost-sensitive (consumer willingness to pay is lower than B2B), the regulatory risk is higher than B2B clinical AI, and consumer health AI products face FTC scrutiny as well as potential FDA oversight. The highest-value patient-facing AI opportunity is embedded in payer or employer benefit offerings where the B2B buyer funds access.

• Category 4: Administrative Healthcare AI

What it is: AI that automates healthcare administrative workflows, prior authorization, revenue cycle management, claims processing, clinical documentation coding (ICD-10/CPT), scheduling optimization, contract management.

Clinical risk profile: Low. Administrative AI does not directly influence clinical decisions or patient treatment. The risk is financial and operational, incorrect coding, incorrect prior authorization, billing errors, rather than clinical.

Regulatory profile: Generally outside FDA jurisdiction. Administrative AI that does not involve patient-specific clinical decision-making is typically not a medical device.

HIPAA profile: Significant. Administrative healthcare AI processes claims data, billing data, and scheduling data, all of which may contain ePHI. HIPAA requirements apply fully.

Market opportunity: Underestimated and growing rapidly. Revenue cycle management AI, prior authorization automation, and clinical coding AI are among the fastest-growing segments of healthcare AI investment in 2026, because the ROI is measurable, the buyer is the CFO and revenue cycle director, and the regulatory complexity is lower than clinical AI.

From a US founder call: “I spent two years building a clinical decision support AI for sepsis prediction. Raised $12M. The health system sales cycle was 18 months. The IRB approval for the clinical validation study took 8 months. The EMR integration took 6 months per health system.

Two of my engineers left and started a company doing prior authorization automation AI. They raised $6M, had their first paying customer in four months, and are at $2M ARR while I am still closing my second health system. I am not saying clinical AI is wrong. I am saying administrative AI is a faster path to revenue for a founder who needs to show traction.”, Series A clinical AI founder, Atlanta.

The Regulatory Stack for Clinical AI, HIPAA, FDA SaMD, FTC, and State Laws

• HIPAA, The Baseline

All patient clinical data processed by clinical AI is ePHI. The Privacy Rule governs what clinical data can be used for AI purposes. The Security Rule governs how clinical data must be protected in the AI pipeline. The Breach Notification Rule applies when ePHI is improperly accessed.

The minimum necessary principle (§164.502(b)) applies to clinical AI: your AI system should access only the patient data elements necessary for the specific AI function being performed. An AI that needs the patient’s medication list to perform drug interaction checking should not have access to the patient’s mental health records, SUD history, or HIV status.

The authorization requirement for using ePHI for AI development: using identifiable patient ePHI to train or fine-tune an AI model is a use of ePHI. Under HIPAA, ePHI can be used for treatment, payment, and healthcare operations without patient authorization, but AI model training may not fall neatly within “healthcare operations” depending on the context. The safest approach for AI training datuse de-identified data (de-identified under the Safe Harbor or Expert Determination method per §164.514), or obtain patient authorization, or use a Limited Data Set under a Data Use Agreement.

• FDA SaMD Framework

For clinical AI specifically: the CDS exemption is narrower than most founders assume. The PCCP is essential for AI/ML SaMD that will be retrained post-clearance. The FDA’s 2023 AI action plan signals increasing regulatory attention to clinical AI across all four categories.

The FDA’s current enforcement posture for clinical AI that has not sought clearance: the FDA has generally focused enforcement on the highest-risk categories first, AI diagnostic tools for cancer, cardiac conditions, and ophthalmology. But the trend is toward broader enforcement, not narrower. Build with the assumption that your product will eventually require regulatory engagement with the FDA.

• FTC, For Consumer-Facing Clinical AI

The FTC’s enforcement authority under Section 5 of the FTC Act (unfair or deceptive acts) and the FTC Health Breach Notification Rule applies to consumer-facing health AI that is not subject to HIPAA. The FTC’s 2023 policy statement on AI makes clear that the FTC considers health-related AI claims, including claims about AI accuracy, clinical evidence for AI recommendations, and data privacy, within its enforcement purview.

For patient-facing AI: avoid overstating the clinical evidence for AI recommendations, avoid making accuracy claims that cannot be substantiated, and ensure your privacy policy accurately describes how patient data is used for AI purposes.

• State Laws, The Emerging Patchwork

Several states have enacted or are considering AI-specific health regulations in 2026:

California SB 1120 (2024): Requires health plan algorithms used for clinical decisions to be disclosed to patients and to be auditable for bias. Applies to payer-side clinical AI.

Colorado SB 169 (2024): Regulates algorithmic decision-making in insurance, including health insurance. Applies to AI used in coverage determinations.

Several states with comprehensive privacy laws (California, Colorado, Virginia) include provisions on automated decision-making that apply to health-related AI.

The state AI regulatory landscape for healthcare is evolving rapidly. A multi-state regulatory analysis from a healthcare attorney is worth the investment before launching clinical AI in a multi-state market.

The 16-Question Clinical AI Readiness Audit

• Have you determined whether your clinical AI product is a medical device?

Regulatory attorney written opinion. Not a verbal discussion. A written opinion that documents the analysis.

• If your product is a medical device, what is your FDA regulatory pathway?

510(k), De Novo, or PMA. If you do not have a pathway, you do not have a product launch timeline.

1. If your product claims CDS exemption, is the exemption analysis documented in writing?

Document the four-part test analysis. Do not assert exemption without written documentation.

• Does your clinical AI process ePHI?

Almost certainly yes. Identify every point in the pipeline where ePHI is processed: audio capture, transcription, EHR data ingestion, LLM inference, output storage, audit logging.

1. Does every service in your ePHI processing pipeline have a HIPAA BAA?

List every service. Confirm BAA availability and execution for each.

• Which LLM are you using and does it have a BAA covering your use case?

AWS Bedrock (under AWS BAA), Azure OpenAI with HIPAA mode, OpenAI Enterprise BAA, Anthropic Enterprise BAA, each has different coverage, different scope, and different conditions. Know exactly which services are covered before clinical data flows through the API.

• Have you designed hallucination guardrails for every clinical AI output that could influence a clinical decision?

Not “do you plan to add guardrails”, have you designed them. Source citation requirements, confidence framing, human review gates, clinical escalation pathways.

• Is there a required human review step before any AI-generated clinical content becomes a clinical record?

Auto-accept is not appropriate for any AI output that will be stored as a clinical record. The provider must review and attest. Design this into the product from Day 1.

• Have you designed the clinical escalation pathway for patient-facing AI?

For mental health AI: crisis detection and 988/emergency services connection. For general patient-facing AI: escalation to a human care navigator or clinical staff when the AI cannot safely respond to a patient query.

• What is your training data governance policy?

What data was used to train or fine-tune your model? Was it de-identified? Was it obtained under appropriate authorization? Is the training data pipeline documented?

• What is your model versioning and audit trail?

Every model version deployed to production must be logged. Every clinical output must be attributed to the model version that generated it. If a hallucination is discovered in a clinical record, you must be able to identify which model version produced it and which other outputs that version may have affected.

• What is your model performance monitoring architecture?

How do you detect performance degradation after deployment? What metrics do you track? What threshold triggers investigation or model update?

• Have you conducted a bias analysis on your AI model?

Performance across demographic subgroups: age, sex, race/ethnicity, primary language, insurance status. Performance across clinical subgroups: disease severity, comorbidity burden, clinical site.

• What is your patient data use policy for AI improvement?

Can you use patient interaction data to improve your models? Under what authorization? With what patient notice? This must be documented in your privacy policy and implemented in your data governance framework.

• Have you designed for minimum necessary data access?

Does your AI access only the patient data elements it needs for the specific function it performs? Or does it have broad access to the full patient record?

• What is your incident response plan for a clinical AI failure?

A hallucinated clinical record is discovered. A patient-facing AI provides incorrect medical information. What is your response? Who is notified? How is the clinical impact assessed? How is the affected output corrected in the clinical record?

LLM Selection Under BAA, The 2026 Decision Tree

This is the decision every clinical AI founder must make explicitly before clinical data enters the AI pipeline. Here is the complete 2026 picture.

The decision framework:

Question 1: Does your AI feature process ePHI, individually identifiable health information from a patient?

If no, any LLM can be used. The BAA requirement does not apply to non-ePHI data.

If yes, proceed to Question 2.

Question 2: Is a BAA available from the LLM provider that covers your specific use case?

• AWS Bedrock:

BACovered under the standard AWS BAA for HIPAA-eligible services. AWS Bedrock is on the HIPAA-eligible services list as of 2026. Models available: Claude (Anthropic), Llama (Meta), Mistral, Titan (Amazon), and others through the Bedrock model catalog. Verdict: Our default recommendation for most clinical AI features processing ePHI.

The BAA situation is unambiguous, it is the standard AWS BAA you already have. No enterprise negotiation required. No separate legal review needed for the LLM specifically. Limitation: Model selection is limited to Bedrock’s catalog. Not every frontier model is available on Bedrock. For use cases where a specific model capability is required that Bedrock does not yet offer, evaluate alternatives.

• Azure OpenAI with HIPAA mode:

BAAvailable under an Azure enterprise agreement with HIPAA configuration enabled. Azure OpenAI is on Microsoft’s list of HIPAA-covered services when configured correctly. Models available: GPT-4o, GPT-4 Turbo, GPT-4, GPT-3.5 Turbo, text embedding models. Verdict: Strong option for clinical AI products already in the Azure ecosystem, or for use cases requiring GPT-4o class performance where Azure enterprise relationship is in place.

Requires Azure enterprise agreement and correct HIPAA configuration, confirm with your Microsoft account team that the specific services you use are in HIPAA mode. Limitation: Requires Azure enterprise agreement. Configuration complexity higher than AWS Bedrock. Microsoft’s covered services list must be validated specifically, not all Azure AI services are in the HIPAA-covered list.

• OpenAI API with Enterprise BAA

BAA available under OpenAI Enterprise agreement. The BAA covers the API (chat completions, embeddings, fine-tuning) under the enterprise agreement terms. Models available: GPT-4o, GPT-4 Turbo, GPT-4, o1, o3, text embedding models. Verdict: Valid option for clinical AI teams with an OpenAI enterprise agreement and legal counsel review of the BAA scope.

The BAA scope, which specific services are covered, what data processing limitations apply, what the data retention terms are, must be reviewed by your healthcare attorney before clinical ePHI flows through the API. Limitation: Requires enterprise agreement (not available on standard API plans). BAA scope review adds legal cost and time. OpenAI’s enterprise pricing is higher than standard API pricing.

• Anthropic API with Enterprise BAA

BAA available through direct enterprise negotiation with Anthropic’s enterprise team. Not yet available as a self-service agreement. Models available: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku, Claude 3.5 Haiku. Verdict: Available for clinical AI use cases where Claude’s specific capabilities (long context, instruction following, clinical reasoning) justify the enterprise negotiation.

Requires direct engagement with Anthropic’s enterprise team, not a self-service signup. Limitation: Enterprise negotiation timeline (4–8 weeks). The BAA terms must be reviewed by your healthcare attorney.

• Google Cloud Vertex AI (Gemini models):

BAGoogle Cloud’s HIPAA BAA covers Vertex AI services when using the Healthcare Data Engine and when the specific Vertex AI services are on Google’s covered services list. Confirm current coverage with Google Cloud’s healthcare team. Models available: Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 1.0 Pro, Med-PaLM 2 (available to select partners).

Verdict: Strong option for clinical AI products in the Google Cloud ecosystem. Med-PaLM 2, Google’s clinical-domain-specific foundation model, is available to select clinical AI partners and shows strong performance on clinical reasoning benchmarks. For products requiring clinical domain performance specifically, the Med-PaLM 2 access pathway is worth investigating.

Limitation: Google’s HIPAA-covered services list must be validated specifically for each Vertex AI service used. Med-PaLM 2 access is not self-service.

• Self-hosted open-source models (Llama, Mistral, Mixtral, clinical fine-tunes):

BANo BAA required, you control the infrastructure. The model runs on your HIPAA-compliant cloud infrastructure under your existing AWS/GCP/Azure BAA. Models available: Llama 3 (70B, 8B), Mistral 7B, Mixtral 8x7B, BioMistral (clinical fine-tune), ClinicalCamel (clinical fine-tune), and others.

Verdict: The appropriate choice when: (1) the clinical data sensitivity is high enough that your legal team is not comfortable with any cloud LLM processing ePHI, (2) the patient population includes individuals with particularly sensitive conditions (SUD, HIV/AIDS, psychiatric history) where data sovereignty concerns are paramount, (3) enterprise health system customers require on-premises deployment without data leaving the health system’s infrastructure.

Limitation: Significant infrastructure overhead, GPU compute, model serving infrastructure, model management, inference optimization. Frontier model performance is generally below cloud-hosted GPT-4o or Claude 3.5 Sonnet for complex clinical reasoning tasks. Self-hosted deployment is the right architectural choice for a minority of clinical AI use cases.

• The BAA is not the only compliance consideration:

Having a BAA with the LLM provider covers the LLM API service. It does not cover:

• The audio transcription service (if you are processing session audio), Amazon Transcribe Medical (covered under AWS BAA), Deepgram for Healthcare (BAA available), AssemblyAI (BAA available for healthcare). Confirm for your specific transcription service.
• The vector database storing clinical embeddings (if you use RAG architecture), Pinecone (BAA available on enterprise plan), Weaviate (BAA available), Chroma (self-hosted avoids the BAA question).
• The prompt logging and observability platform, Langsmith, Helicone, and similar LLM observability platforms may capture prompt content including ePHI in logs. Confirm BAA availability before enabling prompt logging for clinical AI features.
• The error monitoring service, Sentry, Datadog, if it captures API request/response payloads including clinical content.

Map every service in the ePHI processing pipeline. Confirm BAA coverage for each. This is the BAA registry for your clinical AI product, maintain it from Day 1.

EB Index 2026: Across 28 clinical AI products we have supported since 2022, the most common BAA gap discovered during SOC 2 readiness assessments is the LLM observability/prompt logging platform.

Founders who enable prompt logging for debugging (a reasonable engineering decision) inadvertently route clinical prompt content, which often includes patient symptoms, medications, and clinical history, through an observability platform that has no HIPAA BAA. Disable prompt logging in production or confirm BAA availability for your observability platform before going live.

Clinical AI Scribes, Ambient Documentation Done Right

Ambient clinical documentation is the most commercially deployed category of clinical AI in 2026. It is also the category with the most well-understood failure modes, which means the guardrail architecture is better established than for more novel clinical AI applications.

The ambient documentation pipeline:

Audio capture: Session audio captured via device microphone (mobile app, desktop app, or dedicated hardware). The audio is ePHI from the moment it is captured, patient voice is individually identifiable and the content is protected health information. Encrypted in transit from the point of capture. Not stored permanently on the capture device.

Transcription: Audio converted to text in real time or near-real time. Transcription service must have a HIPAA BAA. Amazon Transcribe Medical (under AWS BAA) is our standard recommendation, it is trained on medical terminology and handles clinical speech patterns better than general-purpose transcription services. Speaker diarization (identifying who is speaking, provider vs. patient) significantly improves note structure.

Clinical entity extraction: NLP processing of the transcript to identify clinical entities, symptoms, diagnoses, medications, procedures, vitals. This step can be performed by the LLM or by a separate NLP layer before the LLM generates the structured note.

SOAP note generation: The LLM receives the transcript (and optionally the extracted clinical entities and the patient’s relevant EHR context) and generates a structured SOAP note. The note structure, what goes in Subjective, Objective, Assessment, Plan, is defined by a system prompt that encodes clinical documentation standards.

Provider review and attestation: The generated note is presented to the provider for review. The provider edits, approves, and signs. The signed note is pushed to the EHR. No AI-generated content enters the clinical record without provider review and attestation.

• The SOAP note generation system prompt architecture:

The system prompt that governs AI scribe behavior is among the most clinically consequential engineering artifacts in the product. It defines:

• What clinical documentation standards the note must follow
• What content should be in each SOAP section
• What the AI should do when content is ambiguous or unclear in the transcript
• What the AI must NOT do, fabricate information not in the transcript, add clinical interpretation beyond what was discussed, include medications not mentioned in the visit

A poorly designed system prompt produces notes that are plausible-sounding but clinically incorrect. A well-designed system prompt produces notes that accurately capture the clinical encounter with appropriate uncertainty markers where the transcript was unclear.

Example system prompt constraint (the constraint that prevents the medication hallucination from the opening story):

You are generating a clinical SOAP note from a provider-patient conversation transcript.

CRITICAL RULES:

– Include ONLY medications explicitly mentioned in the transcript.

– If a medication is NOT clearly mentioned in the transcript, DO NOT include it in the Plan section.

– If a clinical finding is unclear or ambiguous in the transcript, use the phrase “Provider to clarify:” followed by the unclear element rather than inferring a specific value.

– Do NOT add clinical interpretation, diagnoses, or recommendations that were not explicitly discussed in the conversation.

– If the transcript is incomplete or inaudible for a portion of the encounter, note “[Inaudible section]” rather than inferring content.

These constraints reduce the creative latitude of the LLM, which means the notes are less polished than an unconstrained LLM would produce. They also mean the notes are far less likely to contain hallucinated clinical content. Clinical safety beats polish. Ship the constrained version.

• The provider review UX, designed for safety, not speed:

The note review interface is where hallucinations are caught or missed. Design it for clinical accuracy, not for fastest possible signing.

What works:

Section-by-section review: The SOAP note is presented one section at a time (S → O → A → P), with the relevant transcript excerpt visible alongside each section. The provider can see exactly what the AI used to generate each part of the note and can verify accuracy against the source conversation.

Confidence indicators: Fields where the AI’s confidence in the transcription or the clinical entity extraction is below a threshold are highlighted for required provider attention. Not all content needs equal scrutiny, the provider’s attention should be directed to the highest-uncertainty content first.

Medication-specific attestation: A separate attestation step for any medication that appears in the Plan section. The provider explicitly confirms that each medication was discussed and prescribed during the visit before it is included in the signed note.

What does not work:

One-click sign with full note visible: Providers under time pressure will sign without reading. Do not design for this pathway.

Auto-populated clinical fields without source citation: Any clinical field populated from the transcript should show the transcript excerpt that supports it. Dark-pattern auto-population that hides the source encourages trust without verification.

• The EHR integration for note delivery:

The reviewed and signed note must be delivered to the EHR in the correct location, the encounter note section for the specific visit date. EHR integration for note delivery requires:

• FHIR R4 Document reference resource creation (for FHIR-capable EHRs)
• HL7 v2 MDM message for legacy EHR note delivery
• Epic SMART on FHIR write-back for Epic customers (requires App Orchard write-back certification)
• Athenahealth API note creation endpoint for Athena customers

The note delivery integration is a separate engineering workstream from the AI scribe itself, scope it explicitly in discovery.

• Audio retention policy:

Session audio is ePHI. Retaining session audio long-term creates ongoing HIPAA compliance obligations and creates a subpoena risk for the full audio of every clinical encounter ever recorded.

Our recommendation: retain session audio for 72 hours after the note is signed, sufficient for any note correction that requires reference to the original recording. Delete automatically at 72 hours. The clinical record is the signed note, not the audio. The audio served its purpose in generating the note. Retain only what is clinically necessary.

This is a data minimization decision that also reduces long-term liability and storage costs. Document the retention policy explicitly and implement automated deletion before going live.

Compliance trap: AI scribe products that use session audio to train or improve their transcription or note generation models must obtain appropriate authorization before using identified audio for this purpose.

An opt-in consent for “audio may be used to improve our service” that is buried in the terms of service is insufficient for using clinical ePHI for model training. Design a clear, affirmative opt-in for model improvement data use, and build the technical mechanism to honor opt-out requests by excluding a provider’s session audio from training pipelines.

Clinical Decision Support AI, The Guardrails That Cannot Be Optional

Clinical decision support AI analyzes patient clinical data and surfaces recommendations, risk scores, care gaps, or clinical insights to providers or care managers. The guardrail architecture for CDS AI is different from ambient documentation, the AI is not documenting what was said, it is generating new clinical insights from data analysis.

The source citation requirement:

Every CDS AI output that influences a clinical decision must be accompanied by the source data that generated it. A sepsis risk score of 78% is not clinically useful without knowing which clinical features drove that score, elevated lactate, tachycardia, hypotension, recent antibiotic administration. A care gap alert for a patient overdue for a mammogram is not clinically useful without knowing the source of the patient’s age and last mammogram date.

The source citation serves two clinical purposes: it allows the clinician to verify that the AI’s data inputs are correct (a common failure mode is AI reasoning from incorrect or outdated data in the EHR), and it allows the clinician to apply their clinical judgment to the recommendation rather than simply accepting the AI’s output.

Technically: the source citation must trace from the AI output to the specific EHR data elements that generated it, with timestamps showing when that data was last updated. This requires that the inference pipeline capture data provenance, which data elements, from which records, at what version, alongside the inference output.

The uncertainty communication requirement:

CDS AI outputs must communicate uncertainty to the clinician. A risk score without a confidence interval, a recommendation without a supporting evidence level, a clinical insight without a stated limitation, these create false certainty in clinical environments where uncertainty is clinically meaningful.

Design patterns for uncertainty communication:

Confidence ranges: “Sepsis risk: 72–84% (moderate confidence)” rather than “Sepsis risk: 78%.”

Evidence level attribution: “Based on SEPSIS-3 criteria applied to structured EHR data. Does not incorporate clinical gestalt or findings not documented in the EHR.”

Data recency warning: “This recommendation is based on lab values last updated 6 hours ago. Clinical status may have changed.”

Limitation disclosure: “This algorithm was validated on adult patients. Performance in pediatric patients has not been validated.”

The minimum necessary data access requirement:

CDS AI must access only the patient data elements necessary for the specific clinical question being answered. A drug interaction checker does not need the patient’s psychiatric history. A diabetic retinopathy screening reminder does not need the patient’s HIV status or substance use history.

This is both a HIPAA requirement (minimum necessary, §164.502(b)) and a clinical ethics requirement. Unnecessary access to sensitive clinical data creates risk, both the risk of data exposure and the risk of AI systems incorporating sensitive data in ways that the patient did not authorize and the clinician did not intend.

Implement a data access manifest for every CDS AI feature: a documented list of the specific data elements the feature accesses, why each element is needed, and a technical control that prevents the feature from accessing data elements outside the manifest.

The algorithmic bias monitoring requirement:

CDS AI trained on historical clinical data inherits the biases in that data. Historical clinical data reflects historical disparities in healthcare, disparities in diagnostic rates, treatment rates, and outcomes across demographic groups. An AI trained to predict hospital readmission risk trained on data from a health system that historically underdiagnosed heart failure in Black women will likely underpredict readmission risk for Black women.

Bias monitoring for CDS AI requires:

Subgroup performance analysis at launch: sensitivity, specificity, PPV, NPV, and AUC-ROC by age group, sex, race/ethnicity, insurance type, and any other clinically relevant demographic subgroup.

Ongoing subgroup performance monitoring post-launch: tracking whether real-world performance differs by subgroup and whether performance gaps emerge over time.

Remediation plan: documented processes for investigating and addressing subgroup performance disparities, including model retraining on more representative data, feature engineering to reduce disparate impact, or, in cases where disparate performance cannot be adequately addressed, limiting the tool’s deployment contexts to those where performance has been validated.

The clinician alert fatigue problem:

CDS AI that generates too many alerts creates alert fatigue, clinicians learn to dismiss alerts reflexively because so many of them are false positives. A CDS system with a 90% false positive rate, even if the 10% true positives are clinically meaningful, will be ignored after the first week.

Alert design principles for CDS AI:

High specificity over high sensitivity for actionable alerts: a few highly specific alerts that the clinician can trust are more clinically valuable than many low-specificity alerts that the clinician ignores.

Alert fatigue monitoring: track the rate at which clinicians dismiss alerts without acting on them. Alert dismissal rates above 70% are a signal of alert fatigue that requires tuning.

Tiered alert urgency: not all CDS insights need to be alerts. Surface time-sensitive, high-confidence, actionable insights as alerts. Surface lower-urgency insights as background information in the patient chart view that the clinician can review at their discretion.

Patient-Facing AI, The Highest UX Stakes in Healthcare

Patient-facing clinical AI is the category where the consequences of design errors are most direct. The patient is the end user. There is no clinician as an intermediary. Here is how to build it safely.

• The clinical scope definition:

Before building patient-facing AI, define explicitly what the AI can and cannot do. The clinical scope definition is not just a product decision, it is a safety architecture decision.

What is in scope: answering general health questions from a curated, clinically-reviewed knowledge base, reminding patients about scheduled appointments and medications, helping patients understand their clinical test results (with appropriate framing), connecting patients to appropriate care resources.

What is never in scope: providing patient-specific diagnosis, recommending specific treatments or medications, advising patients to change or stop prescribed medications, providing emergency medical guidance in place of emergency services contact.

The scope definition must be implemented technically, not just stated in a disclaimer. The system prompt must explicitly prohibit out-of-scope responses. The AI must be configured to recognize when a patient question falls outside the defined scope and to respond with an appropriate referral to clinical care rather than attempting to answer.

• The crisis escalation architecture:

For any patient-facing AI serving a general health population, crisis situations will occur. Patients will disclose suicidal ideation, express hopelessness, describe symptoms of acute medical emergencies, or indicate that they are in danger. The product must be designed for these moments.

Crisis detection layer: keyword and semantic pattern matching for explicit and implied crisis language. Categories: suicidal ideation, self-harm, acute medical emergency (chest pain, stroke symptoms, severe allergic reaction), domestic violence or abuse, substance use crisis.

Immediate crisis response: when crisis language is detected, by keyword match, semantic analysis, or both, the product immediately surfaces crisis resources to the patient. 988 Suicide and Crisis Lifeline for mental health crises. 911 for medical emergencies. National Domestic Violence Hotline for safety concerns. The crisis resources must be surfaced within the product, not as an external link that the patient may not follow, and must be accessible within two interactions from any screen.

Escalation to human support: for platforms with a human care navigator or clinical support function, a crisis detection event automatically creates an escalation notification to the clinical support team. The escalation is logged: what the patient said, when, what the AI responded, and when the clinical support team was notified.

Do not rely solely on AI to manage crisis situations. The AI’s role is crisis detection and immediate resource surfacing. A human clinical support pathway must exist for every patient-facing AI platform serving health-related queries.

• Health literacy and accessibility design:

Patient health literacy in the US is lower than most clinical AI products assume. Approximately 36% of US adults have basic or below-basic health literacy, meaning they have difficulty reading and understanding health information presented at a college reading level.

Patient-facing AI must be designed for low health literacy: use plain language (sixth-grade reading level or below for all patient-facing content), avoid medical jargon or define it immediately when used, use short sentences and simple vocabulary, confirm patient understanding at key points in the interaction.

Language access: for patient populations that include non-English speakers, the AI must provide responses in the patient’s preferred language or provide clear pathways to human language interpretation services. An AI that responds only in English to a Spanish-speaking patient has created a health equity problem, not a health equity solution.

• The “I don’t know” pattern:

Patient-facing AI must know the limits of its knowledge and communicate those limits clearly. When a patient asks a question that is outside the AI’s knowledge base, or that requires clinical judgment the AI cannot provide, the AI must say so, directly and without hedging, and direct the patient to appropriate clinical care.

Bad: “That’s a great question! While I can’t give medical advice, it sounds like your symptoms might be related to [plausible but unvalidated clinical speculation].”

Good: “I’m not able to answer that question safely, your symptoms need a clinician’s evaluation. Please contact your doctor, or if this feels urgent, go to an urgent care center or emergency department.”

The second response is less engaging. It is also safer. Build the “I don’t know” response as a first-class feature, not as an edge case handler.

• Content moderation for patient-generated inputs:

Patients will provide inputs that are outside the expected scope, abuse, harassment, personally distressing content, content that indicates acute crisis. Patient-facing AI must have content moderation that:

Detects out-of-scope inputs and routes them appropriately (crisis → escalation, abuse → graceful response and topic change)

Does not reinforce harmful patterns, AI responses to distressed patients must not validate harmful thoughts or behaviors, even implicitly

Maintains appropriate clinical boundaries, the AI is not a therapist, it is not a friend, and its responses must not encourage the patient to treat it as a primary emotional support relationship.

AI for Administrative Healthcare Workflows, The Underrated Opportunity

Administrative healthcare AI is the fastest path to clinical AI revenue for a founder who needs to show traction before a Series A. Here is why and what it takes to build it.

• Prior authorization automation:

Prior authorization, the process by which a provider requests insurance approval before delivering specific services, is one of the most burdensome administrative processes in US healthcare. It consumes an estimated $35 billion per year in administrative costs across the US healthcare system. Providers spend an average of 13 hours per week on prior authorization. 94% of physicians report that prior authorization delays care, and 34% report it has led to a serious adverse event.

AI for prior auth automation: an AI that reads the clinical criteria for a specific payer’s prior auth requirement, pulls the relevant clinical documentation from the patient’s EHR (diagnosis codes, clinical notes, lab results, imaging reports), and pre-populates the prior auth request with supporting documentation, reducing the provider’s time from 45 minutes to 5 minutes per request.

Clinical risk profile: low, the AI is assisting with documentation, not making clinical decisions. Regulatory profile: generally outside FDA jurisdiction. Revenue model: per-authorization fee ($3–$10/auth), or monthly subscription per provider. Time to first customer: 3–6 months.

• Revenue cycle management AI:

Clinical documentation is the foundation of the revenue cycle, inaccurate or incomplete documentation leads to denied claims, delayed reimbursement, and lost revenue. AI that analyzes clinical documentation and suggests more specific or complete coding, identifying that a note describes “diabetes with peripheral vascular disease” and should be coded E11.51 rather than just E11, reduces claim denial rates and increases revenue capture.

Clinical risk profile: low, the AI is suggesting coding, not making clinical decisions. Regulatory profile: generally outside FDA jurisdiction. Revenue model: percentage of additional revenue captured (2–5%), or monthly subscription per provider. Time to first customer: 4–8 months.

• Clinical documentation coding (ICD-10/CPT):

Computer-assisted coding (CAC), AI that reads clinical notes and suggests appropriate ICD-10 diagnosis codes and CPT procedure codes, is an established market with significant AI-driven improvement opportunity. Traditional CAC systems use rule-based engines. LLM-based CAC can handle the full complexity of clinical language, including free-text notes, with significantly higher accuracy.

This is an area where EngineerBabu’s CMMI Level 5 credential and healthcare experience create a meaningful differentiation from pure-AI startups without healthcare process expertise.

The Clinical AI Data Architecture, Training, Fine-Tuning, and Inference Under HIPAA

• The data governance hierarchy for clinical AI:

Level 1, De-identified data (preferred for model development): Clinical data de-identified under the HIPAA Safe Harbor method (§164.514(b)), removing all 18 HIPAA identifiers, or the Expert Determination method (§164.514(b)(1)), statistical certification that re-identification risk is very small. De-identified data is not ePHI and is not subject to HIPAA restrictions on use.

This is the preferred data governance level for model training and fine-tuning. De-identify training data before using it for model development whenever possible.

Level 2, Limited Data Set under Data Use Agreement: A Limited Data Set (LDS) retains some data elements that are not in the Safe Harbor de-identification standard, dates, geographic data below state level, but has the 16 most direct identifiers removed. LDS use requires a Data Use Agreement with the data source and limits use to research, public health, or healthcare operations purposes.

Level 3, Identified ePHI under authorization or treatment/operations: Using identified ePHI for AI development requires either patient authorization or a determination that the use falls within treatment, payment, or healthcare operations under HIPAA. AI model training is not clearly within “healthcare operations” without careful legal analysis. Using identified ePHI for AI development without patient authorization and without a clear healthcare operations basis is a HIPAA risk.

• The RAG (Retrieval-Augmented Generation) architecture for clinical AI:

For clinical AI features that need patient-specific clinical context, CDS that reasons about a specific patient’s medical history, ambient scribes that access the patient’s medication list to contextualize the visit conversation, RAG is the architectural pattern that provides real-time clinical context without fine-tuning the LLM on ePHI.

RAG pipeline for clinical AI:

1. Patient-specific clinical data is retrieved from the EHR via FHIR API at inference time
2. Retrieved data is formatted as clinical context documents
3. Context documents are included in the LLM prompt alongside the user query
4. The LLM generates a response grounded in the retrieved patient-specific context
5. The response is returned to the provider or patient
6. Retrieved data and generated response are logged with ePHI-appropriate audit trail

The RAG approach provides patient-specific clinical context at inference time without requiring ePHI to be embedded in model weights through fine-tuning. This is a significantly simpler HIPAA compliance posture than fine-tuning on ePHI.

• The vector database for clinical RAG:

Clinical RAG architectures often use a vector database to store embeddings of clinical documents, medical literature, clinical guidelines, formulary data, institutional protocols, that the AI can retrieve as context alongside patient-specific EHR data.

HIPAA consideration: if the vector database stores embeddings of patient-specific ePHI documents (clinical notes, lab results, imaging reports), the vector database is handling ePHI and requires BAA coverage.

Vector databases with healthcare BAA availability:

• Pinecone: BAA available on enterprise plan
• Weaviate: BAA available for enterprise deployments
• Chromself-hosted deployment avoids the BAA question

If the vector database stores only non-patient-specific content (medical literature, guidelines, formulary data), it is not handling ePHI and does not require a BAA.

• Model versioning and audit trail for clinical AI:

Every model version that is deployed to production must be logged in your model registry with:

• Model version identifier
• Model architecture and parameters
• Training data description (what data, what de-identification method, what date range)
• Performance metrics on the validation dataset (overall and by subgroup)
• Deployment date
• Deprecation date (when the model version was retired from production)

Every clinical output must be attributable to the model version that generated it. If a hallucination is discovered in a clinical record six months post-deployment, you must be able to identify: which model version generated that output, when it was deployed, what its performance characteristics were, and whether other outputs from that model version may have the same failure mode.

This is not just a regulatory requirement, it is a clinical liability management requirement.

Hallucination Guardrails, The Engineering Architecture That Protects Patients

Hallucination is the defining reliability challenge of LLM-based clinical AI. Here is the engineering architecture that addresses it.

Layer 1: Prompt engineering constraints

The system prompt must explicitly prohibit hallucination-prone behaviors:

• Prohibit adding clinical information not present in the source data or conversation
• Prohibit speculating about diagnoses or treatments not explicitly discussed
• Require uncertainty markers when the source is ambiguous or incomplete
• Require the “I don’t know” response when the question falls outside validated knowledge
• Prohibit fabricating references, statistics, or clinical guidelines

Prompt constraints reduce hallucination frequency. They do not eliminate it. Every other guardrail layer is necessary.

Layer 2: Grounding validation

For clinical AI that generates outputs based on specific source data, ambient documentation based on a transcript, CDS based on EHR data, a grounding validation step verifies that each claim in the AI output is supported by specific source content.

Implementation: after the LLM generates the clinical output, a validation pass extracts specific clinical claims from the output and attempts to trace each claim to the source data. Claims that cannot be traced to source data are flagged as ungrounded and either removed from the output, highlighted for mandatory provider review, or cause the output to be regenerated with a more constrained prompt.

This is the architectural layer that would have caught the hallucinated NSAID prescription from the opening story, the medication was not in the transcript, and a grounding validation would have identified it as an ungrounded claim.

Layer 3: Clinical entity validation

Clinical entities in the AI output, medications, diagnoses, procedures, lab values, are validated against clinical reference databases:

Medications: validated against RxNorm (is this a real medication with the stated dosage and route?) Diagnoses: validated against ICD-10-CM (is this a real diagnosis code?) Drug interactions: flagged if the generated note includes a medication that interacts with a medication already in the patient’s EHR Allergies: flagged if the generated note includes a medication to which the patient has a documented allergy

The allergy check alone would have prevented the harm scenario from the opening story. It must be automated, not dependent on the provider catching it in review.

Layer 4: Confidence scoring

The LLM is prompted to generate a confidence score or uncertainty level for each clinical claim in its output. Low-confidence claims are surfaced to the provider with visual highlighting and a mandatory review flag.

Implementation: structured output from the LLM (using JSON schema output mode) that includes both the clinical content and a confidence assessment for each content element. Claims below the confidence threshold are highlighted in the review interface.

Layer 5: Human review gate

Every clinical AI output that will enter a clinical record must pass through a provider review gate before being stored as an official record. The review gate design:

• Presents the AI output with source citations visible
• Highlights ungrounded claims, low-confidence claims, and flagged clinical entities
• Requires explicit provider action on each flagged element before the note can be signed
• Records the provider’s review and attestation with timestamp and provider identity in the audit log

The human review gate is the last line of defense against hallucinated clinical content. Every previous layer reduces the frequency of errors that reach the review gate. The review gate catches what gets through.

Layer 6: Post-deployment hallucination monitoring

After deployment, monitor for hallucination patterns in clinical outputs:

Provider edit rate by field type: if providers consistently edit a specific field (e.g., the medication list in the Plan section), that field’s generation logic has a systematic error that needs investigation.

Provider rejection rate: if providers frequently delete entire sections of AI-generated notes, the generation quality for those sections needs improvement.

Clinical entity anomaly detection: flag AI outputs that include clinical entities (medications, diagnoses) that are statistically unusual for the visit type, a weight loss medication in a pediatric visit note, a cardiac medication in a well-child visit note.

Hallucination monitoring requires that you collect and analyze data on provider edits to AI-generated content. This data is clinically sensitive, it contains both the AI output and the provider’s corrections. Handle it under the same HIPAA framework as any clinical data.

The Real Cost Stack for Clinical AI Development in 2026

• Engineering (what you pay us):

Clinical AI Scribe MVP (ambient documentation, SOAP note generation, EHR integration for one EHR, provider review workflow): $140K–$220K / 14–20 weeks

Clinical Decision Support AI MVP (EHR data ingestion, risk stratification model, CDS alert delivery, bias monitoring infrastructure): $180K–$290K / 18–26 weeks

Patient-Facing AI MVP (symptom checker or care navigation, crisis escalation, health literacy design, one language): $110K–$185K / 12–18 weeks

Administrative AI MVP (prior auth automation or revenue cycle coding, EHR integration, provider workflow): $90K–$155K / 10–16 weeks

Dedicated clinical AI pod post-MVP: $28K–$46K/month

• LLM infrastructure costs:

AWS Bedrock, Claude 3.5 Sonnet: $3.00/million input tokens, $15.00/million output tokens (2026 pricing, confirm current rates)

At 500 ambient documentation sessions/day × 4,000 tokens average per session:, Input: 500 × 4,000 = 2,000,000 tokens/day = $6/day = $2,190/year, Output: 500 × 1,500 = 750,000 tokens/day = $11.25/day = $4,106/year, Total LLM cost at 500 sessions/day: approximately $6,300/year, well within reasonable product economics

At 5,000 sessions/day: approximately $63,000/year, needs margin management but commercially viable

Amazon Transcribe Medical: $0.0086/second of audio At 500 sessions/day × 15 minutes average: 500 × 900 = 450,000 seconds/day = $3,870/day This is expensive at scale, audio transcription is the dominant LLM infrastructure cost for ambient documentation at volume. Optimize session length and consider batched transcription for non-real-time use cases.

• HIPAA compliance infrastructure:

BAA enterprise agreements (OpenAI, Anthropic, where applicable): $15K–$40K/year enterprise plan Vector database with healthcare BAA (Pinecone enterprise): $2K–$8K/month depending on index size and query volume Clinical NLP tooling (Amazon Comprehend Medical, Azure Text Analytics for Health): $0.01–$0.05/API call

• Clinical validation:

IRB submission and approval: $3K–$10K (legal and administrative) Clinical study conduct for CDS AI (site costs, data collection, monitoring): $80K–$250K depending on study design Human factors validation study for patient-facing AI: $25K–$60K

• Regulatory (if FDA SaMD pathway is required):

For ambient documentation and administrative AI that qualifies for CDS exemption or is outside FDA jurisdiction: regulatory costs are limited to the classification opinion ($5K–$15K) and ongoing compliance monitoring.

EB Index 2026: The median total first-year cost for a clinical AI scribe product, engineering, LLM infrastructure, HIPAA compliance, clinical validation, and EHR integration for two health system customers, was $347,000. The median time from project start to first paying health system customer was 11 months. The largest timeline driver was health system clinical governance review at a median of 14 weeks per health system.

The 14-Week Clinical AI MVP Sprint

This timeline covers a clinical AI scribe MVP development, the most common clinical AI category for a first build. Adjust for CDS AI (longer clinical validation workstream), patient-facing AI (longer health literacy and crisis UX workstream), or administrative AI (shorter regulatory workstream).

Week 1: Discovery, Regulatory Scoping, and Data Architecture Design

Intended use statement written. Regulatory attorney CDS exemption analysis initiated. BAA mapping: every service in the ePHI pipeline identified, BAA availability confirmed for each. LLM selection finalized (AWS Bedrock recommended). Transcription service selected (Amazon Transcribe Medical). Training data governance policy documented. Minimum necessary data access manifest designed for each AI feature.

Week 2: BAA Execution and Prompt Architecture

AWS enterprise agreement confirmed (if not already in place). LLM system prompt architecture designed, clinical constraints documented, hallucination prevention rules written, uncertainty marker requirements specified. Grounding validation approach selected. Clinical entity validation reference databases identified (RxNorm for medication validation, allergy check integration designed). Audio retention policy documented (72-hour default).

Week 3: Infrastructure Provisioning and Pipeline Foundation

HIPAA-eligible cloud infrastructure provisioned. Audio capture infrastructure built, encrypted in transit from capture device, not stored on device. Transcription pipeline built, Amazon Transcribe Medical integrated, speaker diarization configured. SBOM generation in CI/CD pipeline. Audit trail service deployed, every ePHI access, every model inference, every provider action logged.

Week 4: LLM Integration and Prompt Implementation

AWS Bedrock integration. System prompt implemented with clinical constraints. Structured output schema designed, SOAP note JSON schema with confidence scores per field. Grounding validation layer implemented, claim extraction from output, source tracing against transcript. Low-confidence field flagging logic implemented.

Week 5: Clinical Entity Validation Layer

RxNorm medication validation integration. ICD-10 diagnosis code validation. Drug-drug interaction check against patient medication list (requires EHR integration or medication list input). Allergy check against patient allergy list (requires EHR integration or allergy list input). Clinical entity anomaly detection (statistically unusual entities flagged for review).

Week 6: Provider Review Interface

SOAP note review interface with source citations, transcript excerpt shown alongside each note section. Confidence indicators on low-confidence fields. Medication attestation workflow, explicit per-medication confirmation required. Mandatory review flags on ungrounded claims and clinical entity validation failures. Audit log of provider review actions, what was reviewed, what was edited, when the note was signed.

Week 7: EHR Integration for Note Delivery

FHIR R4 Document reference write-back (if target EHR supports FHIR write). HL7 v2 MDM message for legacy EHR note delivery. Epic write-back (if Epic is target EHR, requires App Orchard write-back certification initiated in Week 1). Athenahealth API note creation. Note delivery confirmation and error handling.

Week 8: Mobile Capture Application

Provider-facing mobile app (iOS, Android) for session audio capture. Session start/stop controls. Pre-session setup (patient selection, visit type selection). Session audio encrypted in transit from app to transcription service. No audio stored on device after transmission.

Week 9: Hallucination Monitoring Infrastructure

Provider edit tracking, every field edit logged with original AI value and provider correction value. Edit rate analysis by field type and by clinical context. Alert for statistically elevated edit rates on specific fields. Post-deployment hallucination monitoring dashboard for engineering team.

Week 10: Bias Analysis and Clinical Validation Setup

Demographic subgroup analysis design for clinical validation. IRB submission prepared (if clinical study requires IRB approval). Clinical validation study protocol drafted. Performance benchmarking on de-identified test dataset, overall performance and by subgroup. Any subgroup performance gaps identified and documented.

Week 11: Internal QA and Clinical Advisor Review

Full test suite including hallucination edge cases, transcripts with absent medications, ambiguous diagnoses, inaudible sections, multiple providers speaking. Clinical advisor review of 50 AI-generated notes against reference notes from the same transcripts. Clinical advisor sign-off on prompt architecture and guardrail design. HIPAA compliance review, BAA registry, audit trail completeness, data flow diagram.

Week 12: Security Review and Penetration Testing

Third-party penetration test scoped and initiated. SAST findings reviewed and addressed. Prompt injection testing, attempts to override the system prompt through adversarial patient or provider inputs. Audio data security review, confirm no audio retained beyond 72-hour retention window. SBOM reviewed against NVD for known vulnerabilities.

Week 13: Pilot Deployment and Clinical Governance

First pilot health system: clinical governance submission prepared. Health system IT onboarding guide written. SMART on FHIR credentials issued for the pilot health system. Pilot deployment to 5–10 providers. Daily provider feedback collection during pilot. Any critical issues identified in pilot: fix within 48 hours, re-pilot before broader deployment.

Week 14: Pilot Review and Commercial Launch Preparation

Pilot results reviewed: provider NPS, documentation time reduction, edit rate by field, hallucination incident count, provider-reported issues. Hallucination monitoring findings reviewed: any systemic hallucination patterns requiring prompt or guardrail updates. Clinical governance approval from pilot health system. Commercial launch preparation: customer onboarding documentation, pricing, customer success model.

Human-in-the-Loop Design, Why Every Clinical AI Feature Needs It

Human-in-the-loop (HITL) design is the architectural principle that ensures a human expert reviews and confirms AI outputs before those outputs influence consequential decisions. In clinical AI, HITL is not optional, it is the primary safety mechanism.

The three levels of HITL for clinical AI:

Level 1, Required review before action: The AI generates an output. A human must review and explicitly approve the output before it is acted upon. No AI output can become a clinical record, a patient communication, a clinical recommendation, or a treatment decision without human review and approval.

This is the minimum HITL requirement for any clinical AI that generates content that could influence patient care. The ambient documentation example, provider must review and sign before the note enters the EHR, is Level 1 HITL.

Level 2, Confidence-gated automatic action: The AI generates an output with a confidence score. High-confidence outputs (above a validated threshold) are acted upon automatically. Low-confidence outputs require human review before action.

This level is appropriate for administrative AI with lower clinical stakes, high-confidence prior auth determinations that meet a validated threshold, high-confidence coding suggestions that match a validated clinical pattern. It is not appropriate for clinical AI that directly influences patient care decisions.

Level 3, Supervised automation: The AI acts autonomously within a constrained domain. A human supervisor reviews AI actions in aggregate and can intervene when patterns suggest systematic errors. The AI does not stop and wait for human review of individual actions.

This level is appropriate for very low-stakes administrative tasks, appointment reminder scheduling, routine refill requests for stable chronic condition patients. It is not appropriate for any clinical AI that generates clinical content or clinical recommendations.

HITL and workflow design:

HITL is not just a safety requirement, it is a product design challenge. A HITL design that adds too much friction will be circumvented by clinicians under time pressure. A HITL design that adds too little friction will not catch clinical AI errors.

The design goal is the minimum friction required to ensure meaningful human review. Meaningful review means the clinician is actually reading and evaluating the AI output, not clicking through it reflexively.

Design strategies for meaningful review without excessive friction:

Focused attention direction: the review interface highlights the content that most needs review, ungrounded claims, low-confidence fields, clinical entity flags, so the clinician’s attention is directed to the highest-risk content first.

Contextual source display: showing the source (transcript excerpt, EHR data element) alongside the AI output makes review faster and more accurate than requiring the clinician to recall or look up the source independently.

Progressive attestation: break the review into stages that match clinical reasoning, chief complaint, then history, then assessment, then plan, rather than presenting the full note at once.

Time feedback: show the clinician how long the review has taken and how it compares to their previous reviews. Clinicians who are reviewing very quickly relative to their typical pace may be clicking through without reading.

Post-Launch: Model Monitoring, Drift Detection, and Retraining Under HIPAA

• Model drift, the clinical AI failure mode that happens slowly:

A clinical AI model that performs well at launch may perform worse six months later, not because anything changed in the product, but because the clinical environment changed. New ICD-10 codes, new medications, new clinical protocols, seasonal illness patterns, changes in patient documentation behavior, all of these shift the distribution of clinical data the model encounters in ways that may degrade performance.

Model drift detection requires:

Continuous performance monitoring: metrics collected in real time from production, provider edit rate for ambient documentation, alert accuracy for CDS AI, patient satisfaction and escalation rate for patient-facing AI.

Drift detection algorithms: statistical tests that compare the current distribution of model inputs and outputs to the baseline distribution from validation. Significant distribution shift triggers investigation and potential retraining.

Canary deployments: when a retrained model is deployed, it is first deployed to a small percentage of production traffic (5–10%). Performance is monitored in the canary cohort before full deployment. If performance is worse than the baseline model in the canary cohort, the deployment is rolled back before full production exposure.

• Retraining under HIPAA:

If clinical AI retraining uses production clinical data, provider edits to AI-generated notes, patient interactions with patient-facing AI, EHR data from clinical encounters, that data may be ePHI. Retraining on ePHI requires the same data governance framework as initial training: de-identification where possible, appropriate authorization or DUA where de-identification is not possible, documented data lineage.

Provider edits to AI-generated notes are particularly valuable for retraining ambient documentation models. A provider’s correction of a hallucinated medication is a labeled training example, the AI output was wrong, the provider correction was right. This is high-quality signal for model improvement.

To use provider edits for retraining: obtain appropriate authorization (provider consent to use their edits for model improvement, which should be in the provider agreement), de-identify the associated clinical content where possible, document the data use in your privacy policy.

• The PCCP for clinical AI that requires FDA clearance:

For clinical AI products that require FDA clearance, the Predetermined Change Control Plan is the mechanism for pre-approving algorithm updates. Include in the PCCP:

• Retraining on expanded clinical datasets with the same clinical indication
• Retraining on datasets from new clinical sites or geographic regions
• Updating preprocessing pipelines that do not change the model architecture
• Adjusting decision thresholds within validated performance ranges

The PCCP makes algorithm improvement operationally feasible without requiring a new FDA submission for every update. It must be negotiated with the FDA in the original submission, include everything you might want to change. The cost of adding a PCCP element after clearance is a new submission. The cost of including it in the original PCCP is the regulatory attorney’s time to write it.

When an Indian Engineering Partner Is Wrong for Your Clinical AI Build

An Indian engineering partner is the wrong call for your clinical AI product if: your clinical AI development involves daily collaboration with clinical advisors who are embedded in the engineering process and available only during US clinical hours, if the clinical review cadence is synchronous and spontaneous in a way that the overlap window cannot accommodate.

If your health system customers require that all AI model training and inference occur on-premises within the health system’s own infrastructure, some academic medical centers and federal health systems have this requirement, and it means the engineering team must be able to work within the health system’s network, which may restrict offshore access.

If your clinical AI product is in an FDA SaMD category where your regulatory attorney has advised that all development personnel have formal FDA-regulated environment training and documentation, an uncommon requirement, but one that some high-risk device developers impose.

If your clinical AI requires real-time collaboration between engineers and clinical staff responding to patient interactions, for example, a clinical AI product with a human escalation function where the engineering team must be responsive to real-time clinical escalations.

For the vast majority of clinical AI founders building ambient documentation tools, CDS AI, patient-facing health navigation, or administrative healthcare AI: the structured collaboration model is viable.

Clinical advisors embedded on the US side, engineering team in Indore with defined US-overlap hours, model training on de-identified data in HIPAA-compliant cloud infrastructure. We have built clinical AI products from Indore that are deployed in US health systems today.

The Clinical AI Product Scorecard™

Score each row 0 (absent), 1 (partial), or 2 (fully present). Maximum score: 70.

#	Criterion	Weight	Your Score
1	Regulatory attorney written opinion on FDA SaMD classification (or CDS exemption documentation)	2×	/4
2	BAA confirmed for every service in the ePHI processing pipeline	2×	/4
3	LLM provider BAA executed and scope reviewed by healthcare attorney	2×	/4
4	System prompt with explicit hallucination prevention constraints	2×	/4
5	Grounding validation layer (AI output claims traced to source data)	2×	/4
6	Clinical entity validation (medication RxNorm check, allergy check, ICD-10 check)	2×	/4
7	Required human review gate before any AI output enters clinical record	2×	/4
8	Source citations displayed alongside AI output in review interface	2×	/4
9	Confidence indicators on low-confidence AI output fields	1×	/2
10	Crisis escalation pathway for patient-facing AI	2×	/4
11	Bias analysis by demographic subgroup before clinical deployment	2×	/4
12	Model version registry with performance metrics and deployment dates	1×	/2
13	Every clinical AI output attributable to specific model version	2×	/4
14	Provider edit rate monitoring post-deployment	1×	/2
15	Audio retention policy documented and automated deletion implemented (if audio captured)	1×	/2
16	Prompt logging disabled in production or prompt logging service has HIPAA BAA	1×	/2
17	Training data governance policy (de-identification method, authorization basis)	2×	/4
18	Minimum necessary data access manifest per AI feature	1×	/2
19	Clinical advisor review of AI feature outputs before production deployment	2×	/4
20	SBOM generated in CI/CD pipeline	1×	/2
21	Incident response plan for clinical AI failure (hallucination in clinical record)	1×	/2
22	Retraining data governance documented (if using production data for retraining)	1×	/2
23	PCCP included in FDA submission (if AI/ML SaMD requiring clearance)	1×	/2
24	Real-world performance monitoring infrastructure operational at launch	1×	/2
25	Health literacy design for patient-facing AI (sixth-grade reading level or below)	1×	/2

Score interpretation:

• 55–70: Strong clinical AI safety and compliance posture, ready for health system deployment and enterprise sales
• 40–54: Proceed with identified gaps remediated, patient safety 2× items are non-negotiable
• Under 40: Significant patient safety and regulatory exposure, do not deploy clinically until gaps are closed

Conclusion

Clinical AI is the highest-stakes software category in health tech. The ambient documentation product that gets it right saves a physician fourteen minutes per patient and hundreds of hours per year.

The product that gets it wrong, the hallucinated NSAID prescription that reaches a patient allergic to NSAIDs, the CDS alert that fires on incorrect data, the patient-facing AI that fails to escalate a suicidal patient to crisis resources, does not get a second chance with the patient who was harmed.

The founders who build clinical AI correctly understand that the guardrail architecture is the product. Not a safety feature added after the core product is built. The foundation the core product is built on.

The LLM is a powerful, unreliable collaborator. Your job is to build the architecture that makes its unreliability clinically acceptable, source citation, grounding validation, clinical entity validation, confidence scoring, mandatory human review, so that the fourteen minutes saved per encounter are genuinely saved, and no patient receives a hallucinated clinical decision as a result.

I have been on 2,000+ calls with US healthcare founders since 2014. The clinical AI founders who succeed are the ones who treat patient safety as the product constraint, the thing that shapes every architecture decision, not as the compliance requirement they address before launch.

If you want 30 minutes to talk through your clinical AI product, which LLM, which HIPAA architecture, what guardrails, what regulatory exposure, book a call with me or Aditi. No slides. No pitch. Just the product conversation.

FAQ

• Can I use ChatGPT or GPT-4 API for a HIPAA-compliant clinical AI product?

Yes, under specific conditions. OpenAI offers a Business Associate Agreement under an enterprise agreement. The BAA covers the API (chat completions, embeddings) under the enterprise agreement terms. The BAA scope must be reviewed by your healthcare attorney to confirm it covers your specific use case. Standard API plans (pay-as-you-go) do not include a BAA. If you are processing ePHI through the OpenAI API without an enterprise BAA, you are in violation of HIPAA.

• What is the difference between AWS Bedrock and using the Anthropic API directly for clinical AI?

AWS Bedrock provides access to Anthropic’s Claude models (and other models) through AWS’s infrastructure, covered under the standard AWS HIPAA BAA without requiring a separate enterprise agreement with Anthropic. The Anthropic API directly requires a separate enterprise BAA negotiation with Anthropic. For most clinical AI products, AWS Bedrock is faster to compliance and operationally simpler, one BAA (AWS) covers both the cloud infrastructure and the LLM. The trade-off is that Bedrock may not offer the latest Claude model versions as quickly as Anthropic’s direct API.

• How do I prevent LLM hallucinations in clinical AI outputs?

No single mechanism eliminates LLM hallucinations. A layered guardrail architecture reduces them to a clinically acceptable frequency: (1) prompt engineering constraints prohibiting fabrication and requiring uncertainty markers, (2) grounding validation that traces AI output claims to source data, (3) clinical entity validation checking medications against RxNorm, allergies against patient records, and diagnoses against ICD-10, (4) confidence scoring that flags low-confidence outputs for mandatory provider review, and (5) a required human review gate before any AI output enters a clinical record. Operate all five layers simultaneously.

• Does an AI clinical scribe need FDA clearance?

An AI clinical scribe that generates clinical documentation from provider-patient conversations is generally not a medical device under FDA definitions, it is generating documentation of what the provider said, not making independent clinical decisions. However, the line between documentation and clinical decision support is not always clear. Get a regulatory attorney’s written CDS exemption analysis before asserting that your product does not require FDA engagement. The analysis is particularly important if your scribe adds clinical interpretation, suggests diagnoses, or recommends treatments beyond what was explicitly discussed in the conversation.

• What is the minimum necessary principle and how does it apply to clinical AI?

The HIPAA minimum necessary principle (45 CFR §164.502(b)) requires that ePHI access be limited to the minimum necessary to accomplish the intended purpose. For clinical AI, this means: your AI system should access only the patient data elements required for the specific AI function being performed. An ambient documentation AI that needs the patient’s medication list to contextualize visit documentation does not need access to the patient’s psychiatric history, HIV status, or SUD records. Implement a data access manifest for each AI feature listing the specific data elements accessed and why each is necessary.

• How do I handle clinical AI model bias?

Clinical AI bias, disparate performance across demographic subgroups, requires a multi-step approach. Before deployment: analyze model performance by age group, sex, race/ethnicity, primary language, insurance type, and any other clinically relevant demographic. After deployment: monitor real-world performance by subgroup continuously. For identified performance gaps: investigate the root cause (training data underrepresentation, feature disparities, labeling bias), retrain on more representative data, and validate the retrained model’s subgroup performance before deployment. Document all bias analysis in your model cards and clinical validation reports.

• What data can I use to train or fine-tune a clinical AI model?

The preferred training data source is de-identified clinical data, data from which all 18 HIPAA identifiers have been removed using the Safe Harbor or Expert Determination method. De-identified data is not ePHI and can be used for model training without HIPAA restrictions. Using identified ePHI for model training requires either patient authorization or a determination that the use falls within healthcare operations, which is not always clear for model training purposes. Get a healthcare attorney’s opinion before using identified ePHI for training data.

• How should a patient-facing AI handle suicidal ideation or mental health crises?

The crisis escalation architecture must be built before the first patient interaction. When crisis language is detected, by keyword matching, semantic analysis, or both, the product must: immediately surface 988 Suicide and Crisis Lifeline contact information, Crisis Text Line, and local emergency services within the product; if the platform has clinical staff, create an escalation notification to the clinical support team; log the crisis event with timestamp, patient content, AI response, and escalation status; and not attempt to provide clinical crisis management through the AI. The AI detects and escalates. A human clinical pathway manages the crisis.

• What is prompt injection and how does it affect clinical AI security?

Prompt injection is an attack where malicious content in the AI’s input, a patient message, a clinical note, an EHR data field, contains instructions that override the AI’s system prompt. In a clinical AI context, a prompt injection attack could cause the AI to: generate fabricated clinical content, bypass hallucination guardrails, disclose ePHI from the current context window, or perform unintended actions. Defenses: test your system with adversarial prompt injection inputs as part of security testing, implement input sanitization for external data sources, use structured input formats that reduce injection risk, and monitor for unusual AI output patterns that may indicate a successful injection.

• How do I handle clinical AI outputs in multiple languages for diverse patient populations?

Patient-facing AI for diverse populations must support the patient’s preferred language. The implementation options: (1) use a multilingual LLM (Claude, GPT-4, Gemini support many languages natively) with a language detection step that switches the response language to match the patient’s input, (2) maintain separate system prompts and clinical content in each supported language with clinical review of each language version, (3) for languages where LLM performance is less validated clinically, route to human language interpretation services rather than relying on AI-generated health content. Clinical content in any language must be reviewed by a bilingual clinical advisor for accuracy and health literacy appropriateness.

• What is the difference between fine-tuning and RAG for clinical AI?

Fine-tuning trains the LLM on a clinical dataset, embedding clinical knowledge into the model weights. RAG (Retrieval-Augmented Generation) retrieves relevant clinical documents at inference time and includes them in the prompt as context, without changing the model weights. For most clinical AI use cases, RAG is preferable to fine-tuning for three reasons: (1) RAG does not require ePHI in training data, you can use a general-purpose LLM with de-identified clinical knowledge in the retrieval database, (2) RAG knowledge can be updated without retraining, you update the retrieval database, not the model, (3) RAG is more transparent, the retrieved documents are visible in the prompt, making it easier to trace the source of AI outputs. Fine-tuning is appropriate when the clinical domain requires specialized language patterns or reasoning that the base LLM does not handle well, and when clean de-identified training data is available.