A live AI medical scribe transcribes a clinical encounter as it happens and drafts the SOAP note while the patient is still in the room. It then finalizes that note from a full-context pass once the visit ends. The live path gives the clinician something to check. The record path gives them something to sign.
Most scribe tutorials, including our step-by-step post-visit tutorial, process the recording after the visit. That's the right default. But clinicians increasingly want to glance at the draft before the patient leaves. They want to catch a missing allergy while they can still ask about it, and to stop writing notes at 9pm.
So this post is the live version: the architecture, the code, and the seams where it breaks. For the category view (what an ambient AI scribe is and how to evaluate one), start there instead.
Two numbers to anchor on. Medical Mode cuts medical entity errors by 87% versus the same base model, and it's one parameter. The whole speech layer for a 15-minute visit, live path and record path together, costs about twenty-seven cents. The interesting engineering is the orchestration, not the ASR.
The architecture: two paths, one note
A live scribe runs two transcription paths over the same audio. They have different jobs.
| Path | Model | Job | Output |
|---|---|---|---|
| Live | AssemblyAI-STT-3.7-Realtime + Medical Mode | Show the transcript and a rolling SOAP draft during the visit | A draft the clinician can glance at and correct |
| Record | AssemblyAI-STT-3.7 + Medical Mode | Re-transcribe the full recording with full-context diarization and entities | The note of record, sent for signature |
Why not just finalize the live draft? Because streaming makes decisions with partial context. Speaker labels get revised. A drug name said once in minute three gets clarified in minute twelve. The pre-recorded pass sees the whole encounter at once, and its per-hour price is the lower of the two: $0.21/hr, against $0.30/hr for AssemblyAI-STT-3.7-Realtime. If you only need that path, the one-request medical note-taker is the simplest version of it.
Here's the full pipeline:
| Stage | Responsibility | Main failure mode |
|---|---|---|
| 1. Capture | Clean audio from the room, streamed and buffered | Mic distance, dropped segments, wifi loss |
| 2. Live transcription | Speaker-labeled turns with medical vocabulary intact | Misheard drug names, fragmented turns |
| 3. Live SOAP draft | Update the note as the encounter unfolds | Inventing findings from half a conversation |
| 4. Record pass | Full-context transcript and final note | Drift between the draft and the final note |
| 5. Redact and deliver | Redact, persist, push to the EHR for signature | Unredacted PHI in retained audio |
Stage 1: capture
Nothing downstream recovers from bad audio. Three decisions matter.
Sample rate and format. 16 kHz mono is enough for speech. Avoid aggressive lossy compression, which strips the consonant detail that separates similar drug names.
Mic placement, and telling the model about it. A badge clip is a different acoustic problem from a tablet across the room. Test with the hardware you ship, not the laptop you develop on. Voice Focus, which tunes for near-field or far-field mics, runs on AssemblyAI-STT-3.7-Realtime-Agent and Universal-3.6 Pro Realtime, not on the AssemblyAI-STT-3.7-Realtime model this live path uses.
Stream and buffer at the same time. Send audio to the streaming endpoint and write it to local storage in parallel. Clinic wifi drops. If the stream breaks, the live draft pauses, but the record pass still has every second.
Stage 2: live transcription with Medical Mode
The live path runs on AssemblyAI-STT-3.7-Realtime, AssemblyAI's streaming model for notetakers and ambient transcription. Medical Mode is the same one parameter as on pre-recorded audio, domain="medical-v1".
import assemblyai as aai
import pyaudio
from assemblyai.streaming.v3 import (
StreamingClient,
StreamingClientOptions,
StreamingEvents,
StreamingParameters,
TurnEvent,
)
API_KEY = "YOUR_API_KEY"
turns = [] # finalized, speaker-labeled turns for the live draft
def on_turn(client, event: TurnEvent):
if not event.end_of_turn:
return # partials feed the on-screen transcript only
# One turn can hold two speakers, so split it on word-level labels
segment = []
for word in event.words:
speaker = word.speaker or event.speaker_label
if segment and segment[-1]["speaker"] == speaker:
segment[-1]["text"] += f" {word.text}"
else:
segment.append({"speaker": speaker, "text": word.text})
turns.extend(segment)
def mic_stream(sample_rate=16000, chunk_ms=100):
frames = sample_rate * chunk_ms // 1000
pa = pyaudio.PyAudio()
mic = pa.open(format=pyaudio.paInt16, channels=1, rate=sample_rate,
input=True, frames_per_buffer=frames)
try:
while True:
yield mic.read(frames, exception_on_overflow=False)
finally:
mic.close()
pa.terminate()
client = StreamingClient(
StreamingClientOptions(api_key=API_KEY, api_host="streaming.assemblyai.com")
)
client.on(StreamingEvents.Turn, on_turn)
client.connect(
StreamingParameters(
sample_rate=16000,
speech_model="assemblyai-stt-3-7-realtime",
domain="medical-v1",
speaker_labels=True,
max_speakers=3,
min_turn_silence=800,
max_turn_silence=2000,
)
)
client.stream(mic_stream()) # 100 ms chunks of 16 kHz PCMFour things about that config.
Singular speech_model on streaming
Streaming takes speech_model as a string. Pre-recorded takes speech_models as a list. It's a small difference that costs people an afternoon.
Turn silence tuned for clinicians
Clinicians pause mid-sentence to read a chart. The default turn settings are built for fast voice-agent dialogue, and they'll split those pauses into separate turns. The streaming Medical Mode docs recommend min_turn_silence of 800 ms. For a live scribe they suggest a max_turn_silence around 2,000 ms, lower than the 3,600 ms they suggest for dictation, so the draft keeps moving.
max_speakers as a hard cap
Clinician, patient, and maybe a family member. max_speakers is a ceiling, not a hint. Give it one slot of headroom above the speakers you expect, or a third voice gets merged into the patient's.
Speaker labels get revised
Streaming diarization labels each turn live, then sends a correction when the session ends. Your live draft has to tolerate a turn changing owners after the fact. That's one reason the record pass exists. A single turn can also hold both speakers, when one answers inside the 800 ms silence window, so the handler above splits each final turn on the word-level speaker field instead of trusting the turn-level speaker_label. Very short replies like “Hi, Doctor” come back as PENDING until the model has enough audio to attribute them.
Sign up free and stream a mock encounter with Medical Mode on. Watch the drug names land before you write any orchestration.
Stage 3: live SOAP notes during the encounter
This is the part that makes a scribe feel live. It's also where the most dangerous mistakes happen. A model drafting a note from half a conversation will happily write an assessment the clinician hasn't made yet.
The pattern that works is a rolling draft. Keep a JSON note with four sections. Every few finalized turns, send the current draft plus only the new turns to a small, fast model. Ask it to update just the sections the new turns support.
import json
import time
import requests
DRAFT_RULES = """You maintain a DRAFT SOAP note during a live clinical encounter.
You receive the current draft as JSON and the newest transcript turns.
Rules:
- Update only sections the new turns support. Never infer a diagnosis,
dosage, or finding that was not stated.
- Leave a section as "Not yet discussed." until the transcript supports it.
- Attribute symptoms to the patient and assessments to the clinician,
using the speaker roles provided.
- Copy medication names and doses exactly as transcribed.
Return JSON only:
{"subjective": "...", "objective": "...", "assessment": "...", "plan": "..."}"""
draft = {k: "Not yet discussed." for k in ("subjective", "objective", "assessment", "plan")}
sent = 0
def update_draft(draft, new_turns, roles):
lines = "\n".join(
f'{roles.get(t["speaker"], t["speaker"])}: {t["text"]}' for t in new_turns
)
resp = requests.post(
"https://llm-gateway.assemblyai.com/v1/chat/completions",
headers={"authorization": API_KEY},
json={
"model": "qwen3.5-4b-32k-fast",
"messages": [
{"role": "system", "content": DRAFT_RULES},
{"role": "user", "content": f"Current draft:\n{json.dumps(draft)}\n\nNew turns:\n{lines}"},
],
},
timeout=10,
)
resp.raise_for_status()
return json.loads(resp.json()["choices"][0]["message"]["content"])
roles = {"A": "Clinician", "B": "Patient"} # resolve these in your app
while client_is_streaming(): # your session state
time.sleep(15)
new_turns = turns[sent:]
if not new_turns:
continue
try:
draft = update_draft(draft, new_turns, roles)
sent += len(new_turns)
render_draft(draft) # your UI
except (requests.RequestException, ValueError, KeyError):
pass # keep the last good draft; the next cycle retriesA few design choices are doing the real work here.
Update on a cadence, not on every turn
Updating on every turn makes the draft flicker and wastes calls on "mm-hm". A 15-second cycle, or every three to five final turns, keeps the draft current without making it jumpy. The clinician is glancing, not reading along.
Use a small model for the draft
The rolling draft is a preview. Speed and cost matter more than prose quality. Qwen3.5 4B Fast on LLM Gateway is hosted by AssemblyAI and priced at $0.10 per million input tokens and $0.50 per million output tokens. Thirty updates over a 15-minute visit cost around a cent or two. Save the larger model for the final note.
"Not yet discussed" is a feature
An empty Plan section halfway through the visit is correct. Make the UI show it as pending rather than missing. The worst live draft is one that fills the Plan section with a plausible guess the clinician never said.
Keep the draft visibly a draft
Label it, grey it, and never let it flow into the EHR directly. The live draft's job is to prompt the clinician ("I didn't ask about allergies"). It is not the record.
Let the clinician correct it in place
If the clinician edits a line during the visit, treat that edit as ground truth. Pass it into the next update as locked text the model can't overwrite. It's also the strongest signal you'll get for the final note.
Stage 4: the record pass and the final note
When the visit ends, re-transcribe the buffered recording with AssemblyAI-STT-3.7 and Medical Mode. This pass sees the whole encounter at once. That means better diarization on short turns and overlapped speech, and better entity accuracy.
config = aai.TranscriptionConfig(
speech_models=["assemblyai-stt-3-7"],
domain="medical-v1",
speaker_labels=True,
entity_detection=True,
)
transcript = aai.Transcriber(config=config).transcribe("./encounters/enc-40118.wav")
if transcript.status == "error":
raise RuntimeError(transcript.error)Pass context if you have it. Feeding the patient's prior-visit note into the request cut missed medical terms by 31% in an internal healthcare test. Your app already has the chart open to write the note, so passing it costs one field.
Then generate the final note from the full transcript, the typed entities, and the clinician's in-visit edits. Use the same safety rules as the draft, plus two more:
- Final-note rules: use only what's in the transcript, write "Not documented in this encounter" for empty sections, and don't normalize brand names to generic.
- Reconcile, don't overwrite: where the final note disagrees with a line the clinician corrected during the visit, keep the clinician's line and flag the conflict.
- Validate structure: ask for JSON, validate it against a schema, and route failures to a human instead of patching them with another model call.
AssemblyAI-STT-3.7 ships with a new speaker diarization model. On English conversations it has the lowest diarization error rate of the pre-recorded APIs we measured, 10.7%, ahead of ElevenLabs Scribe v3 at 12.0%, and against Universal-3.5 Pro at launch it cuts cpWER (concatenated minimum-permutation word error rate) by 20% (see the AssemblyAI-STT-3.7 research post). cpWER scores the words in each speaker's transcript, which is exactly what a note generator consumes. If your final note ever attributes a patient's symptom to the clinician, this is where it came from. Our guide to measuring medical transcription covers why entity metrics beat aggregate WER for this workload.
Upload an encounter, turn on Medical Mode and diarization, and inspect the JSON the record pass returns.
Stage 5: redaction and delivery
Redact both the transcript and the audio. If you keep the recording, it contains a voice saying the patient's name and date of birth, and a redacted transcript doesn't change that. Transcript redaction costs +$0.08/hr and audio redaction +$0.05/hr on pre-recorded audio.
Scope it on purpose. The clinical note needs condition and medication text, so redact those only from what goes to analytics or long-term storage. Our walkthrough on redacting PII from audio and video shows how to turn redaction on; apply it per destination.
Deliver the final note for signature, not for filing. Keep transcript timestamps attached to each section so a reviewer can jump to the audio behind any line in one click. That single affordance does more for clinician trust than any accuracy number.
What breaks in production
The draft and the final note disagree. Expect it. Show the clinician a short diff for anything the record pass changed, especially medications.
The stream drops mid-visit. Pause the draft, keep recording locally, and reconnect. The record pass is your safety net.
Multilingual encounters. AssemblyAI-STT-3.7-Realtime and AssemblyAI-STT-3.7 both code-switch across 32 languages. Medical Mode covers English, Spanish, German, and French. A Spanish-English visit gets both. A Hindi-English visit gets the code-switching without the Medical Mode entity boost.
Long encounters. A 90-minute intake overflows a single note-generation call. Chunk on speaker-turn boundaries with overlap and reconcile section by section.
Consent state. Whether the patient consented to recording is application state. The pipeline should refuse to start without it, and stop cleanly if consent is withdrawn mid-visit.
Compliance, briefly
AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and is considered a business associate under HIPAA. It maintains SOC 2 Type 2, ISO 27001:2022, and PCI DSS v4.0, and audio submitted under the BAA isn't used for model training. Paid accounts sign the BAA self-serve; details are in the BAA FAQ. If you sell into health systems with residency rules, ask about EU routing and self-hosted deployment before you architect, not after.
What it costs per encounter
For a 15-minute visit at pay-as-you-go rates (November 2026):
| Component | Rate | 15-minute visit |
|---|---|---|
| Live path: AssemblyAI-STT-3.7-Realtime + Medical Mode + streaming diarization | $0.30 + $0.15 + $0.12 = $0.57/hr | about $0.14 |
| Record path: AssemblyAI-STT-3.7 + Medical Mode + diarization | $0.21 + $0.15 + $0.02 = $0.38/hr | about $0.10 |
| Redaction (transcript + audio) | $0.08 + $0.05 = $0.13/hr | about $0.03 |
| Rolling draft (Qwen3.5 4B Fast) | Per token | a cent or two |
That's roughly twenty-seven cents of speech processing, plus the final-note LLM call. Drop the live path and you're back near thirteen cents. Full rates are on the pricing page.
Teams shipping clinical documentation on this stack include Commure, Sully AI, Heidi Health, Deepscribe, Knowtex, and Magentus Healthcare.
Where this goes
Today the live draft and the final note are two passes, and the seam between them is where clinician trust is won or lost. That seam is going to shrink. Streaming models are closing the accuracy gap with full-context passes, and the context that improves transcription also improves note structure.
Design the two-path architecture so you can collapse it when the gap closes. Whatever happens to the models, keep the timestamp links to source audio and keep the clinician's in-visit edits as ground truth. The model will keep improving. The clinician's ability to check it is the part that has to survive every upgrade.
Talk to our team about BAA terms, self-hosted and EU-residency deployment, and volume pricing for scribe workloads.
Frequently asked questions
Can an AI medical scribe generate SOAP notes in real time?
Yes. Stream the encounter through a realtime model with Medical Mode, and update a rolling SOAP draft with a small, fast LLM every few finalized turns. Treat that draft as a preview. Finalize the note of record from a full-context pre-recorded pass after the visit.
How much does an AI medical scribe cost per encounter?
On AssemblyAI, the speech layer for a 15-minute visit costs about $0.10 for the pre-recorded record path with Medical Mode and diarization. Adding a live transcript and rolling draft brings it to roughly $0.25 before the final-note LLM call. Rates are pay-as-you-go with no minimums.
Should I use streaming or pre-recorded transcription for a medical scribe?
Use pre-recorded for the note of record, because it sees the whole encounter and runs at $0.21/hr, against $0.30/hr for AssemblyAI-STT-3.7-Realtime. Add streaming only when clinicians need a live transcript or draft during the visit. Running both is common: streaming for the live view, pre-recorded for the record.
How do I keep a live SOAP draft from inventing findings?
Constrain the prompt: update only sections the new turns support, write "Not yet discussed" otherwise, and never infer a diagnosis or dosage. Keep the draft visibly labeled as a draft and never send it to the EHR directly. The final note comes from a separate full-context pass.
Can AI scribes be used with PHI under HIPAA?
Yes, when every vendor that processes PHI signs a BAA and your own system handles access, retention, and consent correctly. AssemblyAI is considered a business associate under HIPAA and signs a standard Business Associate Addendum (BAA) for customers processing PHI. It also offers PHI redaction across audio and transcripts.
How does speaker diarization separate the doctor from the patient?
Diarization returns anonymous labels such as A and B, and your application maps them to roles. Use signals like who speaks first, who asks most questions, and who produces most clinical entities. Streaming labels can be revised when the session ends, so final role mapping should use the pre-recorded pass.