6 Ways Telehealth Platforms Can Leverage Speech-to-Text AI

A telehealth visit is already a digital audio stream. The patient's voice and the clinician's voice arrive at your servers as packets. You are, whether you've built anything on it or not, sitting on the cleanest, most structured clinical audio pipeline in healthcare — no room mic, no ambient noise from the hallway, no hardware to install in 400 exam rooms.

That's the asymmetry worth acting on. An in-person practice that wants ambient documentation has to solve capture first. You've already solved capture. What's left is deciding what to do with the stream, and the answer has changed considerably in the last couple of years: transcription got accurate enough on medical vocabulary to trust in a chart, streaming latency dropped low enough for live interaction, and the cost per hour fell far enough that running it on every visit stopped being a budget conversation.

Here are six things virtual care platforms are doing with speech to text right now, with the implementation details and the actual per-hour costs. Some are documentation. Some are revenue. One of them replaces a phone tree.

1. Live transcription during the visit

The simplest use, and the one patients notice. A running transcript in the clinician's view — and optionally the patient's — while the visit happens.

This does more than it looks like it does. It gives the clinician something to scan when a patient mentions a medication in passing at minute two and the plan gets written at minute eleven. It gives hearing-impaired and hard-of-hearing patients real access to the conversation without a third party on the call. And it's the substrate everything else in this list runs on.

What it takes technically

Universal-3.5 Pro Realtime streams over a single WebSocket at wss://streaming.assemblyai.com/v3/ws, at $0.45/hr base. Turn detection defaults to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset, which is what makes a live transcript feel like it's keeping up rather than lagging behind. Three modes — min_latency, balanced, and max_accuracy — let you decide where to sit on that tradeoff. For a transcript a human reads, use max_accuracy; nobody is measuring milliseconds on a display.

Add domain: "medical-v1" and you get Medical Mode's clinical vocabulary in the live stream, at +$0.15/hr for $0.60/hr all in.

import assemblyai as aai

client = aai.streaming.v3.StreamingClient(
    aai.streaming.v3.StreamingClientOptions(
        api_key="YOUR_API_KEY",
        api_host="streaming.assemblyai.com",
    )
)

client.connect(
    aai.streaming.v3.StreamingParameters(
        sample_rate=16000, speech_model="universal-3-5-pro",
        domain="medical-v1",
        speaker_labels=True,
        mode="max_accuracy",
        voice_focus="near-field",
    )
)

Telehealth audio is near-field by definition — headset or laptop mic, close to the speaker — so set voice_focus accordingly. Note streaming uses speech_model singular; pre-recorded requests use speech_models plural.

2. Ambient note generation from the visit recording

The highest-value item on this list, measured in clinician time. Transcribe the visit, then have a language model turn the transcript into a structured note the clinician reviews and signs.

The quality ceiling here is set entirely by the transcript. A structuring model handed "Celexa" when the clinician said "Celebrex" will write "Celexa" into the medication list with complete confidence. No prompt fixes that.

Measure entity accuracy, not word accuracy

This is where most evaluations go wrong. Word error rate weights "um" the same as "metoprolol," and clinical conversation is mostly filler by volume — so a model can post a great WER while missing a meaningful share of the drugs and conditions. The metric that predicts whether a note is usable is Missed Entity Rate. We laid out the full argument in WER vs MER for medical transcription.

Universal-3.5 Pro with Medical Mode records a 3.2% Missed Entity Rate, the lowest across the providers we've benchmarked. Methodology is on the benchmarks page. Against the base model with Medical Mode off, the domain produces roughly 20% fewer missed medical entities and 87% fewer entity errors.

Pass the chart along with the audio

Contextual prompting is the most underused feature in clinical transcription. Feeding a patient's prior-visit note into the request cut missed medical terms by 31% in an internal healthcare test. Your telehealth platform has the chart open — it needed it to render the visit. Passing it costs you one field and buys accuracy you'd otherwise chase with vocabulary lists.

Pre-recorded transcription with Medical Mode is $0.36/hr. On a 15-minute visit that's nine cents.

3. Voice agents for intake, scheduling and follow-up

This is the newest item on the list and the one that changes your cost structure rather than your clinician workflow. A voice agent handles the calls around the visit: intake questions, insurance verification, appointment scheduling and rescheduling, medication reminders, post-discharge check-ins.

Historically this meant stitching together three vendors — speech to text, a language model, text to speech — and eating the latency of every hop plus the operational cost of three integrations. The Voice Agent API collapses that into one WebSocket at a flat $4.50/hr, which is the number to compare against a phone queue rather than against a transcription line item.

What makes a medical voice agent work

Two things. First, entity accuracy on the way in — an agent that mishears a medication name during a refill request creates a clinical problem, not a customer service problem. Second, turn-taking that doesn't interrupt an elderly patient mid-sentence. The streaming model's turn detection, tunable through min_turn_silence and max_turn_silence, is the mechanism there.

If you're building the pipeline yourself rather than using the Voice Agent API, agent_context on Universal-3.5 Pro Realtime is worth knowing about: it feeds the reply your agent just spoke back into the decode, and passing context cut WER 10.2% across 20,000 voice agent files, with detailed context cutting medical-term entity errors 43%. That's a meaningful gain for a parameter.

4. Behavioral health session documentation

Teletherapy is the single best fit for speech AI in virtual care, and also the most sensitive. Sessions are long, verbal, and almost entirely conversation — there's no physical exam, no imaging, no labs. The note is the entire clinical record.

The technical requirement that dominates here is speaker separation. In a therapy session the clinician and patient trade short turns, talk over each other, and pause in ways that break naive diarization. A note that attributes a patient's disclosure to the therapist isn't just wrong, it's clinically misleading.

Why diarization measurement matters

Diarization Error Rate scores how much audio time got assigned to the wrong speaker, which rewards handling long monologues and barely penalizes dropping a two-word turn. Concatenated minimum-permutation WER — cpWER — scores the words in each speaker's transcript, which is what a summarization step consumes. Universal-3.5 Pro's diarization is optimized for cpWER rather than DER, and it's the most accurate we've shipped specifically on short turns and overlapped speech. Streaming diarization includes revision, correcting earlier speaker assignments as more context arrives, for up to 10 speakers — useful for group and family sessions.

Sentiment and topic analysis over the transcript can also flag sessions for supervisory review, though that's a clinical governance decision rather than a technical one.

5. Multilingual visits without a third party on the line

Virtual care removed the geographic constraint on who a patient can see, which means language mismatch shows up more often, not less. And the actual pattern in bilingual households isn't "the visit is in Spanish" — it's code-switching mid-sentence, with a family member translating in fragments.

Universal-3.5 Pro code-switches natively across 18 languages with no configuration and no per-utterance language hinting. A patient moving between languages inside one sentence stays intelligible in the transcript.

Medical Mode itself covers English, Spanish, German and French. Those are two distinct facts and worth keeping separate when you scope coverage: a Spanish-English visit gets both native code-switching and clinical entity accuracy; a Vietnamese-English visit gets the code-switching without the Medical Mode entity boost.

6. Quality review and operational analytics across visits

Once every visit produces a transcript, you have a corpus. That corpus answers questions your product analytics can't: which conditions are driving repeat visits, where clinicians are spending time that could be handled asynchronously, whether a new intake script is actually shortening visits, which patient concerns come up but don't make it into notes.

Speech Understanding gives you the pieces — entity detection across 50+ types, sentiment, topic detection, summarization — and the LLM Gateway lets you ask open-ended questions across many transcripts at once rather than building a separate analysis pipeline.

The practical constraint is that this is the use case most likely to touch PHI at scale, which is the next section.

PHI handling for all six of these

Everything above produces or stores protected health information, so this isn't an appendix — it's a design input.

Redaction works in both directions, and teams routinely forget the second one. Transcript text redaction is +$0.08/hr. Audio redaction — replacing the spoken PHI in the retained recording — is +$0.05/hr. If you keep visit recordings, the recording contains a voice saying the patient's name and date of birth, and redacting only the transcript leaves that exposed. Entity detection covering 50+ types is what drives both. The implementation detail is in our walkthrough on redacting PHI from medical transcripts.

On the contracting side: AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, acting as a business associate under HIPAA, and maintains SOC 2 Type 2. Files aren't used to train models and aren't shared with third parties. Details are in the BAA FAQ and on the BAA page. Ask about data residency and self-hosted deployment early if you're selling into health systems that require either — retrofitting is expensive.

What each of these costs per hour

A 15-minute visit with ambient documentation and transcript redaction runs about eleven cents. That's the number that makes "transcribe every visit" a default rather than a pilot. Current rates are always on the pricing page.

Where to start if you're building this

Start with number 2 — ambient notes from the recording — because it's pre-recorded, has no latency requirement, and delivers the clearest clinician-time win. Get entity accuracy right there, on your own audio, with your own annotators. Then add the live transcript, which reuses the same configuration over a WebSocket. Voice agents are a separate product decision and should be scoped as one.

Teams building virtual care and clinical documentation products on AssemblyAI include Commure, Heidi Health, Sully AI, Knowtex and Chapter.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

For the deeper builds: the end-to-end medical scribe architecture has working code, the ambient scribe evaluation guide covers vendor selection, and medical transcription use cases and the speech-to-text product page cover the rest of the surface. API reference is in the docs.

Conclusion

Every item on this list treats the visit audio as something to record and process afterward — even the live transcript is really just a faster read-only view. The next step is audio that participates: a system that notices the clinician hasn't confirmed an allergy, or that the patient described a symptom that contradicts the chart, and says so during the visit rather than in a report afterward. Telehealth platforms will get there before in-person practices do, for the same structural reason they should be doing all six of these already — the audio is already in the pipe, and nobody has to install anything in an exam room to get it. The constraint was never capture. It's how much you trust the transcript, which is a decision you can make with data.

Frequently asked questions

What's the best speech-to-text API for telehealth?

Evaluate on medical entity accuracy and speaker separation, not word error rate. Universal-3.5 Pro with Medical Mode records 3.2% Missed Entity Rate — the lowest across benchmarked providers — with cpWER-optimized diarization, native code-switching across 18 languages, PHI redaction for both audio and transcripts, and a BAA available. Reproduce the numbers on your own visit recordings before committing.

Can speech to text run live during a telehealth visit?

Yes. Universal-3.5 Pro Realtime streams over a single WebSocket at $0.45/hr base, with turn detection defaulting to min_turn_silence 128ms and max_turn_silence 1280ms on the balanced preset, and three latency modes. Add Medical Mode for clinical vocabulary in the live stream at $0.60/hr combined. Set voice_focus to near-field for headset and laptop audio.

How does AssemblyAI handle patient data in telehealth recordings?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI, acting as a business associate under HIPAA, and maintains SOC 2 Type 2. PHI redaction is available across both audio and transcripts, built on entity detection covering 50+ types. Customer audio isn't used to train models or shared with third parties.

Does it handle visits where the patient switches languages?

The base model code-switches natively across 18 languages with no configuration — a speaker moving between languages mid-sentence stays intelligible. Medical Mode's clinical vocabulary covers English, Spanish, German and French, so a Spanish-English visit gets both, while other pairs get the code-switching without the Medical Mode entity boost.

How much does transcribing every telehealth visit cost?

Pre-recorded with Medical Mode is $0.36/hr, so about nine cents for a 15-minute visit. Live streaming with Medical Mode is $0.60/hr. Redaction adds $0.08/hr for transcript text and $0.05/hr for audio. See the pricing page for current rates.

How does AssemblyAI handle HIPAA and PHI?

AssemblyAI signs a Business Associate Addendum (BAA) for customers processing PHI and acts as a business associate under HIPAA. Alongside the BAA there's PHI redaction across audio and transcripts and SOC 2 Type 2. The BAA FAQ covers how to put one in place.