Inserting at the cursor

When the user dictates into a field that already has text, the words have to fit where the cursor is. Dictating “tomorrow works” into I think| we should go should give I think tomorrow works we should go. By default, the rewrite returns a complete sentence, with a capital letter at the start and a full stop at the end. Insert that at the cursor as it is, and you get I thinkTomorrow works. we should go. To get it right, split the job in two:

The rewrite fixes the words. Tell it what comes before the cursor, and it writes the dictation as a continuation: lowercase for ordinary words, capitals kept for names. Only a model can tell those apart.
Your code fixes the joins. Spacing and end punctuation follow fixed rules from the text on either side of the cursor, so code gets them right every time. The rewrite output is trimmed, so it can’t return a leading space anyway.

The four steps below explain each part. For working code, go to the complete example.

1. Read the text around the cursor

Take the text before the cursor and the text after it, each up to a few hundred characters. How you read them depends on the platform:

Platform	Text before and after the cursor
Web	`value.slice(0, selectionStart)` and `value.slice(selectionEnd)` on the `<textarea>` or `<input>`
iOS keyboard extension	`textDocumentProxy.documentContextBeforeInput` and `documentContextAfterInput`
macOS	The focused element’s `kAXValueAttribute` and `kAXSelectedTextRangeAttribute`, through the Accessibility API

Some fields don’t expose their text, such as password fields and some canvas-based editors. Treat those as empty, and add no spacing.

2. Set `rewrite_mode` to `smart`

Send every insertion with "rewrite_mode": "smart". The instruction in the next step only works on smart. On the default fast mode, the rewrite usually ignores it and returns the whole sentence, so the text before the cursor appears twice: I think I think tomorrow works. smart adds some latency to the rewrite. See Choosing a rewrite mode for the trade-off.

3. Tell the rewrite where the words land

Decide in code whether the cursor sits mid-sentence:

At the start of a sentence, when the field is empty or the text before the cursor ends with ., !, ? or a line break: send no instruction. The default cleanup writes a normal sentence.
Mid-sentence: send an llm_instruction that says the dictated words continue the text before them.

In both cases, send the text before the cursor as stt_prompt too. It’s context for the transcription itself, so spellings, names and casing carry over from what the user already wrote. Keep it as plain text, with no instructions in it. See Prompting and keyterms. For I think| we should go, the config is:

{
  "rewrite_mode": "smart",
  "stt_prompt": "I think",
  "llm_instruction": "Remove filler words and false starts from the dictated words, then write them as a mid-sentence continuation of this text, which comes before them: \"I think\". Return only the dictated words, with no terminal punctuation, since the sentence carries on after them."
}

The instruction describes the shape you want, a mid-sentence continuation, instead of listing rules such as “lowercase unless it’s a name” or “no final punctuation”. The model leans toward complete sentences, and a rule against that has to fight it on every request, so it’s followed less reliably.

4. Join the result in code

When the response comes back:

Spacing. Add a space before the words when the text before the cursor doesn’t end in whitespace, and after them when the text after the cursor doesn’t start with whitespace or punctuation.
End punctuation. Remove a trailing ., ! or ? when the text after the cursor carries on the sentence, such as a lowercase word or a comma.
A failed rewrite. If the rewrite fails, llm_response is null. Apply the same joins to text, the verbatim transcript. In the SDKs, final_text already falls back to it.

Complete example

Each tab is a complete program. Add your API key, save a recording as clip.wav, and run it. The four helpers are the same in every tab, so you can copy them into your app as they are:

Python	JavaScript	What it does
`continues_sentence`	`continuesSentence`	Decides whether the cursor sits mid-sentence
`carries_on`	`carriesOn`	Decides whether the text after the cursor continues the sentence
`rewrite_instruction`	`rewriteInstruction`	Builds the `llm_instruction` mid-sentence, and returns nothing at the start of one
`fit_at_cursor`	`fitAtCursor`	Joins the result onto the text around the cursor

Python SDK
JavaScript SDK
Python
JavaScript

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

SENTENCE_ENDS = (".", "!", "?")


def continues_sentence(before):
    """True when the cursor sits mid-sentence, False at the start of one."""
    if not before or before.endswith("\n"):
        return False
    tail = before.rstrip()
    return bool(tail) and not tail.endswith(SENTENCE_ENDS)


def carries_on(after):
    """True when the text after the cursor continues the same sentence."""
    next_char = after.lstrip()[:1]
    return next_char.islower() or next_char in (",", ";", ":")


def rewrite_instruction(before, after):
    """The llm_instruction for this insertion, or None for the default cleanup."""
    if not continues_sentence(before):
        return None
    ending = (
        "with no terminal punctuation, since the sentence carries on after them"
        if carries_on(after)
        else "ending the sentence"
    )
    return (
        "Remove filler words and false starts from the dictated words, then write "
        "them as a mid-sentence continuation of this text, which comes before them: "
        f'"{before.rstrip()[-300:]}". Return only the dictated words, {ending}.'
    )


def fit_at_cursor(dictated, before, after):
    """Join the dictated text onto the text around the cursor."""
    text = dictated.strip()
    if not text:
        return text
    # The text after the cursor carries on the sentence: no full stop.
    if carries_on(after):
        text = text.rstrip(".!?")
    # One space between this and the text before the cursor.
    if before and not before[-1].isspace() and text[0] not in ".,;:!?":
        text = " " + text
    # One space between this and the text after the cursor.
    if after and not after[0].isspace() and after[0] not in ".,;:!?":
        text = text + " "
    return text


def dictate_at_cursor(audio, before, after):
    """Transcribe a dictation and return the text to insert at the cursor."""
    config = aai.DictationConfig(
        rewrite_mode="smart",  # required: see step 2
        stt_prompt=before[-1000:] or None,
        llm_instruction=rewrite_instruction(before, after),
    )
    result = aai.DictationTranscriber().transcribe_live(audio, config)
    # final_text is the rewrite, or the transcript if the rewrite failed.
    return fit_at_cursor(result.final_text, before, after)


before, after = "I think", " we should go"  # the text around the cursor
insert = dictate_at_cursor("clip.wav", before, after)
print(before + insert + after)  # I think tomorrow works we should go

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: "<YOUR_API_KEY>" });

const SENTENCE_END = /[.!?]$/;

// True when the cursor sits mid-sentence, false at the start of one.
function continuesSentence(before) {
  if (!before || before.endsWith("\n")) return false;
  const tail = before.trimEnd();
  return tail.length > 0 && !SENTENCE_END.test(tail);
}

// True when the text after the cursor continues the same sentence.
function carriesOn(after) {
  return /^[\p{Ll},;:]/u.test(after.trimStart());
}

// The llm_instruction for this insertion, or null for the default cleanup.
function rewriteInstruction(before, after) {
  if (!continuesSentence(before)) return null;
  const ending = carriesOn(after)
    ? "with no terminal punctuation, since the sentence carries on after them"
    : "ending the sentence";
  return (
    "Remove filler words and false starts from the dictated words, then write " +
    "them as a mid-sentence continuation of this text, which comes before them: " +
    `"${before.trimEnd().slice(-300)}". Return only the dictated words, ${ending}.`
  );
}

// Join the dictated text onto the text around the cursor.
function fitAtCursor(dictated, before, after) {
  let text = dictated.trim();
  if (!text) return text;
  // The text after the cursor carries on the sentence: no full stop.
  if (carriesOn(after)) text = text.replace(/[.!?]+$/, "");
  // One space between this and the text before the cursor.
  if (before && !/\s$/.test(before) && !/^[.,;:!?]/.test(text)) text = " " + text;
  // One space between this and the text after the cursor.
  if (after && !/^[\s.,;:!?]/.test(after)) text = text + " ";
  return text;
}

// Transcribe a dictation and return the text to insert at the cursor.
async function dictateAtCursor(audio, before, after) {
  const config = {
    rewrite_mode: "smart", // required: see step 2
    stt_prompt: before.slice(-1000) || undefined,
    llm_instruction: rewriteInstruction(before, after) ?? undefined,
  };
  const result = await client.dictation.transcribeLive(audio, config);
  // final_text is the rewrite, or the transcript if the rewrite failed.
  return fitAtCursor(result.final_text, before, after);
}

const before = "I think"; // the text around the cursor
const after = " we should go";
const insert = await dictateAtCursor("clip.wav", before, after);
console.log(before + insert + after); // I think tomorrow works we should go

import json

import requests

SENTENCE_ENDS = (".", "!", "?")


def continues_sentence(before):
    """True when the cursor sits mid-sentence, False at the start of one."""
    if not before or before.endswith("\n"):
        return False
    tail = before.rstrip()
    return bool(tail) and not tail.endswith(SENTENCE_ENDS)


def carries_on(after):
    """True when the text after the cursor continues the same sentence."""
    next_char = after.lstrip()[:1]
    return next_char.islower() or next_char in (",", ";", ":")


def rewrite_instruction(before, after):
    """The llm_instruction for this insertion, or None for the default cleanup."""
    if not continues_sentence(before):
        return None
    ending = (
        "with no terminal punctuation, since the sentence carries on after them"
        if carries_on(after)
        else "ending the sentence"
    )
    return (
        "Remove filler words and false starts from the dictated words, then write "
        "them as a mid-sentence continuation of this text, which comes before them: "
        f'"{before.rstrip()[-300:]}". Return only the dictated words, {ending}.'
    )


def fit_at_cursor(dictated, before, after):
    """Join the dictated text onto the text around the cursor."""
    text = dictated.strip()
    if not text:
        return text
    # The text after the cursor carries on the sentence: no full stop.
    if carries_on(after):
        text = text.rstrip(".!?")
    # One space between this and the text before the cursor.
    if before and not before[-1].isspace() and text[0] not in ".,;:!?":
        text = " " + text
    # One space between this and the text after the cursor.
    if after and not after[0].isspace() and after[0] not in ".,;:!?":
        text = text + " "
    return text


def dictate_at_cursor(path, before, after):
    """Transcribe a dictation and return the text to insert at the cursor."""
    config = {"rewrite_mode": "smart"}  # required: see step 2
    if before:
        config["stt_prompt"] = before[-1000:]
    instruction = rewrite_instruction(before, after)
    if instruction:
        config["llm_instruction"] = instruction

    with open(path, "rb") as f:
        response = requests.post(
            "https://dictation.assemblyai.com/v1/transcribe/live",
            headers={"Authorization": "<YOUR_API_KEY>"},
            files={
                "config": (None, json.dumps(config), "application/json"),
                "audio": ("clip.wav", f.read(), "audio/wav"),
            },
            timeout=90,
        )
    response.raise_for_status()
    result = response.json()
    # llm_response is null if the rewrite failed: fall back to the transcript.
    return fit_at_cursor(result["llm_response"] or result["text"], before, after)


before, after = "I think", " we should go"  # the text around the cursor
insert = dictate_at_cursor("clip.wav", before, after)
print(before + insert + after)  # I think tomorrow works we should go

import { readFileSync } from "fs";

const SENTENCE_END = /[.!?]$/;

// True when the cursor sits mid-sentence, false at the start of one.
function continuesSentence(before) {
  if (!before || before.endsWith("\n")) return false;
  const tail = before.trimEnd();
  return tail.length > 0 && !SENTENCE_END.test(tail);
}

// True when the text after the cursor continues the same sentence.
function carriesOn(after) {
  return /^[\p{Ll},;:]/u.test(after.trimStart());
}

// The llm_instruction for this insertion, or null for the default cleanup.
function rewriteInstruction(before, after) {
  if (!continuesSentence(before)) return null;
  const ending = carriesOn(after)
    ? "with no terminal punctuation, since the sentence carries on after them"
    : "ending the sentence";
  return (
    "Remove filler words and false starts from the dictated words, then write " +
    "them as a mid-sentence continuation of this text, which comes before them: " +
    `"${before.trimEnd().slice(-300)}". Return only the dictated words, ${ending}.`
  );
}

// Join the dictated text onto the text around the cursor.
function fitAtCursor(dictated, before, after) {
  let text = dictated.trim();
  if (!text) return text;
  // The text after the cursor carries on the sentence: no full stop.
  if (carriesOn(after)) text = text.replace(/[.!?]+$/, "");
  // One space between this and the text before the cursor.
  if (before && !/\s$/.test(before) && !/^[.,;:!?]/.test(text)) text = " " + text;
  // One space between this and the text after the cursor.
  if (after && !/^[\s.,;:!?]/.test(after)) text = text + " ";
  return text;
}

// Transcribe a dictation and return the text to insert at the cursor.
async function dictateAtCursor(path, before, after) {
  const config = { rewrite_mode: "smart" }; // required: see step 2
  if (before) config.stt_prompt = before.slice(-1000);
  const instruction = rewriteInstruction(before, after);
  if (instruction) config.llm_instruction = instruction;

  const form = new FormData();
  form.append("config", new Blob([JSON.stringify(config)], { type: "application/json" }));
  form.append("audio", new Blob([readFileSync(path)], { type: "audio/wav" }), "clip.wav");

  const response = await fetch("https://dictation.assemblyai.com/v1/transcribe/live", {
    method: "POST",
    headers: { Authorization: "<YOUR_API_KEY>" },
    body: form,
  });
  if (!response.ok) throw new Error(`Dictation failed: ${response.status} ${await response.text()}`);
  const result = await response.json();
  // llm_response is null if the rewrite failed: fall back to the transcript.
  return fitAtCursor(result.llm_response ?? result.text, before, after);
}

const before = "I think"; // the text around the cursor
const after = " we should go";
const insert = await dictateAtCursor("clip.wav", before, after);
console.log(before + insert + after); // I think tomorrow works we should go

What to expect

These are results from the complete example, run against Dictation:

Text before the cursor	The user says	Text after the cursor	Result
`I think`	”tomorrow works”	`we should go`	I think tomorrow works we should go
`I'll send it to`	”the team in London”	`, then we can review.`	I’ll send it to the team in London, then we can review.
`We need to`	”uh, finish the report and, and send it out before Friday”	None	We need to finish the report and send it out before Friday.
`The meeting is on`	”um, Thursday at three”	None	The meeting is on Thursday at 3.
`That works for me.`	”let’s meet at noon”	None	That works for me. Let’s meet at noon.
None	”let’s meet at noon”	None	Let’s meet at noon.

The rewrite lowercases ordinary words mid-sentence but keeps names such as “London” and “Friday” capitalized, drops filler words and repeats, and ends with a full stop only when the sentence ends there.

Transcript rewriting: llm_instruction, rewrite_mode and writing a good instruction
Prompting and keyterms: stt_prompt and keyterms_prompt
Error handling: what to do when llm_response is null

Getting started

Features

API reference

Advanced

Guides

Inserting at the cursor

1. Read the text around the cursor

2. Set `rewrite_mode` to `smart`

3. Tell the rewrite where the words land

4. Join the result in code

Complete example

What to expect

​1. Read the text around the cursor

​2. Set rewrite_mode to smart

​3. Tell the rewrite where the words land

​4. Join the result in code

​Complete example

​What to expect

​Related pages

1. Read the text around the cursor

2. Set `rewrite_mode` to `smart`

3. Tell the rewrite where the words land

4. Join the result in code

Complete example

What to expect

Related pages