Speech UnderstandingLeMUR - LLMs for Speech

Ask questions about your audio data

In this guide, you’ll learn how to use LeMUR to ask questions and get answers about your audio data.

If you want a Quickstart, see Apply LLMs to audio files.

Before you start

To use LeMUR, you need an

AssemblyAI account

with a credit card set up.

Basic Q&A example

To ask a question about your audio data, define a prompt with your questions and call the LeMUR Task endpoint. The underlying transcript is used as additional context for the model.

1import assemblyai as aai
2
3aai.settings.api_key = "<YOUR_API_KEY>"
4
5# Step 1: Transcribe an audio file.
6
7# audio_file = "./local_file.mp3"
8
9audio_file = "https://assembly.ai/sports_injuries.mp3"
10
11transcriber = aai.Transcriber()
12transcript = transcriber.transcribe(audio_file)
13
14# Step 2: Define a prompt with your question(s).
15
16prompt = "What is a runner's knee?"
17
18# Step 3: Apply LeMUR.
19
20result = transcript.lemur.task(
21prompt, final_model=aai.LemurModel.claude3_5_sonnet
22)
23
24print(result.response)

Example output

1Based on the transcript, runner's knee is a condition characterized
2by pain behind or around the kneecap. It is caused by overuse,
3muscle imbalance and inadequate stretching. Symptoms include pain
4under or around the kneecap and pain when walking.

Q&A with specialized endpoint

The LeMUR Question & Answer function requires no prompt engineering and facilitates more deterministic and structured outputs. See the code examples below for more information on how to use this endpoint.

1import assemblyai as aai
2
3aai.settings.api_key = "<YOUR_API_KEY>"
4
5audio_url = "https://assembly.ai/meeting.mp4"
6transcript = aai.Transcriber().transcribe(audio_url)
7
8questions = [
9    aai.LemurQuestion(
10        question="What are the top level KPIs for engineering?",
11        context="KPI stands for key performance indicator",
12        answer_format="short sentence"),
13    aai.LemurQuestion(
14        question="How many days has it been since the data team has gotten updated metrics?",
15        answer_options=["1", "2", "3", "4", "5", "6", "7", "more than 7"]),
16]
17
18result = transcript.lemur.question(
19    final_model=aai.LemurModel.claude3_5_sonnet,
20    questions,
21    context="A GitLab meeting to discuss logistics"
22)
23
24for qa_response in result.response:
25    print(f"Question: {qa_response.question}")
26    print(f"Answer: {qa_response.answer}")

Custom Q&A example (Advanced)

This example shows how you can run a custom LeMUR task with an advanced prompt to create custom Q&A responses:

Cookbook: Custom Q&A with LeMUR Task

More Q&A prompt examples

Try any of these prompts to get started:

Use caseExample prompt
Question and answer”Identify any patterns or trends based on the transcript”
Closed-ended questions”Did the customer express a positive sentiment in the phone call?”
Sentiment analysis”What was the emotional sentiment of the phone call?”

For more use cases and prompt examples, see LeMUR examples.

API reference

Improve the results

To improve the results, see the following resources: