Ask questions about your audio data — AssemblyAI

In this guide, you’ll learn how to use LeMUR to ask questions and get answers about your audio data.

If you want a Quickstart, see Apply LLMs to audio files.

Before you start

To use LeMUR, you need an AssemblyAI account with a credit card set up.

Basic Q&A example

If you want to send a custom prompt to the LLM, you can use the LeMUR Task and apply the model to your transcribed audio files.

Python

TypeScript

Go

Java

C#

Ruby

To ask question about your audio data, define a prompt with your questions and call transcript.lemur.task(). The underlying transcript is automatically used as additional context for the model.

1 import assemblyai as aai
2 
3 aai.settings.api_key = "<YOUR_API_KEY>"
4 
5 # Step 1: Transcribe an audio file.
6 # audio_file = "./local_file.mp3"
7 audio_file = "https://assembly.ai/sports_injuries.mp3"
8 
9 transcriber = aai.Transcriber()
10 transcript = transcriber.transcribe(audio_file)
11 
12 # Step 2: Define a prompt with your question(s).
13 prompt = "What is a runner's knee?"
14 
15 # Step 3: Apply LeMUR.
16 result = transcript.lemur.task(
17     prompt, final_model=aai.LemurModel.claude3_5_sonnet
18 )
19 
20 print(result.response)

Example output

1 Based on the transcript, runner's knee is a condition characterized
2 by pain behind or around the kneecap. It is caused by overuse,
3 muscle imbalance and inadequate stretching. Symptoms include pain
4 under or around the kneecap and pain when walking.

Q&A with specialized endpoint

The LeMUR Question & Answer function requires no prompt engineering and facilitates more deterministic and structured outputs. See the code examples below for more information on how to use this endpoint.

Python

TypeScript

Java

C#

Ruby

To use it, define a list of aai.LemurQuestion objects. For each question, you can define additional context and specify either a answer_format or a list of answer_options. Additionally, you can define an overall context.

1 import assemblyai as aai
2 
3 aai.settings.api_key = "<YOUR_API_KEY>"
4 
5 audio_url = "https://assembly.ai/meeting.mp4"
6 transcript = aai.Transcriber().transcribe(audio_url)
7 
8 questions = [
9     aai.LemurQuestion(
10         question="What are the top level KPIs for engineering?",
11         context="KPI stands for key performance indicator",
12         answer_format="short sentence"),
13     aai.LemurQuestion(
14         question="How many days has it been since the data team has gotten updated metrics?",
15         answer_options=["1", "2", "3", "4", "5", "6", "7", "more than 7"]),
16 ]
17 
18 result = transcript.lemur.question(
19     final_model=aai.LemurModel.claude3_5_sonnet,
20     questions,
21     context="A GitLab meeting to discuss logistics"
22 )
23 
24 for qa_response in result.response:
25     print(f"Question: {qa_response.question}")
26     print(f"Answer: {qa_response.answer}")

Custom Q&A example (Advanced)

This example shows how you can run a custom LeMUR task with an advanced prompt to create custom Q&A responses:

Cookbook: Custom Q&A with LeMUR Task

More Q&A prompt examples

Try any of these prompts to get started:

Use case	Example prompt
Question and answer	”Identify any patterns or trends based on the transcript”
Closed-ended questions	”Did the customer express a positive sentiment in the phone call?”
Sentiment analysis	”What was the emotional sentiment of the phone call?”

For more use cases and prompt examples, see LeMUR examples.

API reference

Improve the results

To improve the results, see the following resources:

Optimize your prompt with the prompt engineering guide.
To alter the outcome, see Change model and parameters.

1	import assemblyai as aai
2
3	aai.settings.api_key = "<YOUR_API_KEY>"
4
5	# Step 1: Transcribe an audio file.
6	# audio_file = "./local_file.mp3"
7	audio_file = "https://assembly.ai/sports_injuries.mp3"
8
9	transcriber = aai.Transcriber()
10	transcript = transcriber.transcribe(audio_file)
11
12	# Step 2: Define a prompt with your question(s).
13	prompt = "What is a runner's knee?"
14
15	# Step 3: Apply LeMUR.
16	result = transcript.lemur.task(
17	prompt, final_model=aai.LemurModel.claude3_5_sonnet
18	)
19
20	print(result.response)

1	Based on the transcript, runner's knee is a condition characterized
2	by pain behind or around the kneecap. It is caused by overuse,
3	muscle imbalance and inadequate stretching. Symptoms include pain
4	under or around the kneecap and pain when walking.

1	import assemblyai as aai
2
3	aai.settings.api_key = "<YOUR_API_KEY>"
4
5	audio_url = "https://assembly.ai/meeting.mp4"
6	transcript = aai.Transcriber().transcribe(audio_url)
7
8	questions = [
9	aai.LemurQuestion(
10	question="What are the top level KPIs for engineering?",
11	context="KPI stands for key performance indicator",
12	answer_format="short sentence"),
13	aai.LemurQuestion(
14	question="How many days has it been since the data team has gotten updated metrics?",
15	answer_options=["1", "2", "3", "4", "5", "6", "7", "more than 7"]),
16	]
17
18	result = transcript.lemur.question(
19	final_model=aai.LemurModel.claude3_5_sonnet,
20	questions,
21	context="A GitLab meeting to discuss logistics"
22	)
23
24	for qa_response in result.response:
25	print(f"Question: {qa_response.question}")
26	print(f"Answer: {qa_response.answer}")