Prompt A Structured Q&A Response Using LeMUR

This Colab will demonstrate how to use AssemblyAI’s LeMUR (Leveraging Large Language Models to Understand Recognized Speech) framework to prompt a structured Question and Answer response using the Task Endpoint.

Quickstart

1import assemblyai as aai
2import xml.etree.ElementTree as ET
3
4aai.settings.api_key = "YOUR_API_KEY"
5audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"
6
7transcript = aai.Transcriber().transcribe(audio_url)
8
9def construct_question(question):
10 question_str = f"Question: {question.question}"
11
12 if question.context:
13 question_str += f"\nContext: {question.context}"
14
15 # Set default answer_format to "short sentence" if not provided
16 if not question.answer_format:
17 question.answer_format = "short sentence"
18
19 question_str += f"\nAnswer Format: {question.answer_format}"
20
21 if question.answer_options:
22 options_str = ", ".join(question.answer_options)
23 question_str += f"\nOptions: {options_str}"
24
25 return question_str + "\n"
26
27def escape_xml_characters(xml_string):
28 return xml_string.replace('&', '&')
29
30questions = [
31 aai.LemurQuestion(
32 question="What are the top level KPIs for engineering?",
33 context="KPI stands for key performance indicator",
34 answer_format="short sentence"),
35 aai.LemurQuestion(
36 question="How many days has it been since the data team has gotten updated metrics?",
37 answer_options=["1", "2", "3", "4", "5", "6", "7", "more than 7"]),
38 aai.LemurQuestion(
39 question="What are the future plans for the project?")
40]
41
42question_str = '\n'.join(construct_question(q) for q in questions)
43
44prompt = f"""You are an expert at giving accurate answers to questions about texts.
45 No preamble.
46 Given the series of questions, answer the questions.
47 Each question may follow up with answer format, answer options, and context for each question.
48 It is critical that you follow the answer format and answer options for each question.
49 When context is provided with a question, refer to it when answering the question.
50 You are useful, true and concise, and write in perfect English.
51 Only the question is allowed between the <question> tag. Do not include the answer format, options, or question context in your response.
52 Only text is allowed between the <question> and <answer> tags.
53 XML tags are not allowed between the <question> and <answer> tags.
54 End your response with a closing </response> tag.
55 For each question-answer pair, format your response according to the template provided below:
56
57Template for response:
58<responses>
59 <response>
60 <question>The question</question>
61 <answer>Your answer</answer>
62 </response>
63 <response>
64 ...
65 </response>
66 ...
67</responses>
68
69These are the questions:
70{question_str}
71"""
72
73result = transcript.lemur.task(
74 prompt,
75 final_model=aai.LemurModel.claude3_5_sonnet
76)
77
78response = result.response
79
80# Escape special XML characters and strip any leading/trailing whitespace
81clean_response = escape_xml_characters(response).strip()
82root = ET.fromstring(clean_response)
83for response in root.findall('response'):
84 question = response.find('question').text
85 answer = response.find('answer').text
86 print(f"Question: {question}")
87 print(f"Answer: {answer}")

Getting Started

Before we begin, make sure you have an AssemblyAI account and an API key. You can sign up for an AssemblyAI account and get your API key from your dashboard.

Find more details on the current LeMUR pricing in the AssemblyAI pricing page.

Step-by-Step Instructions

In this guide, we will prompt LeMUR with a structured Q&A format and generate an XML response.

First, let’s install the AssemblyAI SDK.

$pip install -U assemblyai

Then we’ll import the SDK and set our AssemblyAI API key.

1import assemblyai as aai
2
3aai.settings.api_key = "API_KEY_HERE"

Next, we’ll use AssemblyAI to transcribe a file and save our transcript.

1audio_url = "https://storage.googleapis.com/aai-web-samples/meeting.mp4"
2
3transcript = aai.Transcriber().transcribe(audio_url)

Construct a formatted string to structure the questions from the LemurQuestion object. This includes the question text, optional context, an answer format (defaulting to “short sentence” if not provided), and any answer options, then returns the formatted string.

1def construct_question(question):
2 question_str = f"Question: {question.question}"
3
4 if question.context:
5 question_str += f"\nContext: {question.context}"
6
7 # Set default answer_format to "short sentence" if not provided
8 if not question.answer_format:
9 question.answer_format = "short sentence"
10
11 question_str += f"\nAnswer Format: {question.answer_format}"
12
13 if question.answer_options:
14 options_str = ", ".join(question.answer_options)
15 question_str += f"\nOptions: {options_str}"
16
17 return question_str + "\n"

Define a list of aai.LemurQuestion objects. For each question, you can define additional context and specify either a answer_format or a list of answer_options.

1questions = [
2 aai.LemurQuestion(
3 question="What are the top level KPIs for engineering?",
4 context="KPI stands for key performance indicator",
5 answer_format="short sentence"),
6 aai.LemurQuestion(
7 question="How many days has it been since the data team has gotten updated metrics?",
8 answer_options=["1", "2", "3", "4", "5", "6", "7", "more than 7"]),
9 aai.LemurQuestion(
10 question="What are the future plans for the project?")
11]

Construct the formatted question string for all the Questions within the list of aai.LemurQuestion objects.

1question_str = '\n'.join(construct_question(q) for q in questions)

Provide detailed instructions to prompt LeMUR to answer a series of questions. This also defines a structured XML template for the responses.

1prompt = f"""You are an expert at giving accurate answers to questions about texts.
2 No preamble.
3 Given the series of questions, answer the questions.
4 Each question may follow up with answer format, answer options, and context for each question.
5 It is critical that you follow the answer format and answer options for each question.
6 When context is provided with a question, refer to it when answering the question.
7 You are useful, true and concise, and write in perfect English.
8 Only the question is allowed between the <question> tag. Do not include the answer format, options, or question context in your response.
9 Only text is allowed between the <question> and <answer> tags.
10 XML tags are not allowed between the <question> and <answer> tags.
11 End your response with a closing </response> tag.
12 For each question-answer pair, format your response according to the template provided below:
13
14Template for response:
15<responses>
16 <response>
17 <question>The question</question>
18 <answer>Your answer</answer>
19 </response>
20 <response>
21 ...
22 </response>
23 ...
24</responses>
25
26These are the questions:
27{question_str}
28"""

Prompt the LeMUR model using the Task Endpoint and return the response.

1result = transcript.lemur.task(
2 prompt,
3 final_model=aai.LemurModel.claude3_5_sonnet
4)
5
6response = result.response
7print(response)

Clean the XML output and print the question and answer pairs.

1import xml.etree.ElementTree as ET
2
3def escape_xml_characters(xml_string):
4 return xml_string.replace('&', '&amp;')
5
6# Escape special XML characters and strip any leading/trailing whitespace
7clean_response = escape_xml_characters(response).strip()
8
9root = ET.fromstring(clean_response)
10
11for response in root.findall('response'):
12 question = response.find('question').text
13 answer = response.find('answer').text
14 print(f"Question: {question}")
15 print(f"Answer: {answer}")