LeMUR

Summarize your audio data

1require 'assemblyai'
2
3client = AssemblyAI::Client.new(api_key: '<YOUR_API_KEY>')
4
5# You can upload and transcribe a local file:
6# uploaded_file = client.files.upload(file: '/path/to/your/file')
7# transcript = client.transcripts.transcribe(audio_url: uploaded_file.upload_url, speaker_labels: true)
8
9# Or use a publicly-accessible URL:
10audio_url = 'https://assembly.ai/sports_injuries.mp3'
11
12transcript = client.transcripts.transcribe(audio_url: audio_url)
13
14result = client.lemur.task(
15 transcript_ids: [transcript.id],
16 prompt: 'Provide a brief summary of the transcript.',
17 final_model: AssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_5_SONNET
18)
19
20puts result.response

If you run the code above, you’ll see the following output:

1The transcript describes several common sports injuries - runner's knee,
2sprained ankle, meniscus tear, rotator cuff tear, and ACL tear. It provides
3definitions, causes, and symptoms for each injury. The transcript seems to be
4narrating sports footage and describing injuries as they occur to the athletes.
5Overall, it provides an overview of these common sports injuries that can result
6from overuse or sudden trauma during athletic activities

Ask questions about your audio data

Q&A with the task endpoint

To ask question about your audio data, define a prompt with your questions and call client.lemur.task(). Use the transcript_ids parameter to send one or more transcripts as additional context for the model.

1require 'assemblyai'
2
3client = AssemblyAI::Client.new(api_key: '<YOUR_API_KEY>')
4
5# Step 1: Transcribe an audio file. For local files see our Getting Started guides.
6audio_url = 'https://assembly.ai/sports_injuries.mp3'
7transcript = client.transcripts.transcribe(audio_url: audio_url)
8
9# Step 2: Define a prompt with your question(s).
10prompt = "What is a runner's knee?"
11
12# Step 3: Apply LeMUR.
13response = client.lemur.task(
14 prompt: prompt,
15 transcript_ids: [transcript.id],
16 final_model: AssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_5_SONNET
17)
18
19puts response.response

#Example output

1Based on the transcript, runner's knee is a condition characterized
2by pain behind or around the kneecap. It is caused by overuse,
3muscle imbalance and inadequate stretching. Symptoms include pain
4under or around the kneecap and pain when walking.

Q&A with the question-answer endpoint

The LeMUR Question & Answer function requires no prompt engineering and facilitates more deterministic and structured outputs. See the code examples below for more information on how to use this endpoint.

To use it, define a list of questions. For each question, you can define additional context and specify either a answer_format or a list of answer_options. Additionally, you can define an overall context.

1require 'assemblyai'
2
3client = AssemblyAI::Client.new(api_key: '<YOUR_API_KEY>')
4
5audio_url = 'https://assembly.ai/meeting.mp4'
6
7transcript = client.transcripts.transcribe(audio_url: audio_url)
8
9response = client.lemur.question_answer(
10 transcript_ids: [transcript.id],
11 final_model: AssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_5_SONNET,
12 context: 'A GitLab meeting to discuss logistic',
13 questions: [
14 {
15 question: 'What are the top level KPIs for engineering?',
16 context: 'KPI stands for key performance indicator',
17 answer_format: 'short sentence'
18 },
19 {
20 question: 'How many days has it been since the data team has gotten updated metrics?',
21 context: 'KPI stands for key performance indicator',
22 answer_options: ['1', '2', '3', '4', '5', '6', '7', 'more than 7']
23 }
24 ]
25)
26
27response.response.each do |qa|
28 printf("Question: %<question>s\n", question: qa.question)
29 printf("Answer: %<answer>s\n", answer: qa.answer)
30end

For the full API reference, as well as the supported models and FAQs, refer to the full LeMUR Q&A guide.

Change the model type

LeMUR features the following LLMs:

  • Claude 3.5 Sonnet
  • Claude 3 Opus
  • Claude 3 Haiku
  • Claude 3 Sonnet

You can switch the model by specifying the final_model parameter.

1response = client.lemur.task(
2 prompt: prompt,
3 transcript_ids: [transcript_id],
4 final_model: AssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_5_SONNET
5)
ModelSDK ParameterDescription
Claude 3.5 SonnetAssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_5_SONNETClaude 3.5 Sonnet is the most intelligent model to date, outperforming Claude 3 Opus on a wide range of evaluations, with the speed and cost of Claude 3 Sonnet. This uses Anthropic’s Claude 3.5 Sonnet model version claude-3-5-sonnet-20240620.
Claude 3.0 OpusAssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_OPUSClaude 3 Opus is good at handling complex analysis, longer tasks with many steps, and higher-order math and coding tasks.
Claude 3.0 HaikuAssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_HAIKUClaude 3 Haiku is the fastest model that can execute lightweight actions.
Claude 3.0 SonnetAssemblyAI::Lemur::LemurModel::ANTHROPIC_CLAUDE3_SONNETClaude 3 Sonnet is a legacy model with a balanced combination of performance and speed for efficient, high-throughput tasks.

You can find more information on pricing for each model here.

Change the maximum output size

You can change the maximum output size in tokens by specifying the max_output_size parameter. Up to 4000 tokens are allowed.

1response = client.lemur.task(
2 prompt: prompt,
3 transcript_ids: [transcript_id],
4 max_output_size: 1000
5)

Change the temperature

You can change the temperature by specifying the temperature parameter, ranging from 0.0 to 1.0.

Higher values result in answers that are more creative, lower values are more conservative.

1response = client.lemur.task(
2 prompt: prompt,
3 transcript_ids: [transcript_id],
4 temperature: 0.7
5)

Send customized input

You can submit custom text inputs to LeMUR without transcript IDs. This allows you to customize the input, for example, you could include the speaker labels for the LLM.

To submit custom text input, use the input_text parameter instead of transcript_ids.

1transcript = client.transcripts.transcribe(
2 audio_url: audio_url,
3 speaker_labels: true
4)
5
6text_with_speaker_labels = (transcript.utterances.map do |utterance|
7 sprintf(
8 "Speaker %<speaker>s:\n%<text>s\n",
9 speaker: utterance.speaker,
10 text: utterance.text
11 )
12end).join("\n")
13
14response = client.lemur.task(
15 prompt: prompt,
16 input_text: text_with_speaker_labels
17)
18
19puts response.response

Submit multiple transcripts

LeMUR can easily ingest multiple transcripts in a single API call.

You can feed in up to a maximum of 100 files or 100 hours, whichever is lower.

1response = client.lemur.task(
2 prompt: prompt,
3 transcript_ids: [id1, id2, id3]
4)

Delete LeMUR request data

You can delete the data for a previously submitted LeMUR request.

Response data from the LLM, as well as any context provided in the original request will be removed.

1response = client.lemur.task(
2 prompt: prompt,
3 transcript_ids: [transcript_id],
4)
5
6deletion_response = client.lemur.purge_request_data(request_id: response.request_id)