How to build an interactive lecture summarization app

The new academic year has just started, and some students are surely wondering how they can leverage recent advancements in AI to aid their studies. Many schools are now uploading recorded versions of lectures so that students can go back and review at their own pace. How might we use AI to analyze these lectures?

In this tutorial, we’ll learn how to build an app that automatically summarizes lecture recordings. The app will also allow you to ask questions about the course material to get specific answers tailored to the lecture content. You can check out a demo of the app here or the source code here.

Getting started

To follow along with this tutorial, you must have:

Python installed
pip installed
Git installed
An AssemblyAI account

First, clone the repo:

git clone https://github.com/AssemblyAI-Examples/lemur-lecture-summarizer.git
cd lemur-lecture-summarizer

Now, create a virtual environment for the project. On MacOS/Linux, run

python -m venv venv  # you may need to use `python3` instead
source ./venv/bin/activate

Alternatively, on Windows, run

python -m venv venv  # you may need to use `python3` instead
.\venv\Scripts\activate.bat

Next, install the required dependencies:

pip install -r requirements.txt

And finally, you can optionally set your AssemblyAI API key as an environment variable. You can do this by renaming the file .env.example to .env and then, in the file, replacing paste-your-key-here with your API key.

If you don’t do this, you will need to supply the API key in the application GUI.

Building the application

The application uses streamlit and the AssemblyAI Python SDK to build an application around LeMUR. A great feature of streamlit is that variables that exist in isolation on their own lines are automatically displayed in the GUI, so it is easy to add text simply by writing a string.

For example, the below lines are in the source code for this application:

"# Lecture Summarizer"
"Use this application to **automatically summarize a virtual lecture** and **ask questions** about the lesson material"

These lines are automatically rendered as markdown in the GUI:

Given this feature, for the rest of this tutorial, we will omit such lines of text in the code in order to focus on the functional components of the application. Let’s get started.

Setting the AssemblyAI API Key

At the top of app.py, we attempt to set the AssemblyAI API key after our imports.

import os

import streamlit as st
import assemblyai as aai
from dotenv import load_dotenv
load_dotenv()

from utils import get_transcript, ask_question, return_ytdlp_fname

environ_key = os.environ.get("ASSEMBLYAI_API_KEY")
if environ_key is None:
	pass
elif environ_key == "paste-your-key-here":
	environ_key = None
else:
	aai.settings.api_key = environ_key

In particular, we try to fetch the ASSEMBLYAI_API_KEY environment variable. This variable can either exist on the system as a normal environment variable, or in a .env file which is loaded to the environment in the script using load_dotenv() of the python-dotenv package.

If the key exists, we set it as the API key for the AssemblyAI Python SDK through aai.settings.api_key = environ_key as seen here. Otherwise, we set it to None.

If no API key is supplied, we render an input text box so that the user can supply it manually.

def set_aai_key():
    """ Callback to change set AAI API key when the key is input in the text area """
    aai.settings.api_key = st.session_state.input_aai_key
    
if not environ_key:
    input_key = st.text_input(
        "API Key",
        placeholder="Enter your AssemblyAI API key here",
        type="password",
        on_change=set_aai_key,
        key='input_aai_key'
    )

The textbox allows a user to enter text, and then calls the set_aai_key callback function in order to set the entered key as the key stored for the AssemblyAI Python SDK.

Selecting a lecture

After an API key is supplied, we render a radio selector that lets the user choose from one of the allowed file types:

if input_key or environ_key:
	ftype = st.radio("File type", ('Local file', 'Remote file', 'YouTube link'))

If a local file is uploaded to the app, we write the contents to a temporary file f so that it can be transcribed by AssemblyAI. Otherwise, we set the filename f to be the URL of the file or YouTube video.

    if ftype == 'Local file':
        # Store the uploaded file in a temporary file
        f = st.file_uploader("File")
        if f:
            uploaded_ftype = f.name.split('.')[-1]
            temp_fname = f"tmp.{uploaded_ftype}"
            with open(temp_fname, 'wb') as fl:
                fl.write(f.read())
            f = temp_fname
    elif ftype == 'Remote file':
        f = st.text_input("Link", 
                          value="https://storage.googleapis.com/aai-web-samples/cs50p-unit-tests.mp3",
                          placeholder="Public link to the file"
                          )
    elif ftype == 'YouTube link':
        f = st.text_input("Link",
                          value="https://www.youtube.com/watch?v=tIrcxwLqzjQ",
                          placeholder="YouTube link"
                          )

Finally, we add a textbox for a user to supply context about the file. This is additional information that helps LeMUR understand the file by providing contextualizing information about it.

    value = "" if ftype == "Local file" else "A lesson from Harvard's CS50P course. The lesson is about Unit Testing in Python."
    placeholder = "Contextualizing information about the file (optional)"
    context = st.text_input("Context", value=value, placeholder=placeholder)

Transcribing the lecture

After a file has been selected and submitted, a transcript of the file is generated using the get_transcript function. This function simply calls the Transcriber.transcribe method in the AssemblyAI Python SDK, which transcribes either a local file or publicly available remote file. If the supplied file is a YouTube video, get_transcript first downloads the video to a temporary local file.

def get_transcript(f, ftype):
    transcriber = aai.Transcriber()

    if ftype == 'YouTube link':
        with st.spinner('Downloading video...'):
            ydl_opts = {'outtmpl': YTDLP_FNAME}
            print("downloading")
            with YoutubeDL(ydl_opts) as ydl:
                ydl.download([f])
                f = YTDLP_FNAME
    
    with st.spinner('Transcribing file...'):
        transcript = transcriber.transcribe(f)
    if transcript.error:
        raise TranscriptionException(transcript.error)
    return transcript

Once the file is transcribed, it is saved to the app’s session state so that the value persists between re-renders, and any temporary local files are removed.

if f:
    entered = st.button("Submit")
    if entered:
        st.session_state['entered'] = True
        
        transcript = get_transcript(f, ftype)
        if ftype == "Local file":
            os.remove(f)
        elif ftype == "YouTube link":
            os.remove(YTDLP_FNAME)

        st.session_state['transcript'] = transcript

Summarizing the lecture

Once the file has been transcribed, the AssemblyAI Python SDK makes it easy to summarize using the lemur.summarize method of the transcript. We simply specify an answer format as markdown and then generate the summary. After the summary has been generated, it is saved to the session state.

        params = {
            'answer_format': "**<part of the lesson>**\n<list of important points in that part>",
            'max_output_size': 4000
        }
        if context: params['context'] = context
        
        with st.spinner("Generating summary..."):
            summary = transcript.lemur.summarize(**params)
        
        st.session_state['summary'] = summary.response.strip().split('\n')
        print('session summary: ', st.session_state['summary'])

The results are then displayed as markdown:

if st.session_state['entered']:
    "## Results"
    
    if st.session_state['summary']:
        for i in st.session_state['summary']:
            st.markdown(i)

Asking questions about the lecture

Finally, the AssemblyAI Python SDK once again makes it easy to ask questions about the lecture through the lemur.question method. We define an ask_question function which uses this method to take in a question and return the answer:

def ask_question(transcript, question):
    questions = [
        aai.LemurQuestion(question=question,)
    ]

    result = transcript.lemur.question(questions)

    if transcript.error:
        raise QuestionException(result.error)

    return result.response[0].answer

We then provide space for the user to enter and submit a question, after which we call the ask_question function in order to get a response. Once a response has been generated, it is displayed in the application GUI:

if st.session_state['summary']:
    "# Questions"
    "Ask a question about the lesson below:"
    
    question = st.text_input("Question",
                              placeholder="What is the point of using Pytest?",
                              )
    
    question_asked = st.button("Submit", key='question_asked')
    if question_asked:
        with st.spinner('Asking question...'):
            answer = ask_question(st.session_state['transcript'], question)
    answer

Running the application

To run the application, execute streamlit run app.py in a terminal from the project directory. The command will output a local URL from which you can access and use the application.