How to Run OpenAI’s Whisper Speech Recognition Model

Yesterday, OpenAI released its Whisper speech recognition model. Whisper joins other open-source speech-to-text models available today - like Kaldi, Vosk, wav2vec 2.0, and others - and matches state-of-the-art results for speech recognition.

In this article, we’ll learn how to install and run Whisper, and we’ll also perform a deep-dive analysis into Whisper's accuracy, inference time, and cost-to-run.

How to Run OpenAI's Whisper

In this section, we'll learn how to install and use Whisper. You can jump down to the Whisper analysis or to a more complicated Whisper advanced usage if you're already up-and-running with Whisper.

Step 1: Install Dependencies

Whisper requires Python3.7+ and a recent version of PyTorch (we used PyTorch 1.12.1 without issue). Install Python and PyTorch now if you don't have them already.

Whisper also requires FFmpeg, an audio-processing library. If FFmpeg is not already installed on your machine, use one of the below commands to install it.

# Linux
sudo apt update && sudo apt install ffmpeg

# MacOS
brew install ffmpeg

# Windows
chco install ffmpeg

Finally, if using Windows, ensure that Developer Mode is enabled. In your system settings, navigate to Privacy & security > For developers and turn the top toggle switch on to turn Developer Mode on if it is not already.

Step 2: Install Whisper

Now we are ready to install Whisper. Open up a command line and execute the below command to install Whisper:

pip install git+https://github.com/openai/whisper.git 

Step 3: Run Whisper

Command Line

First, we'll use Whisper from the command line. Simply open up a terminal and navigate into the directory in which your audio file lies. We will be using a file called audio.wav, which is the first line of the Gettysburg Address. To transcribe this file, we simply run the following command in the terminal:

whisper audio.wav

The output will be displayed in the terminal:

(venv) C:\Users> whisper audio.wav
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: english
[00:00.000 --> 00:10.000]  Four score and seven years ago, our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal.

The transcription is also saved to audio.wav.txt, along with a file audio.wav.vtt to be used for closed captioning.

Python

Using Whisper for transcription in Python is very easy. Simply import whisper, specify a model, and transcribe the audio.

import whisper

model = whisper.load_model("base")
result = model.transcribe("audio.wav")

The transcription text can be access with result["text"]. The result object itself contains other useful information:

{
  "text": " Four score and seven years ago, our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal.",
  "segments": [
    {
      "id": 0,
      "seek": 0,
      "start": 0.0,
      "end": 10.0,
      "text": " Four score and seven years ago, our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal.",
      "tokens": [
        50364,
        7451,
        6175,
        ...,
        2681,
        13,
        50864
      ],
      "temperature": 0.0,
      "avg_logprob": -0.1833780391796215,
      "compression_ratio": 1.3858267716535433,
      "no_caption_prob": 0.05988641083240509
    }
  ],
  "language": "en"
}

OpenAI Whisper Analysis

The below figure from the Whisper paper compares Whisper's accuracy, using Word-Error-Rate (WER), to current state-of-the-art speech recognition models. As you can see, Whisper reports achieving state-of-the-art results, which is an exciting development for the field of speech recognition, especially given that Whisper is an open-source model.

While these results are exciting, speech recognition remains an open problem - especially for non-English languages. The below figure reports Whisper's Word-Error-Rates for each supported language. While Whisper achieves state-of-the-art results on several Romance languages, German, Japanese, and more, performance is comparatively lacking for other languages.

Below we see the distribution of languages as a function of word error rate. Of the 82 languages in the plot above, 50 of them have Word-Error-Rates greater than 20%,

Anecdotal Comparisons

At Assembly, our API is powered by a state-of-the-art Conformer-CTC model trained on ~100,000 hours of labeled data. To explore Whisper's accuracy, we decided to compare Whisper with our model in a few side-by-side comparisons.

First, we show a comparison of the Micro Machines example from the Whisper announcement post:

Micro Machines

0:00

/29.88839

1×

The second example is a clip from a podcast

Podcast

0:00

/90.510431

1×

The final audio file is a board of directors meeting

Board of Directors Meeting

0:00

/159.52975

1×

As we can see, Whisper performs very well and is a fantastic addition to the available state-of-the-art options for speech recognition today.

Whisper Inference Time

Whisper is available in five sizes - tiny, base (default), small, medium, and large - getting more accurate with size. The large model, therefore, has the best accuracy and is the model used in benchmarks reported in the paper and in the graphs above. Whisper can be used on both CPU and GPU; however, inference time is prohibitively slow on CPU when using the larger models, so it is advisable to run them only on GPU.

The Micro Machines example was transcribed with Whisper on both CPU and GPU at each model size, and the inference times are reported below. First, we see the results for CPU (i5-11300H)

Next, we have the results on GPU (high RAM GPU Colab environment)

Here are the same results side-by-side

Cost to Run Whisper

We provide the cost to transcribe 1,000 hours of audio using Whisper in GCP (1x A100 40 GB) for each model size using different batch sizes, the values of which can be found in the legend.

The costs in this graph depict the raw compute cost to run Whisper after it has been deployed. The costs do not include headcount for building out relevant infrastructure, fixing model bugs, or improving and updating the model. To continuously keep performance competitive, a dedicated research or research engineering team would be required.

Final Words

Our above analysis showed that Whisper achieves state-of-the-art results for speech recognition in several languages. Whisper will serve as a valuable tool to researchers and hackers alike, both for its accuracy and ease-of-use compared to other open-source options. Whisper's performance stems in part from its compute intensity, so applications requiring the larger, more powerful versions of Whisper should make sure to run Whisper on GPU, whether locally or in the cloud.

Whisper Advanced Usage

We got acquainted with Whisper in the How to Run OpenAI's Whisper section above. For a more complicated example, we'll review a modified version of the multilingual ASR notebook. Execute the following commands to download the example code and install the necessary requirements:

git clone https://github.com/AssemblyAI-Examples/whisper-multilingual.git
cd whisper-multilingual
pip install -r requirements.txt

Next, simply run python main.py to transcribe and translate several Korean audio files into English. Each datum will take about 3 minutes to process on CPU. We use a total of 10 data points, so let the process run in the background while we examine the main.py code.

First, we perform all necessary imports, and then define a class that will be used to download and store the audio data. The details of this class are not relevant, so they have been omitted for the sake of brevity.

import io
import os

import torch
import pandas as pd
import urllib
import tarfile
import whisper

from scipy.io import wavfile
from tqdm import tqdm

class Fleurs(torch.utils.data.Dataset):
	pass

Next, we set some parameters for displaying the result with pandas, set the device to use for inference, and then set the variables which specify the language of the audio. The first the Korean language code used to download the data, and the latter is the Korean language code used with the Whisper model.

# Display options for pandas dataset
pd.options.display.max_rows = 100
pd.options.display.max_colwidth = 1000

# Set inference device
device = "cuda" if torch.cuda.is_available() else "cpu"

# Set language (korean)
language_google = "ko_kr"
language_whisper = "korean"

Now we create the dataset using the class we defined above, selecting a subsample of 10 audio files to make the processing quicker.

# Create dataset object, selecting only 10 examples for brevity
dataset = Fleurs(language_google, subsample_rate=1, device=device)
dataset = torch.utils.data.random_split(dataset, [10, len(dataset)-10])[0]

Next, we load the Whisper model that we will be using, opting for the "tiny" model version to make inference quicker. We then set transcription and translation options.

# Load tiny Whisper model
model = whisper.load_model("tiny")

# Set options
options = dict(language=language_whisper, beam_size=5, best_of=5)
transcribe_options = dict(task="transcribe", **options)
translate_options = dict(task="translate", **options)

Finally, we iterate through the dataset, transcribing each audio file to Korean, and translating each audio file to English. Note that the translation happens directly on the audio data and does not translate the generated transcription to English. We save the transcriptions and translations to lists, in addition to the ground truth reference for comparison.

# Run inference
references = []
transcriptions = []
translations = []

for audio, text in tqdm(dataset):
    transcription = model.transcribe(audio, **transcribe_options)["text"]
    translation = model.transcribe(audio, **translate_options)["text"]

    transcriptions.append(transcription)
    translations.append(translation)
    references.append(text)

Finally, we create the pandas DataFrame which stores the results, and then print the results and save them to CSV.

# Create dataframe from results and save the data
data = pd.DataFrame(dict(reference=references, transcription=transcriptions, translation=translations))
print(data)
data.to_csv("results.csv")

The results can be seen below