ASR: Automatic Speech Recognition

Definition

Automatic speech recognition (ASR) converts a speech signal to text, mapping a sequence of audio inputs to text outputs. Virtual assistants like Siri and Alexa use ASR models to help users every day, and live captioning and note-taking during meetings are other user-facing applications. The Transformers automatic speech recognition guide shows how to fine-tune Wav2Vec2 on the MInDS-14 dataset to transcribe audio to text.

The Transformers automatic speech recognition guide also shows how to use the fine-tuned model for inference. In the guide's fine-tuning example, the word error rate (WER) metric is loaded.

At a glance

Wav2Vec2 is loaded with AutoModelForCTC. The reduction to apply is specified with the ctc_loss_reduction parameter.

Training hyperparameters are defined in TrainingArguments.
The Transformers ASR guide says that, in TrainingArguments, the only required parameter is output_dir, which specifies where to save the model.
Setting push_to_hub to True pushes the model to the Hub, and being signed in to Hugging Face is needed to upload it.
The training arguments are passed to Trainer along with the model, dataset, tokenizer, data collator, and compute_metrics function.
Calling train() fine-tunes the model.
A function passes the predictions and labels to compute to calculate the WER.

The source does not state the order of these points, so they are listed without arrows.

How it works

A Wav2Vec2 processor is loaded to process the audio signal. The MInDS-14 dataset has a sampling rate of 8000Hz, so it must be resampled to 16000Hz to use the pretrained Wav2Vec2 model. The preprocessing function calls the audio column to load and resample the audio file.

The preprocessing function also extracts the input_values from the audio file and tokenizes the transcription column with the processor. Transformers does not have a data collator for ASR, so DataCollatorWithPadding must be adapted to create a batch of examples. The data collator dynamically pads text and labels to the length of the longest element in its batch instead of the entire dataset, so they are a uniform length.

Unlike other data collators, this data collator applies a different padding method to input_values and labels. The sampling rate of the audio file is resampled to match the sampling rate of the model if needed. The guide notes that padding text in the tokenizer function with padding=True is possible, but dynamic padding in the data collator is more efficient.

Example

The Transformers automatic speech recognition guide loads a smaller subset of the MInDS-14 dataset from the Datasets library. The dataset's train split is split into a train and test set with the train_test_split method. The audio field is a 1-dimensional array of the speech signal that must be called to load and resample the audio file.

The transcription field is the target text. A pipeline for automatic speech recognition is instantiated with the model, and the audio file is passed to it. The results of the pipeline can also be manually replicated.

A processor preprocesses the audio file and transcription, and the processor returns the input as PyTorch tensors. The inputs are passed to the model to return the logits, and the processor decodes the predicted input_ids with the highest probability back into text. The MInDS-14 dataset contains fields such as lang_id and english_transcription, but the Transformers ASR guide focuses on the audio and transcription fields.

Related tools listed on this site.