中文

TheWhisper

TheWhisper is a fine-tuned Whisper model that can process audio chunks up to 30 seconds.

Verified

Core facts

Read by a script from the official API, not rewritten by a model.

License
MIT
Languages
Python
Maintainer
TheStageAI
Verified

What it is

TheWhisper is a fine-tuned Whisper model that can process audio chunks up to 30 seconds. Unlike the original Whisper models, TheWhisper does not require padding audio with silence to reach 30 seconds. For the first release, this repository provides open-source transcription models with streaming inference support.

Main features

  1. TheWhisper provides open weights fine-tuned versions of Whisper models.

  2. TheWhisper provides a streaming implementation for NVIDIA and macOS.

  3. The project reports that its TheStage AI inference engines (NVIDIA GPU) reach 220 tok/s on L40s for the whisper-large-v3 model.

  4. TheWhisper provides benchmarks for latency, memory, power, and ASR accuracy (OpenASR).

Getting started

  1. You can build a macOS desktop app with real-time transcription.

  2. TheWhisper provides a local RestAPI with frontend examples using JS and Electron.

  3. TheStage AI built an Electron demo app that is certified by Apple.

From the README · bash
git clone https://github.com/TheStageAI/TheWhisper.git
cd TheWhisper
From the README · bash
pip install .[apple]
From the README · bash
pip install .[nvidia]
From the README · bash
pip install 'thestage-elastic-models[nvidia]==0.2.4.post0' --index-url https://thestage.jfrog.io/artifactory/api/pypi/pypi-thestage-ai-production/simple --extra-index-url https://pypi.nvidia.com --extra-index-url https://pypi.org/simple
pip install .[nvidia]
pip install thestage
From the README · python
import torch
from thestage_speechkit.apple import ASRPipeline

model = ASRPipeline(
    model='TheStageAI/thewhisper-large-v3-turbo',
    # optimized model with ANNA
    model_size='S',
    chunk_length_s=10
)

# inference
result = model(
    "path_to_your_audio.wav", 
    return_timestamps="word"
)

print(result["text"])
From the README · python
import torch
from thestage_speechkit.nvidia import ASRPipeline

model = ASRPipeline(
    model='TheStageAI/thewhisper-large-v3-turbo',
    # allowed: 10s, 15s, 20s, 30s
    chunk_length_s=10,
    # optimized TheStage AI engines
    batch_size=32,
    device='cuda'
)

# inference
result = model(
    "path_to_your_audio.wav", 
    chunk_length_s=10,
    generate_kwargs={'do_sample': False, 'use_cache': True}
)

print(result["text"])

Who it is for

The project describes TheWhisper as ideal for real-time captioning, live meetings, voice interfaces, and edge deployments. TheWhisper lists support for L40S, H100, RTX 4090, and RTX 5090. TheWhisper lists Nvidia Jetson support.

TheWhisper lists time-stamp support on Nvidia. The Nvidia GPU requirements list these supported GPUs: RTX 4090, RTX 5090, L40s, H100, A100, and Jetson-Thor. The Nvidia GPU requirements give the operating system as Ubuntu 20.04+.

The Nvidia GPU requirements give a minimum RAM of 2.5 GB, with 5 GB recommended for the large-v3 model. The Nvidia GPU requirements give the CUDA version as 11.8 or higher. The Nvidia GPU requirements give the driver version as 520.0 or higher.

The Apple Silicon requirements list these supported chipsets: M1, M1 Pro, M1 Max, M1 Ultra, M2, M2 Pro, M2 Max, M2 Ultra, M3, M3 Pro, M3 Max, M4, M4 Pro, and M4 Max. The Apple Silicon requirements give the operating system as macOS 15.0 (Ventura) or later, and iOS 18.0 or later. The Apple Silicon requirements give a minimum RAM of 2 GB, with 4 GB recommended for the large-v3 model.

For quality benchmarks, TheWhisper used multilingual benchmarks. The acknowledgements list OpenAI Whisper for the original Whisper model and pretrained checkpoints. The acknowledgements list Hugging Face Transformers for model, tokenizer, and inference utilities.

The acknowledgements list the MLX community for the MLX Whisper implementation for Apple Silicon.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.