How to convert video/speech to text?

Whisper AISpeech/Audio to Text

November 21, 2024

Read More
image

Convert Speech or Video to Text Using Google Colab and Whisper AI

Google Colab is a powerful, cloud-based platform that allows you to run Python code with minimal setup. Combined with OpenAI’s Whisper AI, it becomes a tool that can easily convert speech or video into text. This guide will walk you through setting up Google Colab, selecting the right runtime and hardware options, and using Whisper for transcribing audio or video.

What is Whisper AI?

Whisper AI is a state-of-the-art speech-to-text model by OpenAI that can transcribe speech from audio and video files into text. Whisper is highly accurate, even in noisy environments, and supports multiple languages. It works with various audio formats, including MP3, WAV, OGG, and MP4.

Step 1: Set Up Google Colab

1. Create a New Notebook

  1. Visit Google Colab and log in with your Google account.
  2. Click File > New Notebook to create a new notebook.

2. Select the Right Runtime

  1. Go to Runtime > Change Runtime Type in your notebook.
  2. Set the following: Runtime Type: Python 3 Hardware Accelerator: GPU
  3. Click Save.

Why Choose GPU?
GPU acceleration is ideal for Whisper, as it significantly speeds up the transcription process, especially for larger or more complex files. Using a GPU can save you a lot of time compared to using a CPU.

Step 2: Install Whisper and FFmpeg

Now, install Whisper and FFmpeg, which is required for audio and video file processing. Run the following commands in your notebook:

# Install Whisper and FFmpeg
!pip install git+https://github.com/openai/whisper.git
!apt-get install -y ffmpeg

This will install the Whisper model and FFmpeg for handling audio and video files.

Step 3: Upload Your Audio or Video File

1. Upload Files

  1. In Colab, click on the Files tab in the left sidebar.
  2. Click Upload and select your audio or video file (e.g., audio1.ogg or video1.mp4).
  3. The file will appear in the Files tab under /content/your_file_name.

2. (Optional) Extract Audio from Video

If you're uploading a video and need to extract the audio:

!ffmpeg -i /content/video1.mp4 -q:a 0 -map a /content/audio1.ogg

This will extract the audio into an audio1.ogg file.

3. (Optional) Manually uploading audio or video

from google.colab import files
uploaded = files.upload()

Step 4: Start Transcribing

With your file uploaded and ready, you can now transcribe it using Whisper. Run the following command:

!whisper "/content/audio1.ogg" --model medium.en
!whisper "audio2.ogg" --model large --language en
!whisper "audio2.ogg" --model large --language hi

Explanation of the Command:

  • !whisper: Executes the Whisper CLI tool.
  • "/content/audio1.ogg": Path to the audio file you want to transcribe.
  • --model medium.en: Specifies which Whisper model to use. The medium.en model is ideal for general transcription, offering a balance between speed and accuracy.

Whisper Models: When to Use Each

Whisper offers various models optimized for different trade-offs between speed and accuracy. The Tiny model is the fastest but has lower accuracy, making it ideal for quick transcriptions of small files or when working on low-resource systems. The Base model is also very fast and provides decent accuracy, which is well-suited for short and simple recordings or real-time transcription.

The Small model offers a good balance between speed and accuracy, making it a solid choice for medium-sized files or environments with moderate background noise. For longer or noisier audio files where higher accuracy is required, the Medium model is a better option, offering improved transcription quality. Lastly, the Large model, while the slowest, provides the highest accuracy and is perfect for complex files, multilingual transcription, or when the best possible accuracy is crucial.

Choosing the right model depends on the size and complexity of the audio you're transcribing and how critical accuracy is for your task. For quick and small files, Tiny or Base will suffice, but for longer, noisier files or when multilingual support is needed, Medium or Large will be the better options.

Recommendation:

  • For quick tasks or small files, use Tiny or Base.
  • For larger files or better accuracy, go for Medium or Large.

Choosing the Right Runtime and Hardware

In addition to selecting the Whisper model, you also need to choose the right runtime and hardware accelerator for your task. Google Colab offers different options for both, and selecting the right ones can dramatically improve performance and efficiency.

1. Runtime Type

Google Colab offers two primary runtime types:

  • Python 3 (Recommended): Python is the primary language supported by Whisper AI, making it the best choice for running the model. Most machine learning libraries and tools are built for Python, including Whisper.
  • R: While R is excellent for data analysis and statistics, it’s not suitable for running Whisper AI, as Whisper is built to work with Python. For AI and machine learning tasks, Python 3 is the only option.

Recommendation: Choose Python 3 for Whisper transcription.

2. Hardware Accelerators

Google Colab offers three hardware accelerators: CPU, GPU, and TPU. The right choice depends on the size of your file and the Whisper model you’re using.

CPU (Central Processing Unit)

  • Best For: Light tasks, small datasets, or when a GPU/TPU is unavailable.
  • When to Use: If you are working with small files or very basic transcriptions, CPU should suffice. It’s also a good option if you are just experimenting and don’t need speed.
  • Performance: Slowest option, but works for simple tasks.

GPU (Graphics Processing Unit)

  • Best For: Running machine learning models like Whisper for faster processing.
  • When to Use: Use GPU when working with large files, longer recordings, or when you need faster transcription. GPU accelerates the model’s inference time, making it ideal for Whisper.
  • Performance: T4 GPU is the most common in Colab and works well for Whisper, significantly speeding up the transcription process.

TPU (Tensor Processing Unit)

  • Best For: Large-scale machine learning models, especially in TensorFlow.
  • When to Use: TPU is highly optimized for deep learning tasks but can be tricky to configure for models like Whisper, which are based on PyTorch. While TPUs offer excellent performance for some types of models, Whisper generally works better on GPUs.
  • Performance: Fastest for TensorFlow-based tasks, but might not be optimal for Whisper.

Recommendation: Choose GPU for Whisper, as it provides the best performance for most transcription tasks without the complexity of TPUs.

Conclusion

Using Google Colab with Whisper AI provides a powerful, easy-to-use solution for converting speech and video into text. By selecting the right runtime and hardware accelerator, you can ensure that your transcription process is as fast and efficient as possible.

  1. Runtime: Choose Python 3 for Whisper.
  2. Hardware Accelerator: Use GPU for optimal performance when transcribing large or complex files.
  3. Whisper Models: Select the appropriate model based on your need for speed vs. accuracy.

With this setup, you can quickly transcribe your audio or video files directly in Google Colab, saving both time and effort.

Sidak Vats

© 2024 Sidak Vats