Testing the latest granite 🪨 speech 5 capabilities on Apple Silicon.

Introduction

As speech-to-text workflows transition toward more deeply integrated architectures, developers need models that don't just transcribe audio mechanically, but understand context, intent, and translation across multiple languages natively. Enter IBM Granite Speech 5.0-2B (ibm-granite/granite-speech-5.0-2b)—a high-performance, open-weights model leveraging a two-pass design (a CTC speech encoder coupled directly with a Granite-4.0-1B LLM backbone). Released under the permissive Apache 2.0 licence, it delivers state-of-the-art multilingual Automatic Speech Recognition (ASR) and Speech Translation (AST) across English, French, German, Spanish, Portuguese, Italian, and Japanese.

In this guide, we will walk through how to construct a complete, production-ready end-to-end demonstration application. We'll use Bob—our local developer workspace tool—to orchestrate the project structure, and connect Granite Speech's powerful transcription and translation features directly to downstream LLM backends (such as Ollama, llama.cpp, or OpenAI) via a unified Gradio web UI.

IBM Granite Speech 5

Excerpt from Hugging Face model card

Granite Speech 5.0 2B is a compact and efficient speech-language model, specifically designed for multilingual automatic speech recognition (ASR) and bidirectional automatic speech translation (AST) for English, French, German, Spanish, Portuguese and Japanese.

The model was trained on 390,000 hours of audio from public corpora for ASR and AST as well as synthetic datasets tailored to support Japanese ASR, keyword-biased ASR and speech translation. Granite Speech 5.0 2B was trained by modality aligning an intermediate checkpoint of granite-4.0-1b-base to speech on publicly available open source corpora containing audio inputs and text targets. Compared to its predecessor granite-speech-4.1-2b, this model has the same parameter count and provides additional capabilities and improvements:

Higher transcription accuracy for English short-form ASR and increased robustness to noise, reverberation and distant microphone speech
Supports input language id prompting for multilingual ASR
Improved long-form and conversational speech translation
Increased keyword list size (up to 2000) for multilingual keyword-biased ASR and AST

The mode card page proposes several sample test codes and implementations runnable as is.

Sample code using Transformers;

 # Usage with transformers  # pip install transformers torchaudio soundfile  import torch import torchaudio from huggingface_hub import hf_hub_download from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor device = "cuda" if torch.cuda.is_available() else "cpu" model_name = "ibm-granite/granite-speech-5.0-2b" processor = AutoProcessor.from_pretrained(model_name) tokenizer = processor.tokenizer model = AutoModelForSpeechSeq2Seq.from_pretrained( model_name, device_map=device, torch_dtype=torch.bfloat16 ) # Load audio  audio_path = hf_hub_download(repo_id=model_name, filename="multilingual_sample.wav") wav, sr = torchaudio.load(audio_path, normalize=True) assert wav.shape[0] == 1 and sr == 16000 # mono, 16kHz  # Create text prompt  user_prompt = "<|audio|>transcribe the speech with proper punctuation and capitalization." chat = [ {"role": "user", "content": user_prompt}, ] prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True) # Run the processor + model  model_inputs = processor(prompt, wav, device=device, return_tensors="pt").to(device) model_outputs = model.generate( **model_inputs, max_new_tokens=200, do_sample=False, num_beams=1 ) # Transformers includes the input IDs in the response  num_input_tokens = model_inputs["input_ids"].shape[-1] new_tokens = model_outputs[0, num_input_tokens:].unsqueeze(0) output_text = tokenizer.batch_decode( new_tokens, add_special_tokens=False, skip_special_tokens=True ) print(f"STT output = {output_text[0]}")

Sample code using vLLM;

# usage vllm # pip install vllm  from transformers import AutoTokenizer from vllm import LLM, SamplingParams from vllm.assets.audio import AudioAsset model_id = "ibm-granite/granite-speech-5.0-2b" tokenizer = AutoTokenizer.from_pretrained(model_id) def get_prompt(question: str, has_audio: bool): """Build the input prompt to send to vLLM.""" if has_audio: question = f"<|audio|>{question}" chat = [ { "role": "user", "content": question } ] return tokenizer.apply_chat_template(chat, tokenize=False) model = LLM( model=model_id, max_model_len=2048, # This may be needed for lower resource devices.  limit_mm_per_prompt={"audio": 1}, ) question = "can you transcribe the speech into a written format?" prompt_with_audio = get_prompt( question=question, has_audio=True, ) audio = AudioAsset("mary_had_lamb").audio_and_sample_rate inputs = { "prompt": prompt_with_audio, "multi_modal_data": { "audio": audio, } } outputs = model.generate( inputs, sampling_params=SamplingParams( temperature=0.0, max_tokens=64, ), ) print(f"Audio Example - Question: {question}") print(f"Generated text: {outputs[0].outputs[0].text}")

Sample Application Features and Functions

The application implements the following features;

Speech Transcription: upload a recorded audio file or speak directly into your microphone. The system converts speech to accurate written text using IBM's Granite Speech 5.0-2B model — trained specifically for high-quality transcription in real-world conditions.

Multilingual Translation: spoken audio in English, French, German, Spanish, Portuguese, or Japanese is automatically translated into any other supported language. Ideal for international teams, multilingual content production, and cross-border customer support.

The example above is a sample translation from English to French. Other languages are available too, such as German for another test;

 Der Bezirksarzt näherte sich mit der Gleichgültigkeit eines Mannes, der es gewohnt ist, die Hälfte seiner Zeit unter den Toten zu verbringen. Er hob dann das über das Gesicht gelegte Blatt und öffnete nur die Lippen.

AI-Assisted Analysis: once a transcript is produced, it is forwarded to an AI language model with your custom instruction — "summarise this meeting," "extract all action items," "identify the customer's main concern." The AI response appears alongside the transcript.

One-Click Full Pipeline: the Full Pipeline tab combines transcription and AI analysis into a single button click. Provide audio and an instruction; the system handles everything end-to-end and saves a structured result file automatically.

Live Configuration: switch AI backends, change models, or adjust processing settings from within the interface — no restart required. Configurations take effect immediately, enabling rapid A/B comparison of different AI backends.
Hardware Intelligence: the system automatically detects the capabilities of the host machine — GPU type, available memory, CPU cores — and self-optimises its processing parameters accordingly, ensuring the best performance on every device.

Architecture

The entire pipeline is structured to ensure modularity, separating hardware capability detection, audio processing, configuration management, and multi-backend LLM interfacing.

Implementation

Project Layout & Core Setup

Project's file and organization:

granite-speech5-test/ ├── app.py # Gradio demo application (entry point) ├── requirements.txt # Python dependencies ├── .env.example # Environment variable template ├── config/ │ └── config.yaml # Application configuration ├── src/ │ ├── speech_processor.py # Granite Speech 5.0-2B wrapper │ ├── hardware_detector.py# Hardware capability detection │ ├── llm_client.py # Multi-backend LLM abstraction │ └── config_manager.py # Configuration management ├── tests/ │ ├── test_speech_processor.py │ ├── test_hardware_detector.py │ ├── test_llm_client.py │ └── test_config_manager.py ├── scripts/ │ ├── setup.sh # One-time environment setup │ ├── start.sh # Launch app in background │ ├── stop.sh # Graceful shutdown │ └── cleanup.sh # Remove artefacts / venv ├── input/ # Put audio files here ├── output/ # JSON results written here └── Docs/ ├── Quickstart.md └── Architecture.md

Hardware Detection & Optimization Advisor

Before loading models, the application queries host resources to assign the optimal compute device (CUDA GPU, Apple Silicon MPS, or CPU) and precision dtype:

""" Hardware capability detection and optimisation advisor. Interrogates the host machine for CPU cores, RAM, GPU VRAM and Apple Silicon presence, then returns a structured HardwareProfile and suggested model-loading parameters. No external heavy dependencies – falls back gracefully when optional packages (torch, psutil) are unavailable so that the module can be imported during the install phase. """ from __future__ import annotations import logging import os import platform import subprocess import sys from dataclasses import dataclass, field from typing import Optional ... # Optional torch import at module level so tests can patch it cleanly. try: import torch # type: ignore except ImportError: torch = None # type: ignore  logger = logging.getLogger(__name__)

Core Speech Processing Wrapper

The core speech module (GraniteSpeechProcessor) handles raw audio files, normalizes inputs to 16 kHz mono waveforms, and executes inference using the proper prompt templates specified by the model card:

""" IBM Granite Speech 5.0-2B — Core speech processing module. Model reference: https://huggingface.co/ibm-granite/granite-speech-5.0-2b Architecture: two-pass design (CTC speech encoder → Granite-4.0-1B LLM backbone) Supported tasks: ASR (transcription) and AST (translation) in 6 languages. Usage constraints (Apache 2.0 licence, no additional restrictions): - Audio must be 16 kHz mono; longer recordings should be split into ≤2-minute segments. - Prompt tokens use the <|audio|> placeholder which the processor replaces with audio embeddings. Performance guide ----------------- On CPU (M-series or x86) a 2-second clip takes ~60-120 s with float32 because the Granite-4.0-1B LLM backbone must run greedy-decode token-by-token. To reach real-time speed, one of the following is needed: 1. CUDA GPU (any card ≥ 4 GB VRAM) → bfloat16 → 1-5× real-time 2. Apple Silicon with MPS → float16 → ~5× real-time 3. INT-8 dynamic quantisation → ~2× CPU speedup (built-in, no GPU) 4. Reduce max_new_tokens → linear speedup (safe for short clips) 5. ONNX runtime (WebGPU / CoreML) → best CPU/edge throughput All optimisation flags are controlled via config/config.yaml [speech.perf]. """ from __future__ import annotations import logging import time from contextlib import contextmanager from dataclasses import dataclass, field from pathlib import Path from typing import Optional import numpy as np import torch from src.speech_processor import GraniteSpeechProcessor, PerfConfig from src.hardware_detector import detect_hardware hw = detect_hardware() # Initialise processor with performance profile processor = GraniteSpeechProcessor( device=hw.recommended_device, perf=PerfConfig(int8_quantize=False, profile=True) ) ... # Transcribe audio file result = processor.transcribe("input/meeting.wav", language="en", verbatim=False) print("Transcript:", result.text) print(f"Real-Time Factor (RTFx): {result.realtime_factor:.1f}×") # Translate speech (e.g., French audio to English text) translation = processor.translate("input/french_speech.wav", source_language="fr", target_language="en") print("Translation:", translation.text)

Downstream Processing with Multi-Backend LLMs

Once a transcript is extracted, you can pipe it directly into local or cloud LLM backends using a uniform interface:

""" LLM backend abstraction layer. Provides a unified LLMClient interface over multiple local and cloud backends: Local ----- • Ollama – http://localhost:11434 (default) • LM Studio – http://localhost:1234 (OpenAI-compatible endpoint) • llama.cpp server – http://localhost:9931 (project rule: llama.cpp port) Cloud ----- • OpenAI – https://api.openai.com/v1 • Anthropic – Claude via Messages API • IBM watsonx.ai – https://us-south.ml.cloud.ibm.com (optional) All backends expose the same .chat(messages) → str interface so the application layer never needs to know which backend is active. Usage ----- >>> from src.llm_client import build_llm_client >>> llm = build_llm_client(backend="ollama", model="granite3.3:8b") >>> llm.chat([{"role": "user", "content": "Summarise this transcript: …"}]) """ from __future__ import annotations import json import logging import time import urllib.request # imported at module level for testability from abc import ABC, abstractmethod from dataclasses import dataclass, field from typing import Any, Optional logger = logging.getLogger(__name__) ... # Build a local Ollama client instance pointing to Granite or Llama models llm = build_llm_client(backend="ollama", model="granite4.2:latest") response = llm.chat([ { "role": "user", "content": f"Transcript:\n{result.text}\n\nInstruction: Summarise the key action items concisely." } ]) print("LLM Reply:", response.text) print(f"Latency: {response.latency_s:.2f}s | Tokens used: {response.input_tokens} → {response.output_tokens}")

End-to-End Application Process

The user interface combines transcription, translation, and LLM interaction tabs into a unified local control center (app.py).

""" IBM Granite Speech 5.0-2B — End-to-End Demo Application (Gradio UI). Tabs ──── 1. Transcribe – upload or record audio → Granite Speech → text output 2. Translate – speech translation between supported language pairs 3. LLM Interact – send a transcript to the configured LLM backend 4. Config – inspect and switch settings at runtime 5. Hardware – display host hardware profile Run ─── python app.py # or via scripts/start.sh """ from __future__ import annotations import json import logging import os import time import urllib.request import urllib.error from datetime import datetime from pathlib import Path from typing import Optional, Tuple import gradio as gr from src.config_manager import get_config, reload_config from src.hardware_detector import detect_hardware from src.llm_client import build_llm_client from src.speech_processor import ( GraniteSpeechProcessor, PerfConfig, SpeechResult, SUPPORTED_LANGUAGES, ) logger = logging.getLogger(__name__) ... # ── Entry point ───────────────────────────────────────────────────────────────  def main() -> None: cfg = get_config() # Ensure output directory exists  Path(cfg.app.output_dir).mkdir(parents=True, exist_ok=True) # Warm up hardware detection  logger.info("Detecting hardware …") hw = _get_hardware_profile() logger.info("\n%s", hw.summary_str()) demo = build_ui() # Build SSL kwargs — when cert/key are present the app serves HTTPS,  # which is required for the browser microphone API to work on localhost.  ssl_kwargs: dict = {} root = Path(__file__).parent certfile = cfg.app.ssl_certfile or str(root / "certs" / "cert.pem") keyfile = cfg.app.ssl_keyfile or str(root / "certs" / "key.pem") if Path(certfile).exists() and Path(keyfile).exists(): ssl_kwargs = { "ssl_certfile": certfile, "ssl_keyfile": keyfile, # ssl_verify=False tells Gradio's internal httpx health-check to  # skip cert validation for its own loopback call — required when  # using a self-signed certificate, otherwise Gradio crashes on  # startup with CERTIFICATE_VERIFY_FAILED before the browser even  # connects.  "ssl_verify": False, } scheme = "https" logger.info("TLS enabled — serving over HTTPS (microphone will work in browser)") else: scheme = "http" logger.warning( "No TLS cert found at %s — microphone requires HTTPS.\n" " Run: openssl req -x509 -newkey rsa:2048 -nodes " "-keyout certs/key.pem -out certs/cert.pem -days 3650 " '-subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"', certfile, ) logger.info("App URL: %s://localhost:%d", scheme, cfg.app.server_port) demo.launch( server_port=cfg.app.server_port, share=cfg.app.share, server_name="0.0.0.0", **ssl_kwargs, ) if __name__ == "__main__": main()