TL;DR
- OVHcloud's Olala-v1.0-mini-7A1B is a 6.8B-parameter mixture-of-experts model with about 1B active parameters per token, released under Apache 2.0 with base, SFT and post-trained checkpoints on Hugging Face.
- The launch coverage quotes "8 GB of GPU memory" while the BF16 file is 13.59 GB. The gap closes only if you use a quantized file: the Q5_K_M GGUF is 5.99 GB and the MLX build is 5.4 GB, per the model cards.
- Speed claims (160 tokens/s on a 20 GB workstation GPU, 50 tokens/s on a CPU) are vendor figures. A bandwidth roofline shows what they imply: roughly 320 GB/s of effective memory bandwidth at BF16 for the GPU number, or about 141 GB/s with the 5-bit file.
- The catch for self-hosters is software, not hardware: Ollama, LM Studio and upstream llama.cpp, vLLM and mlx-lm cannot load the architecture yet. You need the vendor's forks, and vLLM must run with
--mamba-ssm-cache-dtype float32. - All numbers below come from the model cards, the file metadata and the press coverage of the launch. I did not run the model.
What exactly was released?
OVHcloud presented Olala mini, its first large language model, on November 17, 2026, and the Hugging Face organisation OVHaiLLM already hosts the checkpoints. The family on the main card:
| Checkpoint | What it is |
|---|---|
| Olala-v1.0-mini-7A1B-base | Pre-trained base model, 12.8T tokens |
| Olala-v1.0-mini-7A1B-SFT | After supervised fine-tuning, before reinforcement learning |
| Olala-v1.0-mini-7A1B | Final post-trained model: reasoning, chat, tool use |
| Olala-v1.0-summarise | Specialised for document summarisation |
| Olala-v1.0-translate | Specialised for machine translation, including financial documents |
The final model's card lists these specifications:
| Property | Value |
|---|---|
| Parameters | 6.8B total, about 1B active per token |
| Layers | 36: 29 Mamba-3 and 7 attention layers (one in five) |
| Feed-forward | Mixture of experts after each layer: 256 routed experts, 6 active, plus 1 shared expert |
| Context | 32K tokens input plus output; a 128K version is announced as coming soon |
| Vocabulary | 151,936 tokens |
| Training data | 12.8T tokens of public datasets and the lab's own synthetic data |
| License | Apache 2.0 |
The weights file confirms the size. The safetensors metadata reports 6,756,461,568 BF16 parameters plus 16,662,576 F32 parameters, 6,773,124,144 in total. The card also says the model was trained on 1,024 NVIDIA GH200 GPUs on JUPITER and on H100 GPUs at Jean Zay, with MeluXina as the third site.
Why does a 7B model advertise an 8 GB minimum?
The press coverage repeated an announced minimum of a graphics card with at least 8 GB of memory. The arithmetic for BF16 does not support that: 6.77 billion parameters at 2 bytes each is about 13.5 GB before any activations or state. The BF16 GGUF on the hub is 13.59 GB, which matches.
The reconciliation is quantization. File sizes listed on the GGUF repository:
| File | Size (GB) | Effective bits per weight |
|---|---|---|
| bf16 | 13.59 | 16.05 |
| q8_0-safe2 | 6.88 | 8.13 |
| q6_k | 6.31 | 7.45 |
| q5_k_m | 5.99 | 7.08 |
I computed the last column as file size times 8 divided by the 6.77B parameter count. The 5-bit file averaging 7.08 bits per weight suggests some tensors are kept at higher precision. That is my inference from the numbers, not something the card states.
So the weights alone fit an 8 GB card with the Q5_K_M file, with roughly 2 GB left for the recurrent state, the small attention cache and the runtime. Whether 2 GB is enough at the full 32K window is not something the cards quantify, so I would test it before promising it to anyone. The model card's own guidance for the NVIDIA path is a 24 GB or larger GPU for vLLM, tested on L4 and L40S.
How much memory bandwidth do the speed claims imply?
Single-stream decoding is usually bound by how fast the active weights stream from memory. A rough ceiling is
def tokens_per_second_ceiling(bandwidth_gb_s, active_params_b, bytes_per_weight): bytes_per_token_gb = active_params_b * bytes_per_weight return bandwidth_gb_s / bytes_per_token_gb # BF16, about 1B active parameters: 2.0 GB read per token print(tokens_per_second_ceiling(320, 1.0, 2.0)) # 160.0 # Q5_K_M, scale active bytes by the file-size ratio (5.99 / 13.59 of BF16) print(tokens_per_second_ceiling(141, 1.0, 2.0 * 5.99 / 13.59)) # about 160 The model is a mixture of experts, so only 6 of 256 routed experts plus the shared expert are read per token, which is why "1B active" and not "6.8B" is the right number for this estimate. Real kernels also read attention and state tensors, and routing adds scattered reads, so treat this as a lower bound on the bandwidth a claim needs.
Two reported figures are worth putting through it. OVHcloud states 160 tokens/s on an RTX A4500, a 20 GB workstation card, and 50 tokens/s on a plain processor without a GPU. The second one is stated without naming the CPU.
For 160 tokens/s you need about 320 GB/s of effective bandwidth at BF16, about 163 GB/s at 8-bit and about 141 GB/s with the 5-bit file. If the claim was measured on a quantized build, the bandwidth need is modest for a workstation GPU, which makes the number plausible. If it was measured at BF16, it asks for most of a workstation card's memory system. The announcement does not say which build was used, and that is the one detail I would want before comparing it to any other model.
The CPU figure is more instructive. 50 tokens/s needs about 100 GB/s at BF16 or about 44 GB/s at 5-bit. A desktop CPU with dual-channel DDR5 is in the tens of GB/s, so the 50 tokens/s claim only works with a quantized file and a well-fed memory system. For contrast, the model card says the research code path in plain PyTorch gives about 6 tokens/s on a 64-core server CPU, because the Mamba-3 layers and experts run unoptimised without the custom kernels. The distance between 6 and 50 tokens/s is kernels, not silicon.
What does the hybrid layout buy at long context?
Only 7 of the 36 layers keep a KV cache. The other 29 are Mamba-3 layers that carry a fixed-size recurrent state. A conventional transformer of the same depth would grow its cache across all 36 layers, so the per-token cache growth here is about 7/36, or 19%, of that baseline at equal per-layer cache size. I am assuming equal per-layer cache size for the comparison. The model config does not map cleanly onto a standard KV-head count (num_key_value_heads is 0 in config.json), so I will not put a byte figure on it.
The practical consequence is that cache memory is a smaller part of your budget than weights for a 32K window. That flips the usual planning order for long-context serving: size the weights first, then check the state.
The state has its own pitfall. The vendor warns that vLLM's default keeps the Mamba recurrent state in bf16, and long generations then degrade into repetition loops. Their fix is a flag:
vllm serve OVHaiLLM/Olala-v1.0-mini-7A1B --trust-remote-code \ --mamba-ssm-cache-dtype float32 \ --reasoning-parser olala --tool-call-parser olala --enable-auto-tool-choice \ --max-model-len 32688 --port 8000 Keeping the state in float32 costs more memory per sequence than bf16 would, which matters when you batch many users. If you plan capacity for a shared endpoint, measure state memory per concurrent sequence on your own hardware.
Can I run it with my usual tools today?
Not with the stock ones. The cards say Ollama, LM Studio and the official llama.cpp, vLLM and mlx-lm cannot load the olala architecture yet. The supported routes are:
| You have | Route | Size |
|---|---|---|
| Apple Silicon, M2 or later, 16 GB | MLX build | 5.4 GB |
| M1 Mac, PC without a GPU, or one NVIDIA GPU | llama.cpp fork with GGUF | 6 to 14 GB |
| NVIDIA server GPU, 24 GB or more | vLLM fork or the prebuilt ovhaillm/olala-vllm image | BF16 |
| No hardware | OVHcloud AI Endpoints, an OpenAI-compatible API | hosted |
The GGUF card adds that the 6-bit file fits a 16 GB MacBook Air M5 and generates about 70 tokens/s, and that the 8-, 6- and 5-bit files score within noise of BF16 on code and math. Both are vendor statements. The first vLLM start compiles GPU kernels, which the card says takes about 8 minutes, and the image needs a recent driver (580 or later, CUDA 13.0).
A minimal hosted call through the Hugging Face router, from the model card:
import os from openai import OpenAI client = OpenAI(base_url="https://router.huggingface.co/v1", api_key=os.environ["HF_TOKEN"]) response = client.chat.completions.create( model="OVHaiLLM/Olala-v1.0-mini-7A1B:ovhcloud", messages=[{"role": "user", "content": "Explain how a heat pump works."}], temperature=0.8, top_p=0.9, ) print(response.choices[0].message.content) Reasoning effort has four levels, disabled, low, medium (the default) and high, which the card recommends for math at the cost of speed. The model writes in channels: reasoning in an analysis channel, the answer in final and tool calls in tools. Parse them with the vendor's olala parsers rather than your own regexes.
What is still unknown?
- Quality numbers in the launch coverage. The press report says the announcement itself gave no evaluation scores. The technical report PDF in the repository is the place to look, and I have not reproduced anything from it.
- Which build produced the speed claims. BF16 and 5-bit differ by a factor of more than two in bandwidth need, as the charts show.
- The 128K version. The card says it is coming soon, and the roadmap in the coverage names the end of 2026. Until it ships, 32K input plus output is the budget.
- Customer results. Qwant is reported to use the model for short summaries, with answers 20% faster and operating costs divided by three against an unnamed earlier model. Without the baseline, that is a direction, not a benchmark.
FAQ
Is Olala-v1.0-mini-7A1B open source?
The weights are released under Apache 2.0, and the repository also carries a public summary of training content. The training data is described as public datasets plus synthetic data.
How many parameters are active per token?
About 1B out of 6.8B, using 6 of 256 routed experts plus 1 shared expert per layer.
How much memory does it need?
The BF16 weights are 13.59 GB. Quantized files range from 5.4 GB (MLX) to 6.88 GB (Q8_0). Add runtime state on top, and use at least 24 GB for the vLLM path per the card.
What is the context length?
32K tokens counting input and output. A 128K version is announced as coming soon.
Why won't Ollama or LM Studio load it?
The architecture mixes Mamba-3 layers with attention and a 256-expert MoE, and upstream runtimes do not implement it yet. The vendor publishes forks for llama.cpp, vLLM and MLX support.
What is the one flag I must not forget on vLLM?
--mamba-ssm-cache-dtype float32. The default bf16 state can cause repetition loops on long generations.
Links
- Model card: https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B
- GGUF card: https://huggingface.co/OVHaiLLM/Olala-v1.0-mini-7A1B-GGUF
- Launch coverage: https://www.actuia.com/en/news/ovhcloud-unveils-olala-mini-its-first-llm-speeds-no-scores/

