Vambo MORENA 1.5B Leads Its 26-Model African Language Benchmark

MORENA 1.5B Tops Vambo's 26-Model African Bits-Per-Byte Table​

The headline number is a bits-per-byte (bpb) score. Bpb measures how efficiently a model predicts raw text, and lower is better. Vambo compares tokenizers of different sizes by normalising per byte of text rather than per token. MORENA 1.5B base is lowest of all 26 at 1.408; the 8B African specialist Lugha-Llama-8B is second at 1.423, ahead of the other four MORENA sizes on the twelve-language mean. The general-purpose models fall far behind: every general-purpose model measured, from 1B to 12B parameters, trails well behind: the best of them, gemma-3-12b-it, reaches only 2.159.

TechRadar's report adds detail on how close the top of the table is. It describes Lugha-Llama-8B as an Africa-adapted Llama 3.1 variant that loses to MORENA in eight of the twelve languages. It also says Google's 12B Gemma model uses roughly 11 times MORENA's compute and still scores worse on the same text. So the gap to the specialist model is small, 1.408 against 1.423. The gap to general-purpose models from the big labs is large.

The languages covered are Shona, Swahili, Hausa, Yoruba, Igbo, isiZulu, isiXhosa, Kinyarwanda, Setswana, Afrikaans, isiNdebele and Nigerian Pidgin, plus English, French and code. Coverage in the evaluation is uneven. isiNdebele (ISO code nbl) has no FLORES+ or MasakhaNEWS coverage and is evaluated on NCHLT transcripts only. All twelve are written in Latin script.

These are Vambo's own measurements, published on its Hugging Face model cards. TechRadar's coverage, which it credits to Glitch Front, repeats those figures rather than reproducing them, and no independent re-run of the 26-model comparison has been published.

Why Vambo Trained MORENA From Scratch Instead of Adapting Llama​

The main design choice is the one in the name of the technical report, MORENA: An African Foundation Model. Most projects serving these languages keep training an existing Llama checkpoint. As TechRadar notes, that means they inherit a vocabulary built for English and programming text. Vambo instead fixed the tokenizer, the data mixture and the language list before training started.

The tokenizer is where that choice pays off first. A tokenizer splits text into the sub-word units a model processes. If it was trained mostly on English, words in Yoruba or isiZulu get broken into many small pieces, so the same sentence costs more tokens. MORENA uses a 65,536-entry byte-fallback BPE trained on the target mix. According to the model card, African text costs 0.249 tokens per byte against 0.234 for English, about 6% more per byte than English in MORENA's vocabulary, but that same African text needs 1.39x fewer tokens than under Gemma 3's vocabulary and 1.53x fewer than under Llama 3.2's. TechRadar reports that the team can't fully explain the remaining 6% gap.

Fewer tokens for the same passage means more text fits in the 4,096-token context window, and less work per sentence for any system that charges or schedules by token. That is an inference from the tokenizer numbers. The ratios alone don't prove an end-to-end speed or cost advantage of any particular size.

Training ran in two stages. The base was pretrained on 251.7 billion tokens over 60,000 steps, then "mid-trained" (annealed) on 63 billion tokens of higher-quality and task data, for 315 billion tokens in total. The architecture is a 28-layer decoder with a 2,048 hidden size, grouped-query attention and tied embeddings. Author Isheanesu Misi trained it on CINECA's Leonardo supercomputer with support from the AI Hub for Sustainable Development. TechRadar also names UNDP as a backer.

Machine-translated data makes up a large share of MORENA's training mix​

One detail matters to anyone judging output quality: much of the African-language data was synthetic. The model card says nine languages were machine-translated from English documents. That material accounts for 24% of pretraining tokens and 28% once mid-training is included, or 57.1 billion tokens on disk. By mid-training, African text made up 50.2% of tokens seen, and 41.6% of all tokens were machine-translated.

Vambo addresses the obvious objection directly. It says MORENA's advantage over Gemma 3 1B is not a translationese artefact, meaning a model that only learned the patterns of machine translation. The advantage is about the same on native news text (1.56x), speech transcripts (1.55x) and the FLORES benchmark (1.61x). The training corpus is not released with the weights, so outsiders can't audit that mix themselves.

The Compute Bill Is Smaller Than the Headline Suggests​

TechRadar describes MORENA as built on more than 22,000 A100 GPU-hours worth about $40,000. The model cards break that figure down, and the breakdown changes the story:

The 22,000-plus figure covers the whole research programme, including experiments. The released instruct model's lineage used about 12,834 hours. Vambo estimates that at $25,000 to $40,000, assuming $2 to $3 per A100-hour. The programme used about 22% of a 100,000 GPU-hour allocation. The base card is also frank about efficiency: about 20% useful utilisation across the whole run, and 28% to 33% during steady-state pretraining.

So the $40,000 is the top of Vambo's estimate for the release lineage, not the full cost of the programme. Either way, it's a small budget for a from-scratch foundation model that tops a 26-model table in its target languages.

The "Beats Google, Meta and Alibaba" Claim Depends on the Benchmark​

The comparison with the big labs holds on the metrics MORENA was built for. On others it gets weaker or reverses.

The "8x smaller" framing fits only one comparison. MORENA's 1.5B parameters are one-eighth of Gemma 3 12B. Against Lugha-Llama-8B, the closest competitor, the ratio is about 5.4x, which TechRadar itself describes as "over five times as many parameters".

Translation depends on direction. On FLORES+ with three example translations in the prompt (three-shot), results for five African languages are:

chrF++ is a character-level overlap score used to grade machine translation. MORENA 1.5B instruct's English-into-African chrF++ of 45.8 is eight points above MADLAD-400-3B (37.8). Translating into English, both MORENA checkpoints trail both reference models by four to six points. TechRadar adds that the instruct model is statistically level with one dedicated translation system and trails a larger one by about 1.4 points on the English-to-African task. It also says general models of similar size typically score between 9 and 14.

General knowledge and reasoning is where Alibaba's model comes out ahead. On English ARC-easy and ARC-challenge, MORENA base scores 0.676 and 0.412, against Qwen2.5-1.5B's 0.668 and 0.440. On MMLU, MORENA scores 0.284 and Qwen2.5-1.5B scores 0.324. The card notes that MORENA is close to the 0.25 floor you'd get by guessing, and says the factual knowledge was traded away on purpose. MORENA is a language specialist and wasn't designed as a general assistant.

The instruct model also gives up some language-modelling score after chat tuning. It scores 1.441 bpb, behind Lugha-Llama-8B's 1.423 overall. TechRadar reports it still leads that model in five shared languages with about 20% of the parameters.

MORENA 1.5B Instruct Documents Its Own Failure Modes​

The instruct model card doesn't just list wins. Its "What it is not good at" section should be read before anyone deploys the model:

• Retrieval-augmented question answering in African languages scores 0.325, against a chance level of 0.25, even though the model reaches a 0.73 grounding score, which shows it is reading the supplied passage.
• Grounded generation stays fully faithful to the given facts in 23% of attempts (22 of 96), down from 31% in the previous version.
• The model fails to answer 41% of ordinary benign requests well, mostly because it refuses. An earlier version refused to recommend a dry cleaner. Vambo has fixed that case, but the broader over-refusal problem remains.
• Multiple-choice reading comprehension (Belebele) averages 0.309 across ten African languages, near the 0.25 chance level. The card says every model under 12B that it measured is at chance on this test.
• Tool calling reaches 98.1% correct tool selection and 100% valid JSON, but only when the tool marker is prefilled. Left to decide for itself, the model almost never calls a tool.

A follow-up from the developer tested tool calling against a real API. In a GitHub repository on fine-tuning the 208M and 503M models to call real APIs from Pidgin, Yoruba, Igbo and Hausa, Misi reports that measured against a menu built from the real Paystack OpenAPI spec, the released models picked the correct tool 9% of the time, and 0% of their calls stayed inside the arguments the schema declares. He attributes the failure to training data rather than the architecture. He says the cause was in the corpus, not the model, because tool menus had been built from a single global pool of about fifteen names. The repository contains the corrected data generator, evaluation harness and a demo.

The safety numbers need care too. The instruct model handled 89.0% of 3,341 harmful and benign test attempts well across 13 languages and 11 categories. Performance is weakest in some languages: Igbo 75%, Yoruba 82%, Setswana 83%, isiXhosa 87%. A model did the grading. Google's Gemma 3 12B instruct judged every answer, and the card says no native speaker has rated an answer yet. Vambo also says the released checkpoint narrowly missed its own pre-set release rule on the privacy, drugs, violence and degenerate-output thresholds. The base model has no safety evaluation because it hasn't been instruction-tuned.

Running MORENA Locally: GGUF Builds, Chat Tokens and Nano​

For developers, the practical part is that all of this can be downloaded and run on an ordinary PC. Each checkpoint ships as bf16 safetensors with a tokenizer, a SHA256SUMS file and modeling_morena.py, a plain PyTorch reference implementation with no dependency on Hugging Face's transformers library. For llama.cpp users, Vambo publishes llama.cpp build of the instruct model: f16, Q8_0, Q4_K_M, plus GGUF builds of the mini and nano. Each GGUF is verified for logit equivalence against the safetensors weights, so the conversion shouldn't change the model's outputs. TechRadar reports that a CPU build runs offline on a laptop. That fits the developer's own tool-calling work, which targets a laptop CPU.

The chat format is the easiest thing to get wrong. MORENA doesn't use the familiar chat-template strings. A single reserved token opens each turn: <reserved_0> for the user and <reserved_1> for the assistant (token IDs 3 and 4). The instruct card warns that <|user|>-style strings aren't in the vocabulary and produce degenerate output. The GGUF card's example invocation is llama-completion -m morena-1.5b-instruct-f16.gguf -p "<reserved_0>\nNdeipi guta guru reZimbabwe?\n<reserved_1>\n" -e -n 64 --temp 0. The sample prompt is Shona and asks for Zimbabwe's capital city.

Each size has a different job:

Vambo's collection notes say the mini beats every outside model on 11 of 12 languages. The nano is intended as 0.2B trunk for ASR rescoring, keyboards and normalisation, meaning a cheap component inside speech-recognition, predictive-text and text-cleanup pipelines rather than a chatbot. TechRadar reports Vambo's claim that nano costs 58 times less per byte than Lugha-Llama-8B and generates 104 tokens per second on one A100. Those throughput figures come from Vambo and haven't been reproduced elsewhere.

On licensing, commercial use is permitted, including fine-tuning and redistributing derivatives, under the terms of the licence. The training corpus is not released with these weights. That makes MORENA open-weight but not reproducible. You can ship products on it, but you can't rebuild it.

What MORENA Means for Developers Targeting African Languages​

If you build software for speakers of these twelve languages, MORENA is worth evaluating now, especially the base model as a fine-tuning starting point and the nano as a pipeline component. If you want a general-purpose assistant, a retrieval-augmented question-answering system or an autonomous tool-using agent, the model card itself says to wait, or to budget for significant fine-tuning and your own evaluation. Native-speaker review is essential before any customer-facing deployment, because none has happened yet.

• The 1.5B base scores 1.408 bpb on Vambo's twelve-language African benchmark, the best of 26 models it tested, just ahead of Lugha-Llama-8B at 1.423 and far ahead of Gemma 3 12B at 2.159.
• MORENA is stronger translating English into African languages (45.8 chrF++ for the instruct model) than translating out of them into English, where both checkpoints trail MADLAD-400-3B and Lugha-Llama-8B.
• The 98.1% tool-selection figure only applies when the tool marker is prefilled. Against a real Paystack API menu, the developer's own testing found the released models chose correctly 9% of the time.
• Chat prompts must use the <reserved_0> and <reserved_1> turn tokens, because standard <|user|> templates produce degenerate output.
• GGUF builds in f16, Q8_0 and Q4_K_M let you run MORENA through llama.cpp on local hardware, and the Apache 2.0 license allows commercial fine-tuning and redistribution.
• About 28% of the tokens the model saw during training were machine-translated, and the corpus isn't published, so test your own domain text rather than relying on benchmark averages.

MORENA's claim to beat much larger models is real within its own benchmark. A tokenizer designed for these languages and a training budget of roughly $25,000 to $40,000 were enough to put a 1.5B model ahead of 8B and 12B systems at modelling Yoruba, isiZulu and Nigerian Pidgin text. The same documentation shows it isn't yet a dependable assistant: it over-refuses, it can't use tools on its own, and native speakers haven't reviewed its answers. The developer's corrected tool-calling data generator is already on GitHub, so the next checkpoints may address the tool-use gap. Independent native-speaker evaluation hasn't been published yet, and that is what production teams will need before relying on it.