📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B model was trained from scratch on half Italian data and large-scale tokens but scored near chance on Italian school exams. This challenges assumptions about native-language data sufficiency in LLM development.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite its extensive training dataset. This unexpected result raises questions about the effectiveness of native-language investment in European sovereign LLM projects.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, using Italy’s national supercomputing resources and funded through Italy’s PNRR initiative. The project trained models ranging from 350 million to 7 billion parameters, with weights and data published openly from the start. Despite the large-scale training and high-quality infrastructure, Minerva-3B’s performance on the INVALSI tests was near chance, indicating that even substantial native-language data and scale may not suffice for complex academic tasks.
Researchers concluded that dataset size and parameter count, while important, are not the sole determinants of language understanding in LLMs. The empirical findings suggest that the European sovereign-LLM strategy must confront the reality that larger investments may still fall short of achieving deep country-specific knowledge, especially at current parameter scales.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not „Italy did it better than Portugal.“ Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Hugging Face Transformers Cookbook: Fine-Tuning Open-Source Models for Niche Industry Tasks
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Perplexity AI: The Research Playbook: Master AI-Powered Research — From Pro Search to Deep Research, Spaces, and Beyond (AI for Everyone)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-LLM Strategies
The low performance of Minerva-3B on Italian academic tests, despite significant investment, highlights a critical challenge for European sovereign-LLM initiatives: scaling alone may not produce the desired depth of country-specific knowledge. This finding questions the assumption that native-language data and large-scale training inherently lead to high-quality, domain-specific language understanding. For policymakers and AI strategists, it underscores the need to reassess investment levels and methodology to achieve meaningful language mastery, especially for complex tasks like education and public service applications.
Background on Italy’s Minerva and European Sovereign LLMs
Italy’s Minerva project represents a deliberate departure from the European trend of continuation pre-training, opting instead for training from scratch on a massive dataset of 2.5 trillion tokens, with a focus on Italian content. The project was publicly launched in April 2024, with models made openly available, and is considered a flagship example of European sovereign AI infrastructure. Previous efforts like Portugal’s AMÁLIA have taken a different approach, layering specialization onto multilingual models, but have not publicly released their weights or data. The empirical results from Minerva challenge the narrative that native-language data at large scale guarantees high performance on complex language tasks.
„Minerva’s low test scores reveal a structural challenge for Europe’s sovereign-LLM strategy, highlighting that larger investments may still fall short.“
— Thorsten Meyer
Unresolved Questions About Scale and Performance
It remains unclear whether further scaling, different training methodologies, or more targeted data curation could improve Minerva’s performance on complex tasks. The current results are based on one benchmark, and additional evaluations are needed to assess broader capabilities. The ongoing development of Minerva and similar models may provide further insights into the relationship between dataset composition, scale, and language understanding.
Next Steps for Minerva and European AI Policy
The Minerva team plans to continue iterating on training techniques and dataset composition, with upcoming case studies in 2025. Policymakers and researchers will likely reassess investment strategies, potentially increasing native-language data focus or exploring hybrid approaches. Further benchmarks and real-world tests will be essential to determine whether scale or methodology can bridge the current performance gap.
Key Questions
Why did Minerva perform poorly on Italian school exams?
Despite extensive training on large-scale Italian data, Minerva-3B scored only 4.9% on the INVALSI tests, suggesting that dataset size and native-language content alone are insufficient for mastering complex academic tasks.
Does this mean large native-language datasets are useless?
Not necessarily; the results indicate that scale alone may not guarantee high performance. Effective training methods, task-specific data, and model architecture are also critical factors.
What are the implications for European AI efforts?
The findings suggest that European sovereign-LLM initiatives may need to reconsider their investment levels and strategies, potentially focusing more on quality, targeted data, and novel methodologies to achieve desired language understanding.
Will further scaling improve Minerva’s performance?
This remains uncertain; ongoing research will clarify whether increased parameters and data can lead to better results on complex language tasks.
Source: ThorstenMeyerAI.com
