Amsterdam's Digital Services Department published "Grip on LLMs," a benchmark suite evaluating 30+ language models across six dimensions tailored for Dutch public-sector use. The paper, released on arXiv August 10, was written by civil servants Laurens Samson, Iva Gornishka, and Gossa Lô alongside University of Amsterdam researchers Yuki M. Asano and Sennay Ghebreab. It is the first systematic framework to jointly weight public administration values and non-English language requirements—a gap that blocked responsible LLM procurement across EU governments.
The six dimensions—factuality, honesty, social bias, energy consumption, cost, and training data transparency—were shaped by an advisory board including a sustainability officer, privacy officer, diversity and inclusion adviser, and business owners. User research and civil servant surveys informed the criteria. The result is a scorecard procurement officers can defend in committee.
| Dimension | What It Measures | Key Stakeholder |
|---|---|---|
| Factuality | Accuracy of model responses in Dutch | All users / committee reviewers |
| Honesty | Model's acknowledgment of its own uncertainty | Citizens receiving advice on benefits or zoning |
| Social Bias | Demographic bias in Dutch-language outputs | Diversity & inclusion adviser |
| Energy Consumption | Wh consumed per query | Sustainability officer |
| Cost | € per million tokens | Business owners |
| Training Data Transparency | Openness and provenance of training data sources | Privacy officer / EU procurement rules |
Testing at amsterdam.github.io/grip-on-llms runs experiments in Dutch, which shifts results materially versus English benchmarks. Energy footprints span two orders of magnitude: TinyLlama-1.1B draws 0.63 Wh per query at €0.01 per million tokens; OLMo-2-0325-32B-Instruct hits 68.59 Wh at €0.89. GPT-4o costs €1.55 per million tokens with no disclosed energy figure. For a city running a chatbot at population scale, a 29× difference in model size (1.1B vs. 32B parameters) yields no proportional quality gain—but generates roughly a hundredfold energy penalty.
| Model | Parameters | Energy (Wh / query) | Cost (€ / M tokens) |
|---|---|---|---|
| TinyLlama-1.1B | 1.1B | 0.63 | €0.01 |
| OLMo-2-0325-32B-Instruct | 32B | 68.59 | €0.89 |
| GPT-4o | Not disclosed | Not disclosed | €1.55 |
Factuality and honesty are independent properties governed by distinct mechanisms. A model answering correctly most of the time need not acknowledge uncertainty. For government use—where confidently wrong responses on benefits or zoning cause real harm—both dimensions must be evaluated separately. High factuality does not imply high honesty.
Bias scores are largely orthogonal to quality and cost. Spending more on a frontier model does not reduce demographic bias in Dutch-language outputs. Training data transparency qualifies as a first-class dimension: only seven of 30+ models tested—TinyLlama-1.1B, SmolLM3-3B, EuroLLM-9B, OLMo-2-1124-7B, OLMo-2-0325-32B-Instruct, Apertus-8B, and Apertus-70B—carry fully open training data. Full disclosure is a hard requirement under some EU procurement rules.
| Model | Parameter Scale |
|---|---|
| TinyLlama-1.1B | 1.1B |
| SmolLM3-3B | 3B |
| EuroLLM-9B | 9B |
| OLMo-2-1124-7B | 7B |
| OLMo-2-0325-32B-Instruct | 32B |
| Apertus-8B | 8B |
| Apertus-70B | 70B |
The dashboard is live at gripopllms@amsterdam.nl for municipalities replicating the methodology. The framework uses a CC BY-NC-SA license, allowing other public-sector teams to adapt it for non-commercial procurement.
For architects building or procuring LLM-backed services in regulated environments: benchmark in your deployment language, not English. Separate factuality and honesty evaluations—they do not move together. Treat energy and cost as primary dimensions, especially at population scale. Verify training data provenance early; retroactive compliance after deployment is expensive.