LLM benchmarks: What are they and can you trust them?

By

Zuzanna Dubanowska

April 12, 2024

•

9

min read

Large Language Model (LLM) benchmarks have become the battlefield of competing LLM vendors and research labs. Basically, everyone is trying to show that their model is the best…

But is it really that easy to say which LLM is better than others?

Let’s talk about what benchmark datasets are, and what’s the issue with the LLM leaderboard.

What are LLM benchmarks?

What we refer to as “LLM benchmarks” are essentially benchmarking datasets (not to be confused with benchmark models).

The goal of benchmarking LLMs is to see if we can create a model that beats published results on a given dataset by completing a standardized set of tasks.

A so-called state-of-the-art model is the best one on a given dataset for a given problem. It’s the title the fight is for.

There are quite a few benchmarking datasets for general-purpose LLMs, so let’s first get familiar with the most common ones before we start discussing some issues with the LLM leaderboard.

What are the most common LLM benchmarks?

Massively Multitask Language Understanding dataset (MMLU)

The first one is the Massively Multitask Language Understanding dataset, MMLU for short. This benchmark dataset for LLMs measures general reasoning and comprehension acquired by models during pre-training. It consists of 57 categories of knowledge spanning Humanities, Social Sciences, STEM, and more.

Here’s an example — a question from the Microeconomics category:

WinoGrande

Then there’s Winogrande that’s another dataset for common-sense reasoning, formulated as binary fill-in-the-blanks tasks which require common sense to get correctly.

An example from the dataset could be:

HellaSwag

The next one, HellaSwag, is a challenge dataset for evaluating common-sense natural language inference that is especially hard for state-of-the-art models, though its questions are trivial for humans.

GSM8K

We also have GSM8K which assesses the mastery of grade school math...

HumanEval

And HumanEval that’s used to evaluate the functional correctness of model-generated code, for example:

These are some of the most popular LLM benchmarking datasets, but there are basically as many ways to evaluate LLMs nowadays as there are LLMs themselves.

The issue is that when the industry itself is both a currently researched frontier of science and a business opportunity, a conflict of interest arises. On the one hand researchers “just” want to figure AI out — on the other hand, companies start competing for customers, and happily overuse “The Best LLM” title.

That brings us to the next point:

Which LLM (if any…) is the best?

There’s no denying that LLM benchmarks provide a standardized way to measure the performance of LLMs across various tasks. They are the only way to guide practitioners on which model to choose, without actually testing every model themselves.

Plus, they quantify the immense progress the field is making day by day.

Source: https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu

However, since AI became trendy and started attracting a lot of attention, everyone is keeping their eyes peeled on LLM benchmarks to see which LLM is better than others — just like they were watching gladiators fight in an arena.

This has sparked an unfortunate trend in which the top of the LLM performance benchmarks become the target, causing research labs and BigTech to compete to market their models as state-of-the-art.

How to make sense of the LLM leaderboard? Don’t buy everything that you read online. Here are some things to keep in mind before you start reading the tech news or your LinkedIn feed again.

Problems with the LLM leaderboard

BigTech reported results should be taken with a grain of salt, because, well… Proprietary models can’t usually be fairly compared to their competitors.

An example?

A couple of days after the Gemini model suite was released by Google, amongst them Gemini Ultra, the Microsoft-OpenAI coalition responded that GPT4 beats Gemini Ultra on a popular benchmark MMLU, by using a new prompting technique called Medprompt.

The problem here though is that Gemini’s score on MMLU using Medprompt was never reported and we have no clue what the score would have been using this novel prompting technique.

This might have well passed as an interesting contribution to the world of prompting techniques but OpenAI/Microsoft seems to try and one-up Google by announcing the GPT4 shows performance improved by 0.06% compared to Gemini Ultra. The truthfulness of this performance cannot really be verified without more access to the model weights.

Source: https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/

We are not defending either of these BigTech companies here, they are both guilty of bending evaluation scores to fit their narrative.

Here’s another example. Gemini 1.0 Ultra announces it’s the first model to outperform human experts and the previous SOTA GPT-4 but, again, we have no way to reproduce these results as we don’t have access to the model weights.

Sensitivity of LLM benchmarks

Another thing to keep in mind is that LLM performance benchmarks are sensitive and with scores not being standardized, companies might publish the single best score which doesn’t reflect performance robustly.

A recently published paper “When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards” shows that for popular multiple-choice question benchmarks (like MMLU) minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions.

Taking this instability into consideration, how significant and trustworthy are the results reported by the industry?

For example, the best Claude 3 model achieved 0.4% better accuracy on some datasets than GPT-4. Yet, it’s hard to compare it really, since they don’t report whether this is an averaged result over N runs or a best-selected result.

Difficulty of telling which LLM is better

Lastly, we have no way of knowing why one LLM is better than the other. Models from Anthropic, Google, or OpenAI are fully-proprietary meaning we neither have access to model weights nor insight into the training data.

However, even for open-source models like Mixtral from Mistral or Meta’s LLaMa where the code is publicly available, the exact training data remains a mystery.

There have been lots of rumors that companies mix in instruction data at the end of pre-training, or some other special sauce, which could help improve these evaluation scores. While we don’t know if the rumors are true, we can’t debunk them either.

One thing is certain, though. Without access to the training data or code, little can be said with confidence as to why LLM evaluation scores are different.

What to trust when the LLM benchmark becomes the target?

Don’t reject LLM benchmarks, they are great indicators of the immense progress of the field. They're a standardized way to measure performance and the only guidance for choosing models without personally testing each one.

But… Take the LLM benchmark results with a grain of salt. The 0.4% advantage some model has above another could just be a marketing trick.

There are ways to compare the models and pick the best LLM for your needs, though, which we’re going to cover in the next article.

Zuzanna Dubanowska

Similar articles

Artificial Intelligence

LLM benchmarks: What are they and can you trust them?

Zuzanna Dubanowska

Artificial Intelligence

What are AI Hallucinations? It’s a feature, not a bug

"What are they, what causes them, and what can you do to prevent that problem... but is it really a problem?"

Zuzanna Dubanowska

Artificial Intelligence

AIaaS: The What, Why, And Wow of AI as a Service

We're witnessing the rise of AIaaS — Artificial Intelligence as a Service. Learn what AIaaS is and why should you care.

Zuzanna Dubanowska

February 23, 2024

Artificial Intelligence

RAG vs Fine-tuning for your business? Here's what you need to know

Considering custom AI solutions for your business? Discover how Retrieval Augmented Generation (RAG) offers a flexible and affordable alternative to fine-tuning LLMs.

Zuzanna Dubanowska

February 22, 2024

Artificial Intelligence

AI Chatbot Buyer Guide: 6 crucial factors to consider

Looking for the perfect AI chatbot for your business? Read this and learn about the 6 unobvious factors that you should pay attention to.

January 31, 2024

Artificial Intelligence

Is ChatGPT a real polyglot? Problems with multilingualism of LLMs

Let's talk about the limitations of Large Language Models in terms of their multilingual capabilities, why things are the way they are, and what can you do about it.

Zuzanna Dubanowska

January 23, 2024

Artificial Intelligence

GPT Tokens Explained - what they are and how they work

Large language models generate text token by token. But what are tokens exactly? And how do they relate to entropy and data compression?

Piotr Grudzień

October 6, 2023

Artificial Intelligence

GPT-4: How Multimodal Learning Takes Us Closer to Human-level Performance

Let's investigate how the integration of multi-modality in the newly released GPT-4 will enhance its knowledge and capabilities.

Maciej Kowalski

September 29, 2023

Artificial Intelligence

One Thing all AI Breakthroughs have in common

All recent AI breakthroughs had one thing in common: a clever way of generating a huge amount of accurately labelled data.

Piotr Grudzień

September 29, 2023

Artificial Intelligence

ChatGPT: what it is, use cases and limitations

In today’s blog post we will introduce you to ChatGPT - the latest project published by OpenAI. We will explain what it is, suggest some use cases but also mention important limitations.

September 29, 2023

Artificial Intelligence

3 Hard Truths about Generative AI

At Quickchat AI, we think and talk about Generative AI a lot. There are some things that rarely get mentioned in popular content on the topic.

Piotr Grudzień

September 29, 2023

Succeed in the new world redefined by AI.
Don't get left behind.

Start building for free