Evaluating AI is Hard; Not Evaluating AI is Stupid

There are a wide variety of methods used for AI evaluation, for a wide variety of purposes. AI developers use some benchmarks for hill-climbing, and (ideally others ones) for reporting capabilities. AI users and deployers have evaluations to assess fit for specific purposes. AI governance and ethics researchers have evaluation methods that can assess safety from misuse, adherence to ethical principles, avoidance of out-of-bounds topics, or loss of control risks. And AI regulators and government stakeholders sometimes use those evaluations for their own purposes.

But evaluation isn’t all one thing, it uses a variety of paradigms, and is part of a broader but messy and overlapping spectrum of approaches that range from economic modelling and social impact evaluation, to audits, to evaluation, to red-teaming. Each of these can be valuable for understanding different things, and all are imperfect in ways that can be misleading.

What’s Being Evaluated?

AI systems are not human, and succeed and fail in very different ways than humans do. Evaluation and measurement of tasks for humans is often fraught and difficult to do well, as I’ve argued in the past, and doing so for ML or LLM based systems can be even worse. The performance on a given evaluation is therefore rarely easy to interpret, despite often sounding intuitive. Sometimes, the disconnect between 1) what an evaluation purports to measure, and 2) what it actually can tell us, is even unintentional.

Even when looking at the same evaluation of the same AI system, there are a variety of confusions; the framework for the evaluation can differ, the exact version of the model, or prompt, or training data cutoff can change, they can run the evaluation at a different model temperature, and when attempts to address these obvious sources of variation are used, it may not help because language models aren’t deterministic, for various reasons, and results can therefore vary.

This is made far worse when comparing models; there’s no way to set identical training data cutoffs for already trained models, and later models often have seen results or even full datasets of evaluations and correct responses used to evaluate earlier models, which contaminates the models, and makes the results invalid. Even when contamination does not occur, model developers often implicitly aim for better performance on known evaluations, or “teach to the test” with proxy datasets. This does not mean the evaluations are meaningless, but it makes comparisons very difficult.

What Does The Score Mean?

After getting past the above issues, we are faced with another; the numbers reported by evaluations often don’t have clear meaning. Is a score of 90% on an evaluation excellent, or merely good? It depends. And comparing different models can give you a (noisy) indication of relative capability, but does not tell you much in an absolute sense.

In many cases, the evaluators have some “human level” value, where they give people the same test and grade their answers similarly. This can be a useful indication, but it’s not a panacea for comparability on either side. Are the humans in question domain experts, or undergraduate students? Are they given enough time to provide answers? Is there any fair comparison between models, which run at computational speeds, and humans, who typically respond more slowly and get fatigued? Are the models given each question in a new context window so that they lack context about the class of questions asked, or are they given them all together so that they exhibit failure modes due to context lengths?

All of the above is to say that all model evaluations are wrong, but to continue paraphrasing George Box, some are useful.

To fail and know we failed

There is no complete solution to the problem, but there are two obvious failure modes; trusting the evaluations, and ignoring them. The post above has given plenty of reasons to question evaluations, but the alternative of ignoring them leads to a different class of idiocy.

There are many who have said that AI systems will not be as capable as humans at different specific tasks or general fields, and have been proven wrong. Some admit their mistake, but others continue to seek reasons to ignore growing capabilities. There are also those who predicted that we would have superhuman general artificial intelligence by now, and some insist that the capabilities of current models prove that they are correct. Both groups are ignoring the imperfect but useful evidence we have from evaluations.

Policymakers desperately need good information about AI capabilities to inform their decisionmaking. Evaluations provide something less than ideal information, but are far better than the very common alternative of evidence-free policymaking.

Interpreting Evaluations

The first critical requirement for useful interpretation is information; what exactly was done, by whom, how, and what were the outputs.

This is currently a mess; many AI firms do not release enough information for users and policymakers to understand what was evaluated, many academic projects do not release code or results, the purpose and scope of evaluations is often implicit or unclear, and what system was evaluated, including model version, framework, and tooling is sometimes not reported.

The second critical requirement for useful interpretation is context; what are the implications of the capability or performance, what has changed since prior versions or what is the projected trajectory over time, and whether the evaluation is a valid measure of what it purports to be.

This is also currently a mess; it’s often unclear what the risks are, it’s unclear if the capabilities are accelerating or reaching a maximum, and it’s unclear whether this is primarily a function of the evaluation itself or the underlying capability.

The third requirement for useful evaluations is actionability. That is, information that doesn’t change what anyone does is in a formal sense valueless. So to make evaluations valuable, there must be some clear framework for what firms are supposed to be doing with the results, either internal policies or external legal requirements. But the value of information in complex environments can be messy.

This is slowly getting fixed; internally with safety frameworks, and externally with regulatory requirements. However, it is difficult to tie narrow technical results into broader frameworks for response, especially when the evaluations themselves are changing over time more rapidly than the frameworks for how to use them.

The Worst Option, or the Pragmatic Option?

The easy alternative is to give up. The default has been not to measure well, not to build robust and interpretable evaluations, and to instead rely on vibes, or external arguments like defaulting to not regulating, or allowing industry to “self regulate” while policymakers lag behind in understanding and in ability to respond.

But as the saying goes, the most valuable data point is the first one. Even bad evaluations give some information, especially when those receiving the information understand what is being evaluated. Expecting perfection instead of understanding the evaluations and accepting the tradeoffs is wasting everyone’s time. The ISO has a standard for LLM evaluation that is being developed, and until then, there are various checklists; Betterbench for technical benchmarks, RAND’s preliminary GPAI checklist, and the STREAM checklist for dangerous misuse evaluations.

But these are different. Partly, different stakeholders have different needs, and evaluations that serve one set of needs well don’t necessarily give useful information for others. This is not only contingent, it’s sometimes fundamental. For example, the evaluations that are useful for building models or fine-tuning them are not useful for demonstrating that those capabilities work – the models are literally trained for the test. Similarly, evaluations built for auditing models may need to be non-public so that model developers cannot cheat, but that necessarily means that the methods are less transparent for the public and for policymakers.

But even fundamental tensions don’t make the processes useless, they just require navigating and accepting the tradeoffs. And that does not require abandoning the goal of improving the evaluations. There is also a cross-stakeholder effort to build consensus about which practices are widely endorsed (which I helped lead.) We can still push for better practices which everyone can adopt, and set better expectations.

Some difficulties with evaluation of AI systems are fundamental, and will not be fixed via advocating better practices. But many more are solvable, and the path to doing so requires only that we insist on better practices – most of which are already agreed to in principle. The next step is getting those changes to happen.

Note: Cross-posted from the AI for Developing Countries Forum Blog: https://af.net/blog/davidmanheim/evaluating-ai-is-hard/