Will the leaderboard winner solve your problem?

Suppose you need a model to spot a useful clue in a fresh incident report. Which leaderboard measures that particular job?

Use rankings to choose what to try

Howard looks at a top-ranked model he has not tried. Public questions may require different skills from your work, and some may be present in training. A correct answer alone cannot distinguish recognition from a new solution.

these metrics are not particularly well aligned with real life usage

Change a familiar question

Bubeck asks GPT-4 for a proof that there are infinitely many primes, with each line rhyming. Now you can inspect mathematical validity and the added constraint separately. One successful demonstration suggests a capability; it does not estimate its success rate.

we don't know what gpt4 was trained on I don't know what gpt4 was trained on

For your incident report, hide the resolution and ask which log line to investigate first. Change one decisive line, agree on what should change with a colleague, and try again. Does the recommendation follow the evidence?