Which model is best for this task?

Football has spent more than a century refining competition to determine the best teams. I built a task-specific model evaluation system borrowing some of those ideas.

The World Cup's just ended. For the last few weeks, teams from around the world had been competing. The tournament started with groups and then moved into a knockout bracket. Those who won moved forward. Those who lost, went home.

This format moves fast and is quite exciting. Every match is critical. A favourite can disappear after one poor evening, while an unexpected team can put together a great run and reach the final.

The winner becomes world champion.

Whether they were truly the best team is always debatable.

The Champions League follows a similar structure. It starts with a league phase and then moves into knockouts until one team wins the final.

Domestic leagues work differently.

In the Premier League, Bundesliga and other leagues, teams play across a full season. They collect points over many matches and slowly build their position on the table.

One loss does not end the competition. A team can recover and keep moving. Finishing first requires consistent performance against many opponents.

Football uses both formats to answer a similar question:

Who is the best?

That is also the question behind LLM League.

Choosing a model is usually a vibe check for most of us

A lot of model selection looks like this:

Scroll through public LLM leaderboards such as LLM Arena or Hugging Face. Shortlist a few models based on what is indicated to perform well, what is popular and what is within your budget.

Open a playground, paste in a few of your prompts, read the responses and decide which of the shortlisted models works best for the task.

LLM League is a task-specific model evaluation system

Usually, your tasks have their own unique requirements, tone, rules and edge cases. Cost also matters just as much as raw quality.

The most useful way to know how models perform is to pit them against each other and see how they perform on your specific task.

That is what LLM League does.

You start by choosing the models you want to evaluate.

You then define the task. This includes the prompts the models should respond to, the judging criteria and any system instructions they need.

Each model becomes a contestant.

Judge models compare the contestants' responses and decide which one performed better.

The competition can run as a league, a knockout or both.

League format

In a league, every contestant faces every other contestant.

A win earns three points. A draw earns one. The results build a standings table showing matches played, wins, draws, losses and total points.

A contestant can lose one fixture and still finish first after performing well against the rest of the field.

The league takes longer to run because the number of comparisons grows as more contestants enter.

It gives you a fuller ranking and shows which models perform well across several matchups.

A live league table updates as each matchup is decided, with contestant and judge costs kept visible.

Knockout format

A knockout is faster.

The contestants enter a bracket and compete in pairs. The winner moves to the next round until one model remains.

It uses fewer comparisons than a full league, and the path through the bracket matters more. A strong contestant can lose one close match and leave the tournament early.

Knockout matches must produce a winner.

For that reason, they require an odd number of judges. Each judge has to select one contestant. Draws are not allowed. The contestant with the majority of votes moves forward.

When the first set of prompts leaves a match level, additional tiebreak prompts can be used. These work like extra time and give the contestants another chance to separate themselves.

A completed knockout keeps the path through every round visible, from the opening fixtures to the champion.

Checking for position bias

Comparing two responses sounds simple.

Give both to a judge model and ask it to choose the better one.

The position of the responses can affect the verdict. A judge may favour the answer shown first or lean towards the one shown second. This phenomenon is usually attributed to LLMs' positional bias or the recency effect.

LLM League avoids this bias using optional home-and-away comparisons.

In the first comparison, the judge sees Response A followed by Response B.

The second comparison flips the order: Response B followed by Response A.

The contestants are anonymous. The judge does not know which models produced the responses. It only sees the original prompt, the judging criteria and the two answers.

The judgment view keeps both response orders and the resulting conflict visible for inspection.

When enabled, flipping the responses makes positional bias easier to detect. It costs more because each judge reviews both response orders, so it can also be turned off when a single comparison is enough.

In league matches, a result that changes after the order is reversed can be recorded as a draw or marked as conflicted.

When home-and-away comparisons are enabled, knockout judges still have to choose a winner after reviewing both positions.

Do the league and knockout agree?

One useful comparison in LLM League is whether both formats produce the same result.

The league and knockout can use the same contestants, prompts and configuration. When both produce the same winner, that is a useful supporting signal, although one execution is not conclusive. A different winner can mean the field is close, or that the format and normal response variation affected the outcome.

However, you can rerun a tournament several times and compare the results. A model may win one league, finish second in another and drop further down in the next. Another may rarely finish first but remain near the top across most runs.

The prompt can also change. The prompt is the pitch the contestants are playing on. One prompt may reward concise answers. Another may require more detail. Another may focus on tone, formatting or instruction following.

A changed prompt creates a new pitch within the same evaluation. Results remain identifiable by pitch, while response-sampling methods are aggregated separately.

LLM League combines completed runs across those pitches into a history of points, finishing positions and honours. This shows whether a model stays strong when the task wording changes, and makes an unusual result easier to distinguish from a repeated pattern.

Fresh performances for every fixture

Models do not always produce the same response to the same prompt.

That variation is part of how they behave in real use. A model can give an excellent response once and a weaker one the next time.

Football works the same way. Teams are judged on how they perform in each match. They do not submit one performance at the start of the season and reuse it against every opponent.

By default, LLM League generates fresh contestant responses for every fixture.

This costs more, but it samples more of the variation users may encounter when a model handles the task repeatedly.

Each fixture therefore tests a new answer instead of carrying one unusually strong or weak response through the entire competition.

The tournament can also be configured to reuse responses.

With reuse enabled, each contestant answers a prompt once for the selected pitch. That response can then be used wherever the contestant appears in that pitch's league or knockout.

This is useful when you want to keep the contestant performances fixed while comparing tournament formats or repeat judgments. Responses are never borrowed from a different pitch.

Fresh responses are recommended for normal model selection. Response reuse is the controlled option.

A panel of referees

The judge models are the referees.

Once a model is selected as a judge, that exact model cannot also compete as a contestant in the same tournament. This avoids a judge directly evaluating its own responses, although related models from the same provider or family can still appear in the panel and contestant pool.

LLM League also tracks how often judges agree with each other.

A judge that regularly disagrees with the rest of the panel may be interpreting the criteria differently. It may also be a poor fit for the task.

That information helps when choosing judges for future tournaments.

Agreement analysis shows where the panel reached the same verdict and which judges most often matched their peers.

Using the strongest frontier models as referees may feel like the safest option. It can also become expensive.

In some evaluations, the judges can cost more than the contestants.

LLM League tracks the decisions and costs of each judge. This makes it possible to see whether a cheaper judge reaches similar conclusions to a more expensive one.

A mid-priced model that agrees with the frontier judges most of the time may be enough for the next tournament.

Track what everything costs

A contestant can finish first while costing several times more than another model that performed almost as well.

LLM League tracks the cost of contestant responses generated in a run and the judge decisions made during that run. Reused responses are labelled separately rather than charged to the new run again.

At the end of the tournament, you can see how much each contestant cost, how much the judges cost and the total cost of the evaluation.

This puts quality and price in the same view.

The model at the top of the table may still be the right choice. A cheaper model that keeps finishing close to it may be a better fit for a task that will run thousands of times.

Football clubs look at performance and the wage bill.

Model selection should do the same.

Bring in the next challenger

New models will keep launching.

Public leaderboards and release posts remain useful for finding contestants worth testing.

Once shortlisted, a new model can enter a new evaluation and compete on the actual task. Changing the contestant pool, judge panel, criteria or format creates a different evaluation rather than another pitch.

It may take the title. It may offer similar quality at a lower cost. It may perform well on general benchmarks and struggle on your prompts.

The tournament gives it a place to prove itself.

What LLM League is trying to do

Which model is best for this task?

Run a league, a knockout or both. Repeat the tournament, change the pitch and track performance alongside cost.

Set the task. Pick the contestants. Play the fixtures.

The invincible that lost the cup

Earlier, this essay asked whether the league and knockout agree. Here is a real case where they did not.

Twelve contestants were evaluated on support response quality, playing a league season followed by a knockout, with fresh responses for every fixture. Then the whole competition was run again.

gpt-oss-120b won both leagues without dropping a match. Twenty-two fixtures, twenty-two wins, including both league meetings with kimi-k2-5.

An invincible season. Twice.

Then came the knockouts. In both brackets, it was kimi-k2-5 that knocked gpt-oss-120b out, once in the final and once on the road to it, before lifting the cup both times.

The league champion went home without a cup. Twice.

Nothing went wrong. The verdict difference already showed the top two were never far apart, and between closely matched contestants a single fixture is close to a coin flip. A knockout hands the trophy to one flip. The league asks the question eleven times against eleven opponents, and both seasons returned the same table from top to bottom.

Season one: gpt-oss-120b takes the league, 11 wins from 11.

Cup one: kimi-k2-5 beats gpt-oss-120b in the final.

Season two: another unbeaten league title.

Cup two: kimi-k2-5 knocks gpt-oss-120b out on the road to the trophy.

The combined history: same top two, same bottom two, both titles.

The executions. The two with fresh per-match responses are the seasons in this case study.