Study accuses LM Arena of helping top AI labs game its benchmark

New paper From Ai Lab Corehere, Stanford, MIT and AI2 accuses the LM Arena, an organization behind the popular AI Benchmark Ana Arena, to help the selected group of AI companies achieve better results on the board at the detriment of the rival.

According to the authors, the LM Arena allowed some leading AI companies such as Meta, Openi, Google and Amazon to privately test several variants of AI model and then not publish the results of the lowest performers. This made it easier for these companies to reach a top spot on the platform board, although the opportunity was not provided to every company, the authors say.

“Just a handful [companies] They were told that this private testing is also available and the amount of private testing that some are [companies] It was only received much more than the others, “said Core’s VP of AI research and co -author of the study, Sara Hooker, in an interview with Techcrunch.” This is a gaming. “

Created in 2023 as an academic research project outside UC Berkeley, Chatbot Arena became a reference value for AI. It works by putting answers from two different AI models in the “battle” in the “battle” and asking users to choose the best. It is not uncommon to see unpublished models competing in the Arena under the pseudonym.

The voices over time contribute to the result of the model – and, therefore, their placement on the Chatbot Arena board. While many commercial actors participate in the Chatbot Arena, LM Arena has long claimed that its reference value is impartial and honest.

However, this is not what the authors of work say they discovered.

One company AI, Meta, was able to privately test 27 models of models on the Chatbot Arena between January and March, which led to the edition of the LLE 4 Tech Giant, the authors claim. In launch, the target only publicly discovered a rating of one model – a model that ranked near the top of Chatbot Arena Leaderboard.

In Techcrunh’s email, co-founder of LM Arena and Professor UC Berkeley Ion Stoica said the study was full of “inaccuracy” and “questionable analysis”.

“We are advocating for FER, evaluations focused on the community and invite all models to submit more models to testing and improve their performance performance performance,” LM Arena said in a statement provided by Techcronych. “If the model provider decides to bear more tests than another model provider, it does not mean that the other model provider is treated unjustly.”

Armand Joulin, Chief Researcher on Google Deepmind, also noticed UA Post on x That some of the study numbers were incorrect, claiming that Google sent only one Gemma 3 AI model in the LM Arena for testing before release. Hooker responded to Joulina to X, promising the authors would make a correction.

Allegedly favored laboratories

Labor authors began conducting their research in November 2024 after learning that some AI companies may have received a preferential approach to the Chatbot Arena. In total, they measured more than 2.8 million Chatbot Arena Battles during a five -month part.

The authors say they have found evidence that LM Arena has allowed certain AI companies, including Meta, Openi and Google, to collect more data from the Chatbot Arena by appearing in more “battles in the model”. This increased sampling rate has given these companies an unfair advantage, the authors claim.

The use of additional data from the LM arena could improve the performance of the model on the hard Arena, another reference LM arena maintains by 112%. However, the LM Arena said Post on x This Arena hard performance does not fit directly with the performance of the Chatbot Arena.

Hooker said it was not clear how many AI companies could receive priority approach, but that it was on the LM Arena to increase its transparency regardless.

In a Post on xLM Arena said that several claims in the work does not reflect reality. The organization pointed to blog blog It was announced earlier this week, pointing out that models from the laboratory that did not appear in several Chatbot Arena battles than the study suggest.

One important restriction on the study is that it relied on “self -instance” to determine what AI models in private testing on the Chatbot Arena are. The authors have encountered AI models several times about their origin and relying on the model answers to classify them – a method that is not stupid.

However, Hooker said that when the authors reached for the LM arena to share their preliminary findings, the organization did not dispute them.

Techcrunch contacted Meta, Google, Openi and Amazon – all mentioned in the study – for comment. None answered immediately.

Lm arena in hot water

In the work, the authors invite LM Arena to implement numerous changes aimed at Chatbot Arena more “honest”. For example, the authors say, the LM arena could install a clear and transparent limit of the number of private tests that the AI ​​laboratory can carry out and publicly detect the results from these tests.

In a Post on x, LM Arena rejected these proposals, claiming that she posted the testing information before issue Since March 2024. Benchmarking organization also said that “there is no point in showing results for models before publishing that are not publicly available” because the AI ​​community cannot test the models on its own.

Researchers also say LM Arena could adjust the Chatbot Arena sampling rate to ensure that all models in the arena occur in the same number of battles. The LM arena is publicly sensitive to this recommendation and indicated that she would create a new sampling algorithm.

The work comes a few weeks after the target is caught in reference to playing in the Chatbot Arena about the launch of its Lem 4 Models mentioned above. The target optimized one of the Lem 4 for a “conversational ability”, which helped him achieve an impressive result on the Chatbot Arena board. But the company never released an optimized model – and vanilla version ended up a lot worse on Chatbot Arena.

At that time, LM Arena said that the target was to be more transparent in its approach comparison.

Earlier this month, LM Arena announced that Starting a companywith plans to collect capital from investors. The study increases the supervision of a private reference organization – and whether they can be believed to evaluate AI models without corporate influence of the process of blurry.