Abstract:
Chinese artificial intelligence company MiniMax released the new generation model M3.1 Flash on October 19, 2026, claiming that the model has achieved significant progress in areas such as agent tasks, programming and network security, while increasing the online inference speed to nearly 3 times that of the previous generation and reducing the cost of use by approximately 70%. However, shortly after the model was released, a controversy surrounding its independent evaluation results quickly attracted attention:
Some researchers suspect that MiniMax may have been exposed to some test questions used by the evaluation agency Artificial Analysis when the previous generation M3 model was evaluated, and used the relevant content to train M3.1 Flash, allowing the new model to achieve exceptionally outstanding results in specific benchmark tests.
It should be emphasized that as of the time of the release of relevant reports, this is still an accusation based on limited phenomena, and there is no public evidence to prove that MiniMax indeed stole or used the evaluation questions. At the heart of the controversy is a long-standing conundrum in the artificial intelligence industry: whether a model's high scores in benchmark tests reflect a true improvement in general capabilities, or whether they come partly from optimization of the test itself.
The key to this controversy lies in Artificial Analysis’s AA-Omniscience review. This is a test that focuses on factual knowledge and answer reliability. The total number of questions in the question bank is about 6,000, of which only about 600 are public, and the remaining questions are not publicly released. According to conventional understanding, undisclosed questions can reduce the possibility of the model improving test scores through specialized training, but if model developers have access to these questions during the evaluation process, this protection mechanism may be challenged.
The timeline of the controversy provides an entry point for speculation. MiniMax released the M3 model through API on June 1, 2026; Artificial Analysis announced the relevant evaluation results on June 4; MiniMax only disclosed the model weights on June 12. In the stage when model weights are not yet open, third parties can usually only test through the API provided by the model developer, and API requests will go through the inference server operated by the developer.
The questioning researchers believe that this arrangement may theoretically bring about a risk: if the evaluation agency sends requests containing private test questions to the model API, and the model developers can identify these requests, then the developers may have access to the questions and their context that were originally not public. However, having access to API requests does not mean that someone has collected, saved or utilized these requests. Existing public information does not prove that MiniMax has taken such actions.
The controversy further heated up due to a test shared on social media by an artificial intelligence researcher named Binkg. He said that he chose an AA-Omniscience question that had been published and slightly modified the question conditions so that its formulation was similar to the original question, but the correct answer changed. According to him, other tested models were able to give answers to the modified questions, but MiniMax M3.1 Flash still output the answers in the original benchmark test, even if the model's inference intensity was adjusted to the highest level.
If this phenomenon could be independently replicated, it would indeed be cause for concern. For a model that relies on reasoning to solve problems, when the problem conditions change and the correct answer changes, the model should reanalyze the conditions instead of mechanically repeating the answers it has seen before. If the model keeps outputting answers that correspond to the original question, it may mean that it has memorized a specific question and its answer and is not handling the modified question correctly.
However, a single case is not enough to prove that the model developer stole the question bank. The model may also give original answers because there are similar questions in the training data, over-association with the question statement, or reasoning errors when handling changes in conditions. To confirm whether the problem comes from training data contamination, inference flaws in the model itself, or other factors, more independent testing, complete experimental records, and comparison of model performance are needed.
Another concept involved in this controversy is "benchmaxxing". This term is often used to describe the behavior of model developers who over-optimize for a specific benchmark. For example, developers may use public test questions for specialized training, adjust prompt words or inference strategies, or repeatedly improve the model based on benchmark test results to achieve high scores on specific rankings, but may not necessarily simultaneously improve its general ability in real tasks.
Brushing the rankings is not the same thing as directly obtaining private test questions from model developers. Optimizing against public benchmarks does not technically necessarily mean a violation; however, if developers use unpublished test questions from evaluation agencies for training, it may undermine the effectiveness of independent evaluations. For businesses and developers who rely on the ability of third-party leaderboard comparison models, both scenarios are cause for concern, but require different evidence to judge.
The attention that MiniMax M3.1 Flash received this time is also related to the results announced when it was released. MiniMax said the model has made significant progress in areas such as complex agent tasks, programming, computer operations and network security. The company claims that the online reasoning speed of the model has reached nearly three times that of the previous generation, and the overall usage cost for agent tasks has dropped by 70%.
In terms of price, MiniMax’s API input price for M3.1 Flash is US$0.15 per million Tokens, the output price is US$0.60 per million Tokens, and the cache input price is US$0.015 per million Tokens. The model has 428 billion parameters, of which approximately 23 billion are activated for each inference. Using this sparse activation design can reduce the number of parameters actually called per calculation while retaining large model capacity.
Artificial Analysis’ independent evaluation also shows that the comprehensive intelligence index of M3.1 Flash is significantly improved compared to the previous generation M3. According to relevant reports, the M3.1 Flash’s intelligence index score at the highest inference intensity was 42 points, 13 points higher than the M3’s 29 points; the average cost of completing each task in the index evaluation also dropped from approximately US$0.51 to approximately US$0.22. The relevant results make it highly competitive among low-cost models.
However, the comprehensive index does not fully reflect all capabilities of the model. Artificial Analysis also found in the AA-Omniscience test that M3.1 Flash has obvious factual reliability issues. Although the model answered more than half of the factual knowledge questions correctly, it often tended to give wrong answers to questions it couldn't answer correctly, rather than admitting that it was unsure. According to the statistical definition used in this review, the hallucination rate reaches 93%.
This number does not mean that 93% of all the model's answers were wrong. This indicator focuses on the proportion of incorrect answers in which the model directly gives wrong answers instead of partial answers or refusals. It reflects the questions the model tends to continue answering under uncertainty, rather than the overall error rate across all outputs.
This also shows that the model's high performance in complex reasoning, software development, and agent workflows does not mean that it is equally reliable in all factual tasks. For businesses that need to automatically modify business records, process financial documents, research customer information, or perform multi-step operations, the ability of a model to identify the boundaries of its own knowledge may be as important as its comprehensive intelligence index.
Existing public materials have not provided sufficient independent investigation conclusions to confirm that MiniMax has obtained or used private review questions, and MiniMax has not publicly responded to this specific allegation.
This controversy reflects that as AI model competition increasingly relies on third-party rankings, the trust mechanism between evaluation agencies and model developers has become increasingly important. Simply setting some questions as non-public does not completely eliminate the risk of test content being accidentally exposed during the evaluation process. In the future, evaluation agencies may need to adopt methods such as regularly changing question banks, using temporarily generated questions, conducting adversarial tests, and reviewing the model's response to subtle changes in questions to reduce the possibility of the model memorizing or optimizing for specific questions.
For MiniMax M3.1 Flash, what really needs to be answered is not just whether it can achieve high scores on the leaderboard, but whether these results can continue to hold true in unseen problems, tasks with changing conditions, and real business scenarios. The current accusations still need to be verified, but the discussion around the boundaries between evaluation data security, model memory and reasoning capabilities has once again become an unavoidable issue in the artificial intelligence industry.
Comments