For decades, video games have served as laboratories for testing the capabilities of various AI algorithms. Whether they are classic platformers or more complex strategy games, these games provide a way for AI systems to learn how to act, adapt to changing environments, and optimize their decisions in order to earn rewards.
For example, the Hao AI Lab recently conducted experiments using Super Mario Bros. In these experiments, each AI model was required to generate Python code to control the game. The results showed that the models became increasingly proficient at planning complex maneuvers and developing various game strategies. In the tests, Claude 3.7 performed the best, followed by Claude 3.5, while reasoning models—such as OpenAI o1—performed particularly poorly.
The use of games offers several advantages for AI development:
-
Large Amounts of Data and Rapid Simulation: The abstract and relatively simple game mechanics enable AI models to simulate extensive gameplay in a short period. This is essential for reinforcement learning since the system continuously receives feedback on which actions lead to success.
-
Clear Goals and Rules: Clearly defined objectives in games (e.g., completing a level or defeating an enemy) help AI algorithms converge quickly and simplify performance measurement.
-
Controlled Environment: The simplified nature of games allows researchers to test learning processes in a controlled setting, which facilitates rapid experimentation and fine-tuning of methods.
Although games are excellent testing grounds, several experts have pointed out that the benchmarks provided by games do not necessarily reflect the complexity of the real world:
-
Lack of Generalizability: AI models are often optimized exclusively for the specific mechanics of a game, so even a slight change can result in significant performance degradation. For example, in the Super Mario Bros. experiments, timing is crucial, and "step-by-step reasoning" models (like OpenAI o1) frequently fail to execute the required actions quickly enough.
-
Oversimplified Environment: Although games effectively simulate certain aspects of AI decision-making, many real-life social and economic interactions involve numerous additional dimensions and variables that games cannot adequately represent.
-
Issues with Metrics: Success in games is often overestimated. As experts such as Richard Socher and Mike Cook have noted, game-based benchmarks do not always provide a comprehensive picture of whether an AI system is capable of genuine, human-level problem solving.
The AI Benchmark Crisis
Recently, an increasing number of researchers have questioned the relevance of existing AI benchmarks—not just those established through games. Andrei Karpathy, a former researcher and founding member of OpenAI, is one of the most prominent critics. In a brief post on X, he stated that he no longer trusts any of the current benchmarks. Other experts, such as Richard Socher, founder of You.com, and Noam Brown, who has developed AI systems that excel in games (e.g., poker), view game-based testing as problematic. They argue that games provide an overly simplified environment that fails to capture the complex, long-term decision-making processes of real life.
Conclusions
The rapid development of artificial intelligence makes it challenging to keep pace with evolving benchmarks, leaving many researchers uncertain about the credibility of the test data. One reason for this crisis of confidence is that many tests rely on outdated measurement practices, such as game-based testing. At the same time, there is an increasing need for robust benchmarks because without them it is difficult to determine whether AI developments are moving in the right direction or which model is superior. It is no wonder that Scale AI—originally focused on data labeling—has experienced rapid growth as it has shifted to verifying the accuracy of AI systems and validating the correctness of their decisions. Their success, alongside the current crisis of confidence, demonstrates that there is an immediate, lucrative market opportunity in this field.
