Video Games in Artificial Intelligence Testing

 For decades, video games have served as laboratories for testing the capabilities of various AI algorithms. Whether they are classic platformers or more complex strategy games, these games provide a way for AI systems to learn how to act, adapt to changing environments, and optimize their decisions in order to earn rewards.

For example, the Hao AI Lab recently conducted experiments using Super Mario Bros. In these experiments, each AI model was required to generate Python code to control the game. The results showed that the models became increasingly proficient at planning complex maneuvers and developing various game strategies. In the tests, Claude 3.7 performed the best, followed by Claude 3.5, while reasoning models—such as OpenAI o1—performed particularly poorly.

The use of games offers several advantages for AI development:

  • Large Amounts of Data and Rapid Simulation: The abstract and relatively simple game mechanics enable AI models to simulate extensive gameplay in a short period. This is essential for reinforcement learning since the system continuously receives feedback on which actions lead to success.

  • Clear Goals and Rules: Clearly defined objectives in games (e.g., completing a level or defeating an enemy) help AI algorithms converge quickly and simplify performance measurement.

  • Controlled Environment: The simplified nature of games allows researchers to test learning processes in a controlled setting, which facilitates rapid experimentation and fine-tuning of methods.

Although games are excellent testing grounds, several experts have pointed out that the benchmarks provided by games do not necessarily reflect the complexity of the real world:

  • Lack of Generalizability: AI models are often optimized exclusively for the specific mechanics of a game, so even a slight change can result in significant performance degradation. For example, in the Super Mario Bros. experiments, timing is crucial, and "step-by-step reasoning" models (like OpenAI o1) frequently fail to execute the required actions quickly enough.

  • Oversimplified Environment: Although games effectively simulate certain aspects of AI decision-making, many real-life social and economic interactions involve numerous additional dimensions and variables that games cannot adequately represent.

  • Issues with Metrics: Success in games is often overestimated. As experts such as Richard Socher and Mike Cook have noted, game-based benchmarks do not always provide a comprehensive picture of whether an AI system is capable of genuine, human-level problem solving.

The AI Benchmark Crisis

Recently, an increasing number of researchers have questioned the relevance of existing AI benchmarks—not just those established through games. Andrei Karpathy, a former researcher and founding member of OpenAI, is one of the most prominent critics. In a brief post on X, he stated that he no longer trusts any of the current benchmarks. Other experts, such as Richard Socher, founder of You.com, and Noam Brown, who has developed AI systems that excel in games (e.g., poker), view game-based testing as problematic. They argue that games provide an overly simplified environment that fails to capture the complex, long-term decision-making processes of real life.

Conclusions

The rapid development of artificial intelligence makes it challenging to keep pace with evolving benchmarks, leaving many researchers uncertain about the credibility of the test data. One reason for this crisis of confidence is that many tests rely on outdated measurement practices, such as game-based testing. At the same time, there is an increasing need for robust benchmarks because without them it is difficult to determine whether AI developments are moving in the right direction or which model is superior. It is no wonder that Scale AI—originally focused on data labeling—has experienced rapid growth as it has shifted to verifying the accuracy of AI systems and validating the correctness of their decisions. Their success, alongside the current crisis of confidence, demonstrates that there is an immediate, lucrative market opportunity in this field. 

Share this post
Artificial Intelligence in Network Management and Maintenance
Ericsson recently presented its strategic plans for 2025 at the Mobile World Congress 2025 (MWC25). These ideas are particularly intriguing as they demonstrate how artificial intelligence is being integrated into industrial processes that impact our daily lives—yet remain unnoticed as long as they function smoothly.
GTC 2025: NVIDIA's Blackwell-Based Servers and DGX Station
The GTC (GPU Technology Conference), held annually since 2009, will be hosted by NVIDIA this year from March 17 to 21. The conference is designed to showcase the latest developments and to promote collaboration and further innovation across different industries. It is attended mainly by developers, researchers, and technology leaders. NVIDIA CEO Jensen Huang has been saying for some time that companies will become token factories in the future—meaning that every workflow will be supported by artificial intelligence. Currently, large servers play a major role in this process, but AI integration will increasingly extend to personal computers. In the future, computers and laptops will have hardware capable of running even large language models in the background. This is necessary because programmers, engineers, and almost everyone will work with AI assistance.
Fedora 42 Beta Available
Fedora 42 beta is now available for testing, with a stable release planned for 15 April. The new version includes several major enhancements designed to improve the user experience, simplify the installation process, and integrate modern desktop environments and technical solutions.
Nature & Technology: Google Calls for Proposals to Support Environmental Solutions
In recent decades, the dramatic loss of biodiversity has created new challenges for the global environmental balance and human well-being. In response, Google's latest announcement introduces three new initiatives aimed at protecting and restoring nature and ecosystems in critical areas.