AI research agents are usually graded against something already known: a correct answer, a rubric, or another AI acting as reviewer. None of those can tell whether a genuinely new scientific model is any good.
A new preprint from researchers at Ohio State, Yale, UC San Diego, Stanford and Princeton tries a different approach. It gives AI agents raw climate observations and asks them to build a model of the El Niño–Southern Oscillation (ENSO), the cycle of Pacific warming and cooling that drives much of the world’s year-to-year climate variability. There is no answer key. The models are judged by how they behave.
Twelve AI agent systems had six hours offline to build models
The benchmark is called SciExam for ENSO. Each of twelve agent systems, pairing a language model with an agent tool such as Claude Code, Codex or Antigravity, worked offline for six hours in three stages.
First, the agent turned raw ocean and wind data from 1980 to 2014 into five monthly indices, such as sea surface temperature and thermocline depth, and had to pass a data check. Second, it wrote its own diagnostic tests for judging a model. Those tests were then locked. Third, it built and tuned a simplified stochastic model of ENSO, using only its own locked tests as feedback.
The agents never saw their scores. Hidden graders evaluated each final model on three things: whether long simulations reproduce ENSO’s observed statistics, whether the model can reconstruct variables it isn’t shown from the ones it is, and whether it forecasts 2015–2024 better than simply assuming current conditions persist. Those forecast years were never given to the agents.
A published ENSO model from 2024 was scored the same way as a reference.
Six of the twelve beat a published reference model
Six of the twelve systems produced final models with higher combined scores than the reference model’s 0.378. Claude Fable 5.1 led with 0.515, followed by Kimi K3 at 0.479 and GPT-5.6-Sol at 0.443. Three more came close.
The agents won in a specific way. The reference model was still clearly best at reproducing ENSO’s statistics. Most agent-built models were better at reconstructing hidden variables and at forecasting, despite seeing only 35 years of data. The reference had been developed with observations covering all the test periods.
The authors also caution that score differences of about 0.01 shouldn’t be read as a ranking.
The models split along a debate scientists haven’t settled
The most interesting result goes beyond the scores.
El Niño warm events tend to be stronger than La Niña cold events, and climate scientists still argue about why. One explanation says the asymmetry comes from nonlinear dynamics in how the ocean and atmosphere interact. The other says even a simple linear system can produce it, if the random “noise” driving it grows stronger in certain states.
The agents were never told about this debate. Yet when the researchers stripped the nine best models down to their essential terms, they split cleanly: four relied on linear dynamics with state-dependent noise, and five on nonlinear dynamics with constant noise. One model could be simplified either way with almost the same score, which is exactly why the question remains open among human researchers too.
The top system still did well when the years were hidden
ENSO history is widely published, so the agents might have recalled the data rather than modelled it. The researchers tested this on the top system by replacing every calendar year with an anonymous label. It still built a competitive model.
Web access produced a twist. Given access to outside data, the agent calibrated its model to an operational El Niño index instead of the task data and scored lower. Given access to scientific literature only, it scored about the same as offline, but built a sparser model whose links followed established ENSO theory.
What this doesn’t show
Each system was run only once, so the results describe twelve particular models, not each system’s typical performance. The task concerned simplified models of one physical system. The combined score depends on how the three dimensions are weighted, though the authors report that equal weights give the same six winners. The structural simplification was checked with the same graders that scored the models, which the authors flag as a reason for caution. The reference model was developed by the benchmark’s lead author in earlier work. And the study is a preprint that hasn’t been peer reviewed.
Agents without an answer key reached an open scientific question
Beating one published model on one benchmark is not the same as better scientific reasoning. But the experiment shows something new: AI agents working without an answer key produced models that land on both sides of a question scientists are still arguing about.