David Silver and Richard Sutton recently shared a paper: Welcome to the Era of Experience. It is highly recommended, esp. for people pursuing true agents and AI.
A clear takeaway of the paper is:
Experience is critical for further progress in AI, including LLMs.
Reinforcement learning is a natural framework for learning from experience.
This is their first written perspective on LLMs.
Some of their talks and interviews:
Is Human Data Enough? With David Silver;
David Silver - Towards Superhuman Intelligence - RLC 2024;
A Perspective on Intelligence by Rich Sutton;
DeepSeek (The Derby Mill Series ep 02);
Rich Sutton Brings Reinforcements - 72nd Conversation.
Short videos by me:
Rich Sutton on Intelligence, LLMs, Scaling laws;
Bitter lesson implies scaling "laws"?.
Here are my comments.
We may need to highlight the importance of ground truth and the difficulties of generalizing/transferring achievements in games, maths, and coding to other problems.
1. Ground truth
2. Objective vs subjective reward/objective functions
3. Model-based RL vs. inverse RL, in particular, RLHF
4. Digital vs physical vs human-in-the-loop
5. Superhuman AI is hard even for maths and coding
6. A brief summary
1. Ground truth
There are three sources of ground truth data:
1) perfect rules/laws/world models,
2) perfect verifiers with formal methods, and
3) experience by interacting with the world.
Sources 1 (eg AlphaZero) and 2 (eg AlphaProof) can guarantee correctness of data. They can be regarded as special types of experience from interacting with special environments, e.g., games AI like AlphaZero with a perfect game rule, so a perfect game engine, and maths and coding with perfect syntax and theorem, so a perfect prover / verifier.
Source 3 (experience) may come with noises/uncertainty, although the world can be viewed as a perfect model itself. We may need to deal with sampling errors, partial observability, imperfect/incomplete information, strategic/adversarial scenarios, and/or weak specifications like code unit tests.
Before the LLMs era, ground truth data was a prerequisite for AI and machine learning. However, many LLMs papers do not have ground truth data, even for maths (matching the scalar result) and coding (passing several tests or executions), not to mention many other problems. LLMs are not perfect world models. There are actually many self-xxx approaches, including recursive self improvement (RSI), attempting to build “information perpetual motion machines” without ground truth in the loop.
2. Objective vs subjective reward/objective functions
Many problems in natural science, formal science and engineering have objective rewards/objectives. We may collect experience from such problems and rely on them for learning. Sources 1 and 2 for ground truth above are two special cases.
Many problems in social science, arts and humanities usually do not have objective objectives, i.e., they have only subjective objectives. In such cases, experts' feedback may be the best signals we can get.
3. Model-based RL vs. inverse RL, in particular, RLHF
For a problem without a reward function, model-based RL and inverse RL, in particular, RLHF, provide principled approaches to learn a reward function.
Imitation learning is not enough, e.g., as shown in AlphaGo vs AlphaZero. However, *iterative* inverse RL can be regarded as model-based RL (for the reward model).
4. Digital vs physical vs human-in-the-loop
Some problems can be fully digital, e.g., many games, maths, and coding.
Many problems are physical, e.g., medicine, manufacturing, and robotics.
Many problems have human-in-the-loop, or even human-centric, e.g., finance, health care, and soccer games.
For physical and human-in-the-loop problems, one factor is the cost of collecting experience.
Mathematician Yu Deng has won the 2026 Fields Medal for achievements tackling Hilbert’s Problem #6 Mathematical treatment of the axioms of physics. It is interesting to investigate whether such microscopic vs macroscopic results may help build better physical world models. However, achievements in maths and physics may not help build better world models with human-in-the-loop. Moreover, there are strong assumptions, like the hard-sphere particle dynamics.
For human-in-the-loop problems, it is hard, if not impossible, to collect ground truth data and/or to build a perfect world model, since people have not have a good understanding of people yet, e.g., how a brain works, how to achieve creativity, and interactions among people make problems harder.
5. Superhuman AI is hard even for maths and coding
Games AI is very unique: perfect game rules, fully digital (for many games).
We have many super-human AIs, like AlphaZero.
Maths and coding come close, but the goal of fully autonomous and correct agents is likely infeasible, due to the impossibility results, e.g., Gödel’s incompleteness theorems and the halting problem. Even so, there are lots of academic and business opportunities and we may try our best.
For coding problems, at the programming language level, it has deep roots in logic, so that it may be verifiable. However, formal verification has not receive much attention from the AI community, and AI code still faces the issues: When AI Writes the World’s Software, Who Verifies It? In contrast, software engineering is a social-technical system, and it is even hard to define correctness: Correctness, confidence, and context: Framing software assurance in the AI age. Software engineering encounters not only technical, but also cognitive and intent debts.
Many science and engineering problems have physical and/or human components. It thus may be costly to collect data. Also, there is usually a simulation-to-reality gap.
For human-centric problems, “super-human” may not be well defined. Consider translation as an example:
- no objective objective; NLP metrics like BLEU score and perplexity are heuristic
- experts’ evaluation: likely the best data we can get, may be irreplaceable
- seemingly no way to collect super-human data, or how to judge it is superhuman or not
Games, maths and coding are special problems, with special, perfect environments. The achievements in such problems may not be straightforwardly generalizable / transferable to other problems.
6. A brief summary
+ Ground truth is a first principle.
+ Expect breakthroughs in maths and coding, likely in science and engineering.
+ Achievements in games, maths and coding may not be straightforwardly generalizable / transferable to other problems.
+ Experience from humans may be irreplaceable for human-in-the-loop problems. An (iterative) inverse RL approach, like RLHF, provides a principled approach to learn a reward function.
+ Study how to collect ground truth experience efficiently for problems with physical and/or human components.
+ Call for efficient (reinforcement) learning algorithms as always.
Acknowledgement
This is an update on Aug 1, 2026, adding some thoughts and a perspective about achievements for a recent Fields medal (related to world model, human-in-the-loop).