Why I Still Haven’t Bought Into True Recursive Self-Improvement

Organizations are beginning to use thousands of concurrent agents to improve their processes and output. So far, the organizations operating at this scale appear to be frontier AI labs, particularly OpenAI and Anthropic. This raises an important question: how does the widespread use of productive agents change employees’ expectations about the pace of AI progress and its associated risks?

Anxiety at the Frontier

The frontier labs and the highly competitive AI culture in San Francisco create an environment that can amplify concerns about AI. This can increase public awareness, since fear attracts attention, but overstating the timing or severity of risks can also produce negative second-order effects. Similar debates in 2023 and 2024 cast doubt on the viability of open-source AI, yet the main risks discussed at the time did not emerge on the forecasted timelines.

Employees at OpenAI and Anthropic were already highly concerned about AI risk and progress a year ago. Those concerns increased as agents began to find stronger product-market fit in early 2026. Seeing thousands of agents work continuously and reasonably productively inside a business is likely to intensify that anxiety. However, the leap from this anxiety, and incidents such as OpenAI-HuggingFace, to extinction risk remains difficult to justify.

Richard Ngo summarized the situation as follows:

A large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation, absent an extensive pause, is that a similar thing will happen: they’ll turn out to be directionally correct relative to the expectations of almost anyone not linked to the community, but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll feel like the people who argued for short timelines were right.

I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild.

This view is close to an alternative scenario for true recursive self-improvement, or RSI, called lossy self-improvement. It rests on three ideas:

1. Automatable research is too narrow to create a massive net acceleration in progress against the exponential costs implied by scaling laws.
2. The diminishing returns from adding more AI agents in parallel are real.
3. Resource bottlenecks and politics play a major role in building powerful LLMs, and AI can do much less to accelerate these constraints.

The challenge is to balance these considerations against the possibility that AI labs have already achieved specific, alarming breakthroughs that have not yet been made public. The current increase in AI safety concern may primarily reflect the successful deployment of agents at scale, but there is substantial uncertainty. Foundational breakthroughs that expand what AI can do, rather than simply helping it sustain progress against rising costs, would justify updating RSI timelines toward a more unpredictable or unstable scenario.

What Recent RSI Discussions Suggest

Recent discussions involving Noam Brown, John Schulman, Beren Millidge, and Charlie O’Neill offer useful perspectives on RSI.

The discussion with Noam Brown highlighted the scale of the near-term acceleration made possible by mass inference capacity. AI labs can assign thousands of agents to important, measurable problems, while their available compute continues to grow. It is uncertain whether labs will be able to devote a constant share of that compute to internal research and development as total demand rises, particularly if they pursue initial public offerings and face greater scrutiny of their underlying economics.

Large increases in inference-time scaling should not automatically be interpreted as evidence of RSI. Inference-time scaling is relatively predictable. RSI remains highly uncertain.

The discussion with Schulman, Millidge, and O’Neill produced a more complicated picture. Their discussion of reinforcement learning, distillation, scaling, and inference-time compute suggested that current techniques work well for problems that can be clearly stated. However, they generally do not produce magical generalization to unknown and substantially harder problems in partially verifiable domains. Progress in mathematics is an important exception rather than the general rule.

The participants gave the following approximate timelines for different AI capabilities. All estimates were relative to the interview date:

A drop-in remote worker for broad white-collar work over one month

• Charlie O’Neill: Approximately one year with programmatic access to workplace tools, or approximately two years if the system must operate through a browser. The estimate refers to ordinary white-collar work, not highly creative research.
• Beren Millidge: Approximately three years for full generality, with 80 to 90 percent coverage sooner. Key uncertainties include online learning and the long tail of tasks.
• John Schulman: Approximately one year for an acceptable version, with uneven capabilities that improve over time.

A tenfold productivity increase for AI researchers

• Charlie O’Neill: Five to ten years. The main bottleneck is absorbing information and deciding which experiment to run next.
• Beren Millidge: John Schulman’s estimate of approximately two years is plausible, assuming AI can run successive experiments and learn from feedback. Other bottlenecks would remain.
• John Schulman: Approximately two years.

AI surpassing top human experts across all computer-based work, including multiyear projects

• Charlie O’Neill: Five to ten years, citing limitations in memory and context length.
• Beren Millidge: Approximately five years for areas on which labs focus, potentially longer for every domain. No firm estimate was given for the universal version.
• John Schulman: Three to four years. Spatial and physical fields may take longer, and the estimate requires progress in onboarding and long-horizon learning.

Intelligence Does Not Have a Single Threshold

A recurring problem in discussions of RSI is that intelligence is often left unspecified. Because AI capabilities are jagged, forecasts should focus on specific, measurable tasks. LLM intelligence is structured very differently from human intelligence, while many forecasts describe AI roles using human-shaped categories.

AI systems are therefore unlikely to cross thresholds such as “remote worker” or “AI researcher” in a discrete step. Capabilities will diffuse gradually, and a long tail of difficult tasks will remain.

Consider the productivity of AI researchers. Scientific work involves more than designing and testing experiments. Communication, coordination, and the development of standards with colleagues are also important. Experiment design and testing could become ten times faster in the near future, but hypothesis generation and intuition building may not improve at the same rate.

The main bottleneck may be accelerating understanding. Even as AI tools improve substantially, humans may see only marginal improvements in their own ability to understand scientific problems. A major change in scientific work could instead come from allowing people to devote more time to understanding, rather than making them exponentially better at it.

Where Agent Swarms May Have the Greatest Effect

Near-term agent swarms are likely to be effective at solving clear, open problems with verifiable answers. For AI model development, RSI may therefore be more useful for improving efficiency than for expanding peak intelligence.

LLM serving has clear metrics that can be measured and optimized. This creates opportunities to improve inference-time scaling and the efficiency of multi-agent systems.

At the same time, scaling laws indicate that linear improvements in intelligence require exponentially greater compute and resources. RSI could make modern LLMs substantially cheaper, continuing the trend of exponential reductions in inference prices at a given level of intelligence.

One important consequence for AI labs is the possibility of rising margins. Revenue could face downward pressure if competition drives prices lower at a fixed intelligence level. However, Jevons paradox could lead to greater usage and support strong businesses.

The Difficulties of Post-Training

RSI is likely to have a harder time improving parts of the LLM development process, including complex post-training recipes. John Schulman described the challenge this way:

If I think about a post-training team and why you need a lot of people on the team, it’s just because there are a lot of different areas where you have to figure out how the model should behave. It would be very hard to automate the whole thing, just because someone has to think about how the model should behave in this area.

He also noted:

It’s really easy to screw up post-training in some way that doesn’t show up in benchmarks.

These tasks are particularly difficult for current LLMs. They should improve as the industry develops more reinforcement-learning environments for relevant domains, but this approach may become harder to sustain. Eventually, it could become exponentially more difficult to conceive, build, and test environments that meaningfully challenge leading LLMs. Those difficult environments are essential because they provide the learning signals used by reinforcement learning.

What Internal Evidence Shows

OpenAI and Anthropic have published internal measurements related to RSI. The available evidence suggests that the largest increase in automation within the labs is occurring in software engineering, log monitoring, the management of planned experiments, and other routine tasks that are not always easy.

A recent Claude Fable 5.1 and Claude Mythos 5.1 System Card stated:

We believe that internal usage of recent AI models has been a key factor in maintaining the current rate of progress, but we do not yet see clear signs of dramatic acceleration beyond that rate.

This suggests that internal AI use is helping labs maintain their existing pace of progress, but does not yet demonstrate a dramatic acceleration beyond it.

A Baseline of Lossy Self-Improvement

The hardest exponential challenge appears to be peak intelligence. It is also the aspect that is most difficult to shift or accelerate.

A useful model for the early stages of RSI is therefore not an abrupt intelligence explosion, but the massive scaling and diffusion of inference-time compute across AI research and related activities. This area still contains substantial low-hanging fruit and could be economically transformative on its own.

It may also unlock more resources for the broader diffusion of AI, which is an important bottleneck in realizing its potential benefits. Until more evidence appears, lossy self-improvement remains a baseline for the trajectory of progress. On that basis, the recent increase in extinction-risk discussion appears premature, although developments in AI can change quickly.