I Wrote an AI Textbook. How Long Until AI Can Write One Better?

The limits of AI writing

Criticism of AI-generated writing often focuses on creative, high-voice work. Such writing depends on a distinct point of view, human expression, and a visible process of thought conveyed through carefully chosen words. As large language models become more refined tools rather than conversational assistants, progress toward genuinely inspiring writing can appear to be moving in the opposite direction.

Non-fiction presents a different case. Producing filler and straightforward copy was one of the early practical uses of large language models. It seemed that the weaknesses in explanatory writing mainly reflected limited model intelligence or training problems, and that continued progress would eventually eliminate them. After several years of using models as writing assistants, however, improvement has been modest. That raises the question of what continues to hold them back.

The models' stagnation in long-form non-fiction should concern anyone who expects them to solve major, open-ended scientific problems autonomously in the near future. Current systems still struggle to organize and compellingly explain some of the most established science in their own fields. Mastering this ability seems like a prerequisite for independently solving broad scientific problems. Until then, AI-assisted science is likely to focus on low-hanging fruit and connections between distant fields rather than revolutionary insight.

This is a cautious assessment from someone optimistic about AI progress. It also comes as Anthropic has reported that Claude made progress on the Riemann Hypothesis. Scientific problems cover a vast range of knowledge, and current models do not appear to have as broad a grasp of that range as many people assume.

Organizing knowledge is a form of compression, and that compression is necessary for producing insight. Current LLMs often increase entropy in long-form non-fiction writing. It is difficult to see how that process could be layered indefinitely without human guidance.

There is still reason for optimism. Narrow advances in areas such as mathematics could translate into broader, more consistent scientific progress. LLMs are among the most powerful assistants scientists have used. Yet their performance on grounded, low-level writing tasks reveals a surprising lack of generalization.

Why has writing quality stagnated?

Much greater progress in non-fiction writing seemed likely. In 2024, it was reasonable to wonder whether publishing a non-fiction book in 2026 would look premature. Instead, some of the models most associated with writing ability are already relatively old, including OpenAI's GPT 4.5 and Moonshot's Kimi K2.

During roughly the same period, models advanced from adequate to superhuman on tasks such as coding and mathematics. Search and research abilities also improved from being largely ineffective to becoming useful. Progress on many skills has been steep, while strong writing appears largely separate from those gains. Writing is probably not being ignored, but it is difficult and lacks well-defined training data for targeted improvement.

There are obvious ways to improve results, including specialized environments such as Claude Code, better prompts, and training setups that encourage models to spend more inference tokens on an answer. These approaches may help, but they are unlikely to multiply the underlying ability. Writing well is difficult, and inference-time scaling has not yet unlocked the same gains for writing that it has produced in other intellectual tasks.

Current models are particularly weak at long-form technical writing. They may produce an accurate sentence, but an entire chapter often contains confusing wording, poor organization, and subtle conceptual mistakes. Models tend to introduce unnecessary cleverness, which can create additional errors. Larger models should reduce some of these problems by storing more world knowledge, but that does not necessarily mean they will become much better at using that knowledge coherently across a long document.

GPT models have long been effective at identifying typos and minor problems. A near-final PDF of a 200-300-page book was reviewed by GPT 5.5 Pro, which found numerous deep and unexpected minor errors throughout the manuscript.

Claude models have been more useful as editors. They tend to show more sensitivity to the task, better understand the writer's mental model, and offer more interesting suggestions for overcoming different forms of writer's block.

These examples share a pattern. Models can inspect individual units of content, such as a sentence, equation, or figure, and they can sometimes improve a specific section. They are much less reliable at revisiting those components and connecting them as the document grows. The result resembles an irreducible accumulation of errors. Mathematics and programming once faced similar problems, but reinforcement learning with verifiable rewards, or RLVR, has substantially reduced them in those domains.

How current models can help writers

A small number of technical explanation sentences in the textbook came from an AI model, well under 1 percent of the book. They were included because, after review by an expert, they seemed to express what readers needed. During editing, the models were especially valuable because the manuscript required close attention while competing with other demands on the author's time.

For example, editor questions were placed throughout a LaTeX file using a delimiter such as:

\\editor{}

Claude Code could navigate to each comment, print the surrounding context, and distinguish a simple typo from a more substantial issue. The response could then be written directly, or Claude could suggest alternatives before the text was revised. Some phrases from those suggestions ultimately appeared in the book.

This approach can become a slippery slope. The suggestions were accepted only during a second full-manuscript review, when the project felt emotionally complete but still required additional work. The experience also reinforced the value of keeping other writing entirely human. Writing with a strong personal voice can be rewarding because the process itself matters. Producing a standard reference work is less inherently enjoyable, and AI tools can become tempting when writing is treated mainly as a way to fill space rather than as a means of learning, feeling, and expressing ideas.

A similar distinction applies to scientific papers. AI models can help draft repetitive sections, such as related work or background material that the researcher already knows well. Using them for the abstract, introduction, experiments, or conclusion is less useful because those sections communicate the work's story and significance. They are also where researchers often discover what their work is really about.

AI tools created substantial value during the textbook project. They made equation writing easier, helped refactor the repository, supported language conversions, and handled many other repetitive tasks. One particularly demanding problem involved maintaining Markdown and LaTeX versions of the book in parallel. Readers reviewed the web version while the Manning editorial team worked from a forked copy. Synchronizing the two versions already took tens of hours, and without AI agents it could have taken roughly five times longer.

However, agents do not increase the pace of understanding itself. Intuition, taste, and instinct remain important, and using AI for non-fiction writing can reduce the opportunity to develop them. Readers who are not already experts may also be unable to identify the model's mistakes.

In this case, there was strong pressure to transfer specialized knowledge from the author's mind to the page. The textbook aimed to provide a single reference for post-training methods such as rejection sampling and character training, for which relatively little material was available online. Completing the book in a useful form was therefore valuable even if additional human effort might have improved it further.

The decision to use AI was also influenced by concerns that the book might become outdated before publication, both because of rapidly improving models and because the field itself was moving quickly. That concern proved misplaced. The finished book appears more durable now than it did at the beginning of the project in 2024, partly because AI models have not matched expectations for non-fiction writing.

The future of technical writing

AI models are effective tools for expressing knowledge in different formats. They can create background material and early drafts of creative filler, such as the first version of presentation slides. In that setting, the slides can serve mainly as talking points for an instructor, while the instructor provides the substance.

The early organizational stage of writing a non-fiction book or reference text is different. Developing the structure, presenting the central framework, and deciding how new knowledge should be arranged are closer to writing a high-voice essay. This is where insight is required, and LLMs remain far from replacing it.

More experts using AI to write small portions of books could help make more specialized knowledge available. The limitation is that current models may save only 10-20 percent of the effort. There is little indication that this contribution will become the majority of the work soon.

Social expectations create another challenge. Some people assume that LLMs will become the best personalized educators, making books and educational content unnecessary. High-quality educational work remains scarce, however. AI is effective at adapting existing material to a student's preferred format, but it is much less capable of creating authoritative content from scratch.

The current situation is a frustrating local minimum. AI may reduce the average effort invested in non-fiction writing while also enabling stronger expression for people who already have the knowledge. At the same time, fewer people may begin ambitious writing projects or persist through the difficult stages.

For the next 2-5 years, the best textbooks are likely to remain heavily shaped by human authors. The longer-term outlook is less certain, but this timeline is already longer than many expected given the amount of knowledge in current models and their ability to reproduce it fluently.

A broader lesson about model capabilities

Current models are strongest in two contexts:

1. Domains where answers can be verified reliably.
2. Tasks that provide substantial context and request a small, specific change, such as finding a bug, solving a narrowly defined mathematical problem, or reviewing existing text.

They are much less reliable when asked to generate open-ended prose. Long-form technical writing may improve before creative writing, but its current weakness is an important signal: models cannot yet express the full extent of their stored knowledge when the task is underspecified.

This limitation matters for efforts to build systems that act like "geniuses in a datacenter" and solve major scientific problems. A model may contain extensive information, but organizing that information into a coherent, accurate, and insightful explanation remains a fundamental challenge.

Part of the problem also lies in how people use these systems. When Claude Fable 5 is asked in Claude Code to write a poem about a goldfish, it can quickly produce an answer. When asked whether it first created and revised a detailed draft in a scratchpad, it indicated that it had not. Instead, it made a minimal plan in its reasoning tokens and then generated the poem autoregressively, without making extensive use of inference-time scaling.

The same pattern likely applies to much AI-assisted writing. Better results require detailed prompts, extensive work before producing an answer, and feedback from separate judge models. Given the genuine skills models already possess, there is a straightforward path to improving long-form output, but that path still depends on deliberate human direction.