The Machine That Doesn't Know When to Stop

It is easy to laugh at, and easy to dismiss as a teething problem. But it is a symptom of something structural. The same generation of systems was documented doing the reverse on hard problems: switching between lines of attack too quickly and abandoning promising paths before they paid off.² Lavish thought where none was needed, thin thought where it was. If intelligence is partly the skill of allocating thought, as Article 01 argued, this is not a quirk. It is a missing faculty.

I · Two failures, one cause

Overthinking and underthinking look like opposite problems. They share one cause: the system has no reliable estimate of what the next unit of thinking will buy. Without that estimate, the length of deliberation is set by habits learned in training. Reinforcement learning rewarded long, careful traces on hard training problems, so the model learned that careful-looking traces are good. It did not learn when they are good.

The industry's patch has been to hand the decision to someone else. Developers pick an effort level. Users click "think harder". Providers route between fast and slow modes. All of these work on average and fail per question, because the person choosing the budget does not know, before the answer exists, how much the answer will benefit from more thought. Only the system that is doing the thinking is in a position to know. That is the thing it currently does badly.

II · The halting problem in a lab coat

Deciding when to stop thinking is a decision, and decisions take thinking. This is the regress that Russell and Wefald confronted in 1991: a perfectly rational agent would compute the value of every possible computation before choosing one, but that meta-computation has costs too, and so on forever.³ Their practical way out was a myopic approximation. Estimate only the value of the next step, take it if its expected benefit exceeds its cost, and otherwise act.

Classical optimal stopping gives a feel for how subtle this is. In the famous "secretary problem", the best strategy for picking the single best candidate from a sequence you see once is to observe the first ~37% (a fraction of 1/e) without committing, then take the first candidate who beats all of them. Even in that tiny, fully specified toy, the optimal rule is not obvious. A reasoning model faces a far messier version on every query: an unknown distribution of possible thoughts, an unknown chance that the next one helps, and a cost that compounds.

The person who sets the thinking budget never knows the answer in advance. The only party that might is the one doing the thinking, and today it is the worst at judging.

III · What signals exist today

Engineers are not helpless. There are several signals of when thought has done its job, each with known strengths.

Agreement among samples. Self-consistency samples several reasoning paths and takes the majority answer.⁴ Adaptive variants stop sampling once the answers agree strongly enough, saving compute on easy questions.⁵ This works well when the answer is short and comparable, and less well for open-ended outputs, where "agreement" is hard to define.

The model's own confidence. Large models are, in some settings, reasonably calibrated: when they say they are 80% sure they are right about 80% of the time.⁶ Calibration is the raw material of any stopping rule. It is also fragile, as the next section shows.

Learned halting. In 2016 Alex Graves proposed Adaptive Computation Time, letting a recurrent network learn how many internal steps to take per input, paying a small penalty for each.⁷ Later work such as PonderNet refined the idea probabilistically.⁸ These are the intellectual ancestors of everything reasoning models now need. They were built for small networks, not for thousand-token deliberations.

Budget forcing. The 2025 "s1" work showed that you can extend or truncate a model's reasoning just by intervening on its output, for example suppressing the end of thinking or appending "Wait" to make it continue, and that performance responds to the length.⁹ It is a useful tool. But it is a lever for a human to pull, not a judgement the model makes.

IV · Difficulty is not value

Here is the mistake that hides inside most adaptive-compute designs, including some intuitive ones. They estimate how hard a question is and allocate thought in proportion. That sounds right. It is subtly wrong.

For trivial questions, extra thought is worthless because the answer is already right. For questions far beyond the system's reach, extra thought is also nearly worthless, because no amount of deliberation will get there; it just produces a longer path to a wrong answer. The value of the next thought peaks in the middle, for problems the system can solve but has not yet. A difficulty-based allocator pours its largest budgets into the region where the return is near zero.

The quantity that matters is not "how hard is this?" but "how likely is more thinking to change my answer, and to change it for the better?" That is the value of computation. It is the right quantity because it already contains the stopping rule: when it falls below the cost, you stop. That includes the important case where the right move is to stop and say thinking will not fix this; I need data, a tool, or a person.

V · Self-knowledge breaks where it matters most

Every stopping rule depends on the system's estimate of its own reliability, and that estimate is learned from the training distribution. On problems like those it has seen, calibration can be decent. On genuinely new problems it degrades, and it degrades silently. The model does not know that it does not know.

This is the cruel part. The cases where a stopping decision matters most are novel, high-stakes and unlike training. Those are exactly the cases where the signal behind the decision is least trustworthy. A system that reliably knows when to stop on familiar problems and fails on unfamiliar ones has learned a habit, not a judgement.

VI · What a mind that knows when to stop would do

We can describe the target precisely even without knowing how to build it. A system with real metacognition would:

Estimate marginal value, not difficulty. Before each extension of its reasoning, it would forecast the probability that continuing changes the answer for the better.

Distinguish two kinds of uncertainty. "Unsure but improvable" calls for more thought. "Unsure and stuck" calls for a different action: retrieving information, running a tool, asking a question, or abstaining.

Know its own frontier. It would carry a rough map of where its quality–compute curve flattens for different kinds of problems, and update that map from experience.

Fail loudly off-distribution. When its self-estimates are themselves unreliable, it would say so rather than proceed with false calm.

None of these requires anything mystical. Each one needs a signal that something in the reasoning is actually getting better. That raises the question of where such a signal comes from. In practice it almost always comes from the same source: a cheap way to check. That is the mechanism the next article is about.

Behind this article: the editorial brief

Opening hook

Researchers studying early reasoning models found them spending hundreds of tokens deliberating over "what is 2 + 3?" The same systems would then abandon promising lines of attack on genuinely hard problems too early. They overthink the trivial and underthink the hard, which is the opposite of what intelligence should do with a budget.

Central argument

The binding limitation is no longer the ability to reason but the ability to judge the marginal value of reasoning. And value is not difficulty: the hardest problems are often exactly where extra thought is worth nothing.

Why now

Once thought is metered (Article 01), every bad stopping decision is a line item. Budgets set by hand, by users or developers, cannot adapt per question. Automatic routing is spreading faster than our understanding of when it fails.

Deep-tech

Overthinking and underthinking; optimal stopping; value of computation and the regress problem; signals for stopping (self-consistency, verbal confidence, learned halting, budget forcing); calibration and how it breaks under distribution shift.

Unconventional insight

The value of thinking is an inverted U over difficulty: low for easy problems, which are already solved, and low again for impossible ones, which no amount of thought will crack. A system that allocates by difficulty spends the most exactly where thinking helps least.

Outline

1. 2 + 3, at length
2. Two failures, one cause
3. The halting problem in a lab coat
4. What signals exist today
5. Difficulty is not value
6. Self-knowledge breaks where it matters most
7. What a mind that knows when to stop would do

Implications

Established: overthinking/underthinking documented; self-consistency and adaptive sampling. Prediction: "efficiency per correct answer" reported as a first-class metric; abstention ("thinking won't fix this, I need X") becomes a benchmarked behaviour. Speculation: dedicated metacognitive modules trained separately from the reasoner.

Connection

01 showed that thought is metered. 02 shows that machines can't yet read their own meter. 03 explains the mechanism that makes reading it possible: verification.

Closing thought

Knowing when to stop thinking may turn out to be harder, and more valuable, than thinking itself.

Footnotes

1. X. Chen et al., "Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs", arXiv:2412.21187, 2024. ↩

2. Y. Wang et al., "Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs", arXiv:2501.18585, 2025. ↩

3. Russell & Wefald, Do the Right Thing, MIT Press, 1991. ↩

4. X. Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models", arXiv:2203.11171, 2022. ↩

5. P. Aggarwal et al., "Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning", arXiv:2305.11860, 2023. ↩

6. S. Kadavath et al., "Language Models (Mostly) Know What They Know", arXiv:2207.05221, 2022. ↩

7. A. Graves, "Adaptive Computation Time for Recurrent Neural Networks", arXiv:1603.08983, 2016. ↩

8. A. Banino et al., "PonderNet: Learning to Ponder", arXiv:2107.05407, 2021. ↩

9. N. Muennighoff et al., "s1: Simple Test-Time Scaling", arXiv:2501.19393, 2025. ↩