When a system sold on the basis of having the ability to reason fails to do so with the reliability of a junior engineer, who would consider a rule to be boilerplate? Large language models are becoming pervasive in labs and product groups, where they are employed as flexible assistants to aid analysis, documentation, and planning. Their productions tend to sound worked solutions, with their steps organized and confident manner. The fact that such surface competence renders a certain category of failures more significant: the models are capable of generating convincing explanations and silently breaking simple rules, missing a major fact or reducing a multi-step plan to contravening fragments.
Recent work by Stanford, Caltech and Carleton College researchers has synthesized these breakdowns into common patterns and, more importantly, has viewed them as engineering failure modes, not as unique “gotcha” warning signs. In spite of these developments, there exist major reasoning errors even in the case of supposedly very simple situations. The categories in the paper include individual cognitive-style biases (anchoring, confirmation effects), social reasoning gaps (Theory of Mind and moral judgment), and brittle reasoning in natural language, in which “trivial” equivalences may fail. The same review mentions that the ensuing fragility exposes them to “jailbreaks and manipulation,” a lesson that the errors of reasoning are not merely scholarly, but they also turn into operational risk when models are operationalised in the workflows.
The real issue is that contemporary appraisals have tended to approve of the limited skill ranges. With the culture of benchmarking being pushed towards math and code, models were trained to appear strong where scoring is clean and automated, but where a broader reason was poorly measured.
That is one of the reasons why the community has shifted to more challenging general tests like BIG-Bench Extra Hard (BBEH) that substitutes saturated tests with more difficult ones meant to offset shortcuts. Even there, the level of performance is limited: the paper is reporting a 9.8% harmonic-mean accuracy on the best general-purpose model, 44.8% on the best reasoning-specialized model, showing a disproportionate ability in skills instead of a consistent progression towards high-quality general reasoning. The same benchmark analysis reports that gains concentrate in more formal domains such as counting and planning, whereas commonsense, humor, sarcasm, and causation are improved more, though not a great deal, the kind of “soft” reasoning engineers experience when there is ambiguity in the requirements and the context changes half way through.
Individual studies have also indicated that “more reasoning” may be counterproductive. In reasoning-induced instruction-following failures A NeurIPS 2025 poster tested over 20 models and discovered that explicit chain-of-thought prompting can make accuracy-based on constraint-based instruction benchmarks. Part of the drop the authors credit to attention shifting out of instruction-relevant tokens is that such strategies promote selective-reasoning and only use step-by-step thinking when it is beneficial to do so.
The reasoning failures also overlap with hallucination the possibility of the model to be both ungrounded and fluent. A more recent survey defines hallucination as text that is both fluent and syntactically correct but factually incorrect or do not have extrinsic support, and charts causes along the lifecycle between data and inference. It further categorizes detection under retrieval, uncertainty, embedding, learning and self-consistency techniques and points out that each method works well in different circumstances. In practical contexts, that drives teams to layered checks, including fact decomposition to possible factual scoring, uncertainty probing in the absence of token probability, and self-consistency tests that are seeking meaning-level instability across generations.
The throughline should be known to aerospace and mechanical engineers: resilience is not often created by assuming a component is “smart,” but by describing its points of failure and creating guardrails. The Stanford-Caltech-Carleton review puts the amendment in the direction of the same root-cause analysis, perennial benchmarks that keep hard cases in history, and the intentional introduction of failure to make known weaknesses become apparent. That is, the current models have the capability to speed up the drafting and exploration, but their reasoning is a subsystem, which needs to be validated, as opposed to being a replacement of it.