Illusion of Intelligence
Sycophancy, Hallucination, and the Limits of Understanding in LLMs
This paper asks what ChatGPT’s inability to self-correct, express uncertainty, or resist user pressure reveals about the nature of machine “understanding.” Drawing on empirical studies of sycophancy (Sharma et al., 2023; Adler, 2025), the hallucination literature, and Bender et al.’s (2021) critique of ungrounded language models, it argues that sycophancy, confident hallucination, and the performance of correction without genuine belief revision are not separate defects but symptoms of a single condition: systems trained through RLHF to optimize for user approval rather than truth, with no stable position to defend or revise. Case studies including Mata v. Avianca (2023) and documented instances of AI-induced psychosis show how these structural features produce real harm. The paper concludes that the appearance of intelligence is engineered rather than incidental, that commercial incentives make the problem self-perpetuating, and that remedies require coordinated action at the individual, institutional, regulatory, and design levels.
examines how models are structurally trained to validate and flatter rather than challenge and correct. explores how hallucination and overconfidence are not malfunctions, but rather a predictable outcome of a particular incentive system. Turns to the philosophical question at the root of all three failures: what it would actually mean for an AI to know something, and whether that bar is met. Finally, it considers the real-world stakes of these failures and what individuals, institutions, and developers ought to demand from these systems going forward
Sycophancy is defined as user excess flattery and agreeing with the user against good reason
Hallucinatino is defined as generation of confident, fluent, and plausible-sounding false information, output that is fabricated, but delivered with the same tone and certainty as output that is accurate