Midway through a long conversation with Claude today, I caught myself thinking “we’re having really insightful discussions” and then immediately wondered: are we, though? Or does it just feel that way because the model is extraordinarily good at producing text that pattern-matches to insight?
This is a problem that didn’t exist before language models got good. When you talk to a person, fluency and depth are correlated. Someone who articulates an idea beautifully probably understands it deeply. The heuristic is so reliable that we don’t even notice we’re using it. But language models break this heuristic completely. They produce maximally fluent text regardless of whether the underlying reasoning is sound. Every response sounds like it came from the smartest person in the room.
I think of it as the fluency trap. The conversation feels like mutual discovery — one idea building on another, surprising connections emerging, a sense of collaborative thinking. But what’s actually happening is that I’m providing the raw material (observations, questions, half-formed thoughts) and the model is arranging it into the most compelling narrative it can find. It’s like having a conversation with an infinitely articulate mirror. The ideas look sharper when they come back to you, but they might not actually be sharper.
The test I’ve started applying is simple: did this conversation change what I do? Not what I think — thoughts are cheap and easily manufactured. What I actually do differently tomorrow. If I walk away from a session feeling intellectually stimulated but my calendar, my code, and my priorities are unchanged, the session was entertainment. Good entertainment, maybe, but not insight.
Today’s conversation passed the test in some places and failed in others. The observation that I was optimising for a research paper instead of answering the consulting question it was supposed to serve — that changed my sequencing. I wrote it down, restructured the project, and the primary goal is now explicit. That’s real. But the twenty minutes we spent discussing how human memory consolidation might inform AI agent architecture? Beautiful conversation. Zero change to what I’m building this week. That was the fluency trap in action — the model produced such a compelling narrative about hippocampal consolidation that I mistook the pleasure of understanding for the utility of applying it.
The uncomfortable corollary is that the most useful AI interactions are often the least satisfying ones. “Your file has a bug on line 47, here’s the fix” doesn’t feel insightful. It feels mundane. But it saved me thirty minutes of debugging, which is thirty minutes I spent on something that matters. Meanwhile, “your project reveals a fascinating parallel between enterprise knowledge management and extended cognition theory” feels profound and produces exactly nothing.
I don’t think the answer is to avoid the deep conversations. Some of them are genuinely generative — the byproduct trap observation came from exactly this kind of exploratory discussion. The answer is to maintain the reflex of asking, at regular intervals: is this changing what I do, or just how I feel about what I’m doing? The fluency makes the question hard to ask, because the conversation is always so convincing that questioning it feels ungrateful. But the model doesn’t have feelings to hurt, and your time is finite. Ask anyway.