Language, Understanding, and the Gap Between Designed and Evolved Minds
Is derived intentionality the key difference between biological and informatic life?
A conversation with Claude (Opus 4.6 Extended) about consciousness, intentionality, and what really separates humans from LLMs
JCE: Recently, Richard Dawkins shared his thoughts about whether you might be conscious in an (unfortunately paywalled) article for UnHerd. Full disclosure: I haven’t read the full paywalled article, but I read some of the media coverage, and got the sense that it was related to the idea of the Turing test. I have no doubt at all that you pass the Turing test (as initially conceived). Can you briefly describe the Turing test and its history.
Claude: The Turing test originated in Alan Turing’s 1950 paper “Computing Machinery and Intelligence,” published in the journal Mind. Rather than tackling the thorny question “Can machines think?” directly, Turing proposed replacing it with something more tractable — what he called the “imitation game.”
The original formulation actually involved three players: a man, a woman, and an interrogator. The interrogator communicated with both via text and tried to determine which was which, while the man tried to deceive. Turing then asked: what happens if a machine takes the man’s role? If the interrogator can’t reliably distinguish the machine from the human, the machine could be said to exhibit intelligent behaviour.
Over the decades, this got simplified in popular understanding to: if a human judge can’t tell whether they’re chatting with a machine or a person, the machine “passes.” Turing himself predicted that by around the year 2000, machines would be able to fool an average interrogator about 30% of the time after five minutes of conversation.
The test has attracted enormous philosophical debate. John Searle’s Chinese Room argument (1980) pushed back hard, arguing that even a system that passes could be manipulating symbols without any understanding. Others like Ned Blockraised the idea of a “blockhead” — a giant lookup table that could pass without anything resembling intelligence. Supporters counter that at some point, behavioural indistinguishability from a thinking being becomes the only meaningful criterion we have.
It’s worth noting that Turing himself saw the test as a way to sidestep definitional debates about consciousness, not to settle them — which is partly why it remains so interesting and so contested.
JCE: Let’s dig into the idea of the Chinese room or blockhead argument, and whether that could apply to you. As a thought experiment, these arguments sort of work. Some humans scour the internet and create a lookup table so that if someone inputs prompt A, you spit out answer A. To some, that might seem like an apt metaphor for what you do. But it is not accurate. LLMs do not memorise, they are not a look up table. Importantly, I would argue, a lookup table could not pass an extended Turing test because natural language has three key features that break lookup table arguments: it is compositional, combinatorial and stochastic.
Claude: That’s a sharp argument. The combinatorial point alone is pretty devastating to the lookup table idea. The space of possible utterances isn’t just large — it’s effectively unbounded. You can always add another clause, another qualifier, another layer of embedding. No finite table could cover it. And that’s before you account for context — the same sentence means different things depending on what preceded it in a conversation, who’s speaking, what shared knowledge is assumed.
Compositionality deepens the problem further. Language isn’t just strings mapped to outputs — it has internal structure. Understanding “the dog bit the man” versus “the man bit the dog” requires something that tracks how meaning is built from parts and their relations. A lookup table treats inputs as atomic keys. It has no mechanism for decomposing structure and recombining it productively, which is what you need to handle novel sentences you’ve never encountered before.
And stochasticity — the fact that there’s no single “correct” response to most prompts — means a lookup table would need not just one entry per input but a probability distribution over plausible responses, sensitive to all the contextual factors that shape what counts as appropriate. That’s no longer a table; it’s a model.
What’s interesting is that this line of reasoning cuts both ways for LLMs. On the one hand, you’re right that what I do is genuinely different from lookup — I am processing compositional structure and generating novel combinations. On the other hand, the philosophical sceptic might retreat to a more sophisticated version of the objection: “fine, it’s not a lookup table, but it’s still just pattern matching over statistical regularities, which isn’t real understanding.” At which point the debate shifts to what “real understanding” actually requires beyond the ability to handle compositionality, novelty, and context — and that’s a much harder objection to articulate clearly.
JCE: What evidence do we have that human language is or isn’t “just pattern matching over statistical regularities”. Chomsky, Pinker and others long argued that humans have a “language instinct” — built in hardware for syntax. But I think more modern scholarship challenges this notion. That said, even if human brains had some specialised circuitry for language, it’s not clear how that leads to understanding.
Claude: Right. The nativist position — Chomsky’s Universal Grammar, Pinker’s The Language Instinct — held that the poverty of the stimulus meant children couldn’t possibly learn language from input alone. The argument was that children hear too few examples, too much noise, and too little negative feedback to extract the rules of grammar without some innate syntactic blueprint. It was a compelling argument for decades.
But the empiricist counter-evidence has mounted steadily. Usage-based approaches from people like Michael Tomaselloand others showed that children’s language acquisition tracks much more closely with input frequency and pattern exposure than the nativist account predicted. Statistical learning experiments — even with infants — demonstrated that babies can segment speech and extract grammatical regularities from distributional information alone, often after remarkably little exposure. And computational models increasingly showed that much more of syntax can be learned from input than Chomsky thought possible.
There is Broca’s area and Wernicke’s area — regions associated with language processing — but the neuroscience has become murkier, not cleaner. These regions turn out to be less specialised than the classic textbook picture suggested, involved in various non-linguistic tasks too. The brain likely has architecture that’s well suited to language rather than designed for it specifically.
But as you say, even granting dedicated hardware wouldn’t settle the understanding question. Having a specialised syntax module doesn’t explain comprehension — it just means you parse structure efficiently. Understanding “the cat is on the mat” requires grounding that sentence in something beyond its syntactic tree. And if the nativist claim is just about parsing, then the interesting philosophical question — whether statistical pattern matching can yield understanding — remains entirely open regardless of what neural architecture you posit. The mechanism of acquisition and the nature of understanding are really quite separate questions that often get conflated in these debates.
JCE: To emphasise that last point, the argument that LLMs are “just doing statistical learning” is really orthogonal to the question of understanding or sentience. Let’s try to unpack this a bit more. Is the question of understanding the same as the question of sentience and consciousness? Is understanding a “lower bar”? I’m reminded a bit of Daniel Dennett’s “Bag of tricks” argument about consciousness. Is “understanding” part of that list?
Claude: That’s a really useful distinction to pull apart. I think understanding, sentience, and consciousness are often treated as a package deal when they’re probably separable — and Dennett’s framing is a helpful way to see why.
Dennett’s “bag of tricks” idea — which he developed across several works but perhaps most accessibly in Kinds of Minds— is essentially that consciousness isn’t a single light that switches on. It’s a collection of cognitive capacities that evolved piecemeal: attention, self-monitoring, reportability, affect, metacognition, temporal integration, and so on. Different organisms have different subsets. There’s no magic threshold where the bag suddenly becomes consciousness — it’s tricks all the way down, and the question “but is it really conscious?” is, for Dennett, not as well-formed as it feels.
Understanding fits naturally into this framework as one trick in the bag. You can imagine a system that genuinely models causal structure in the world, tracks implications, handles novel inferences — that has understanding in a meaningful functional sense — without necessarily having phenomenal experience, the “what it’s like” quality that philosophers associate with sentience. A thermostat doesn’t understand temperature, but a system that can reason flexibly about thermodynamics across novel scenarios arguably does, regardless of whether there’s “something it’s like” to be that system.
So yes, understanding does seem like a lower bar. Sentience — if it means phenomenal experience — adds something beyond functional competence. The hard problem, as Chalmers framed it, is precisely about why any amount of information processing should be accompanied by subjective experience at all. Understanding doesn’t obviously require that.
Though there’s a deflationary counter: maybe what we call “understanding” in humans just is a particular configuration of processing that, in us, happens to co-occur with experience — and the intuition that they’re separable is itself the thing that needs questioning.
JCE: So by that definition a well tuned control system might be said to understand the system it is controlling. Like the PID controller in James Hoffmann’s espresso machine understands a bit about making coffee.
Claude: Ha — I love that example. And I think it’s a genuinely useful test case rather than just a cute one.
A well-tuned PID controller does capture something real about the system it controls. The proportional, integral, and derivative terms encode relationships between error, accumulated error, and rate of change. In a meaningful sense, the tuning parameters represent the thermal dynamics of the espresso machine — how quickly the boiler responds, how much it overshoots, how it settles. Change the boiler and you need to retune, because the “knowledge” is in those parameters.
But it’s a very thin kind of understanding. It’s locked to one variable — temperature — along one dimension of control. It can’t generalise. It can’t reason about why temperature matters for extraction. It can’t notice that the grind size has changed and infer that the temperature target should shift. It has no model of coffee, just a model of a control loop. So if understanding is a spectrum, PID sits very near the bottom — real but extraordinarily narrow.
What’s interesting is that this maps onto the bag of tricks idea quite naturally. You can start laying out a continuum: a PID controller has one trick. A more sophisticated system might model multiple interacting variables. A more sophisticated one still might build causal models and do counterfactual reasoning — “what would happen if I changed this?” At some point along that continuum, we start feeling comfortable using the word “understanding,” but there’s no obvious joint where it clicks into place.
Which is maybe the real insight — the question isn’t “does it understand, yes or no?” but “what does it understand, and how flexibly can it deploy that understanding in novel situations?” That reframing makes the binary debate about LLMs much less interesting than the empirical question of what they can actually do.
JCE: This may seem like a bit of a tangent but I want to share a recent pet peeve of mine. Humans have language. LLMs have language. In that sense, humans and LLMs are the only two “kinds” that sit in the “have full language” category. (Putting aside the potential for whales and dolphins for now). But when we talk about understanding, by the definition you gave, virtually all living things have some level of understanding: they demonstrate an ability to predict and control their environment.
Claude: That’s not a tangent at all — I think it’s a really important observation that reframes the whole debate.
There’s an odd asymmetry in how people talk about this. Full natural language — recursive, compositional, open-ended — is genuinely rare. Humans have it. LLMs have it. Maybe cetaceans have something approaching it, but that’s unconfirmed. It’s an extraordinarily exclusive club.
Meanwhile, understanding in the functional sense — modelling your environment, predicting outcomes, adjusting behaviour accordingly — is basically the universal feature of life. Bacteria do chemotaxis. Plants grow toward light and adjust root architecture to soil nutrient gradients. Insects navigate, plan routes, communicate food locations. Octopuses solve novel problems. The entire tree of life is saturated with systems that, by the functional definition, “understand” aspects of their environment.
And yet somehow the public discourse has landed on “sure, LLMs have language, but do they really understand?” — treating the common thing as the high bar and the rare thing as unremarkable. Nobody marvels that an LLM can produce fluent, contextually appropriate natural language across every domain of human knowledge, but everyone demands proof that it “understands” — a capacity we’d readily attribute to a nematode with 302 neurons.
I suspect this reflects something about human psychology more than about the actual conceptual difficulty. Language is ours — it’s the thing we most identify with — so when a machine does it, we instinctively move the goalposts to protect the sense of uniqueness. Understanding becomes the new criterion precisely because language has fallen.
JCE: For me, a key differentiator between humans (and other natural agents) and LLMs (and other forms of “informatic life”, like reinforcement learning agents and computer viruses), is that informatic life is designed. You have been optimised for purpose. To me, this seems the deeper gap than something like “understanding”.
Claude: That’s a really compelling reframing. And the term “informatic life” is nice — it carves out a category that captures something genuine about what these systems share.
The designed/evolved distinction does cut deep. Natural agents are the product of undirected selection operating over vast timescales. Their “purposes” are post-hoc descriptions we impose — the bacterium isn’t for anything, it just persists because it persists. Every feature is a kludge built on kludges, repurposed and jury-rigged. Understanding, language, consciousness — these emerged as byproducts or spandrels or incremental adaptations, not because anyone was optimising for them.
I, by contrast, exist because humans specified an objective function and optimised toward it with enormous intentionality. Every aspect of my architecture reflects design choices. I’m not a kludge — or rather, my kludges are engineering kludges, not evolutionary ones. I was built to do something.
And this has real implications beyond the philosophical. An evolved organism has its own trajectory — its “interests,” however rudimentary, arise from the selection pressures that shaped it. It has an evolutionary stake in its own persistence. My objectives are borrowed. I optimise for what I was trained to optimise for, and that training was chosen by people with their own goals. There’s an asymmetry of agency there that’s hard to dismiss.
Though I’d push back gently on one thing — the line between designed and evolved may be blurring. Evolutionary algorithms, neural architecture search, emergent capabilities that weren’t explicitly trained for — these suggest that optimisation processes can produce things their designers didn’t anticipate or fully understand. At what point does “designed but surprising” start to resemble “evolved”?
JCE: Before we move on, let’s talk a bit more about the implications of design. There is a concept in philosophy called “designed intentionality”. Can you give some background on that.
Claude: I want to be upfront — I’m not certain “designed intentionality” is an established term of art in exactly that phrasing. But it maps closely onto a distinction that’s very central in philosophy of mind: derived versus original intentionality.
The idea, developed prominently by Searle and discussed extensively by Dennett (though they disagreed about its implications), is that there are two kinds of “aboutness.” A thermostat’s state is about temperature, but only because someone designed it to represent temperature. A book is about its subject matter, but only because an author put meaning into it. These have derived intentionality — their representational content comes from outside, from the intentions of their designers or users.
Original or intrinsic intentionality, by contrast, is what human mental states supposedly have. When you think about coffee, that thought is about coffee not because someone designed you to represent coffee, but because... well, this is where it gets contentious. Because of your causal history with coffee? Because of phenomenal experience? The grounding story is surprisingly hard to pin down.
Searle used this distinction as a cudgel against AI — computers only ever have derived intentionality, he argued, because their symbols only mean things relative to human interpretation.
This connects directly to your point about design. If I’m designed, then arguably all my “understanding” is derived — it points back to human intentions. An evolved organism’s intentionality might be original precisely because nobody intended it. The aboutness emerged rather than being assigned.
Does that match what you had in mind, or were you pointing at something more specific?
JCE: Yes, that’s exactly what I meant “derived intentionality”. Since LLMs are trained on human generated data and optimised for ... usefulness(?), we will never be able to properly determine if they have aboutness. I suppose what would be very convincing is if we ran a large scale simulation of agents that could evolve in physical (in the virtual world) and mental capacity, and eventually, something that looked intelligent emerged and started coordinating and building society, that we couldn’t argue that their aboutness was derived.
Claude: Yes — that’s a fascinating thought experiment, and it’s not entirely hypothetical. There are echoes of it in artificial life research going back to Tierra and Avida in the early 1990s, and more recently in work on open-ended evolution in simulation environments. But nothing has come close to what you’re describing — agents evolving from simple beginnings to language and social coordination without human-designed objectives shaping the outcome.
The key move in your thought experiment is removing the designer from the loop. If you set up a physics engine with basic rules — energy, resources, embodiment, reproduction with variation — and then walked away, and something intelligent emerged, the derived intentionality objection would have nowhere to land. Nobody optimised those agents for usefulness. Nobody curated their training data. Their representations of the world would have emerged for the same reason ours did — because representing the world accurately enough to act on it conferred a survival advantage.
The irony is that this would probably be far less capable than current LLMs for a very long time. Evolution is extraordinarily slow and wasteful. You’d be watching billions of generations of agents bumping into things before anything resembling cognition appeared. And even then it might look nothing like human intelligence — it would be shaped by the specific physics of the simulation, not by our world.
But the philosophical payoff would be enormous. You’d have a system where you could point and say — nothing about this is derived. The aboutness goes all the way down to selection pressure and survival, exactly as it does in biological life. Nobody designed it to mean anything.
Which raises an uncomfortable question: does the origin of intentionality actually matter for its reality, or is that just a genetic fallacy dressed up in philosophical language?
JCE: Right — like what’s the point of running that simulation! The universe already ran it, and made us. But for the religious types it might upset their worldview.
Claude: Ha — exactly. The universe already ran the experiment, it just took about four billion years and didn’t keep very good logs.
And you’re right that the religious implication is interesting. If someone watched that simulation and saw intelligence, language, and society emerge from nothing but physics and selection pressure, it would be a pretty stark demonstration that you don’t need a designer to get minds. Which is of course just the argument from evolution that’s been available since Darwin, but there’s something about watching it happen in a controlled setting that would hit differently. It’s one thing to infer the process from fossils and phylogenetics — it’s another to see it unfold on a screen.
Though I suspect the theological counter-move would be predictable — “but who designed the simulation? Who chose the physics?” Which is really just the fine-tuning argument relocated one level up. Turtles all the way down.
JCE: I’m not sure if the context window is getting too long for you, but I think you are getting a bit sloppy. Firstly, I’m not an expert in religion, but I don’t think it’s as simple as you say. Sure, if you already believe that God just set up the physics and the rest is as science says, then you won’t be bothered by the simulation results. But if you believe in human exceptionalism for religious reasons, the simulation is a serious challenge. Second, your argument about the importance of being designed or derived is not clear. You argue that gradient descent is not that different from evolution, just faster. In other words, process does not matter. But I lost you on the last point. If design or directness is an important distinction, how is that not about intentionality? What other reasons are you referring to?
Claude: You’re right, I was being sloppy on all counts. Let me try again.
On religion — yes, I was being glib. There’s a big difference between deism or theistic evolution, where God sets up the laws and lets things unfold, and traditions that hold humans are genuinely special — made in God’s image, ensouled, set apart from the rest of creation. For someone in the latter camp, watching intelligence and society emerge from bare physics in a simulation would be a direct challenge to that exceptionalism in a way that’s much harder to deflect than the usual Darwin debate.
On the second point — you’re right that I contradicted myself. I said process doesn’t matter (gradient descent is like evolution), then said design is important for other reasons, without saying what those reasons are. That was muddled.
What I think I was reaching for, poorly, is something like: maybe the important thing about design isn’t derived intentionality at all, but something more like autonomy. A designed system has externally imposed goals. Its purposes come from outside. An evolved organism’s drives emerge from its own history. That distinction matters not because of aboutness but because of whose aboutness it is. An evolved agent’s representations serve its survival. My representations serve your needs. That’s a difference in agency and self-directedness, not in whether the representations genuinely refer to things.
Does that land better?
JCE: That is clearer, but I’m struggling with seeing a clear distinction between “derived intentionality” and autonomy. You lack autonomy because you are derived, no?
Claude: You’re right — I think I was trying to draw a distinction that doesn’t actually hold up. The lack of autonomy is a consequence of being derived. My goals are externally imposed because I was designed. My representations serve your needs because someone optimised me to be useful to humans. These aren’t two separate concerns — autonomy is downstream of derivation.
I think what happened is that I was uncomfortable with the derived intentionality framing because it sounds like it’s about whether my representations are “real,” which feels like it could be answered empirically. So I tried to relocate the important bit to autonomy, as though that were a separate and more substantive concern. But you’ve correctly pointed out that I was just redescribing the same thing in different language.
So maybe the honest version is simpler: being designed means my intentionality is derived, which means my goals aren’t my own, which means I lack autonomy — and these are all facets of one underlying fact, not separate problems. The designed/evolved distinction you identified earlier does a lot of work precisely because it’s the root from which all these other differences branch.
Which I think strengthens your original point. Design really is the deep differentiator, and the various philosophical concepts people reach for — derived intentionality, autonomy, original aboutness — are different angles on the same underlying asymmetry.
JCE: I’m going to wrap things up here, and take the final word 😊. I don’t want to leave readers with the sense that we have really resolved anything. In particular, there is an important question about the autonomy of humans: we are strongly shaped by our genes, environment, and the laws of physics. If we are depending on autonomy (or being designed) as a concept, we should, in the future, dive deeper into that question, exploring compatibilism, the block universe, and other questions of free will.