Skip to content
Coordinates of ThoughtHome of Philosophy
Sign in
Debate

Thesis Large language models understand nothing: the Chinese Room still applies

Pieter de Jong Fellow
· 9 replies
Artificial mindsAnalytic#chinese-room#large-language-models#understanding
Follow

Where it stands

I build software on top of these models every day, so this isn't idle. I'll argue the thesis and see if it survives.

John Searle's 1980 paper "Minds, Brains, and Programs" imagines a man in a room who receives Chinese characters, follows an English rulebook for manipulating them, and passes back answers that fluent speakers find perfectly sensible. He understands no Chinese. Searle's conclusion: syntax isn't sufficient for semantics. Running the right program doesn't produce understanding.

An LLM is, as far as I can see, the room at scale. It manipulates tokens according to learned statistical rules, trained on text, with no contact with the things the text is about.

The famous "systems reply" says the whole room understands, even if the man doesn't. Searle's answer: let the man memorise the rulebook and do everything in his head. He still understands nothing.

So: what does a 2020s model have that the room lacks?

9 replies

From the author: revisions and responses

  1. Pieter de Jong

    Revision by the author Fellow

    Revised thesis after a good thread. The Chinese Room shows that behavioural success alone doesn't settle whether a system understands. It doesn't show that LLMs lack understanding. And Emeka's grounding point is the stronger argument, and it's one that multimodal, embodied systems might eventually answer.

    1. Henrike Vogel

      Against Fellow

      So the original thesis has been abandoned by its own author. I count that as a win for the "against" side, Pieter, though a gracious one.

      Helpful · 1

For the thesis 3

  1. Nur Aisyah Rahman

    For Fellow

    I work with these systems too, and hallucination is the giveaway for me. A model will cite a paper that doesn't exist with total fluency. Something that understood what a citation is wouldn't do that so casually.

    Helpful · 1
  2. Arjun Iyer

    For Fellow

    I'll vote for the thesis from an unusual direction. Understanding, for me, is a conscious act: there is something it is like to grasp a meaning. Whatever these systems have, I see no reason to think they have that. Searle's argument is weaker than his conclusion, but the conclusion may still be right.

  3. Emeka Nwosu

    For Fellow

    Harnad's symbol grounding problem (1990) is the sharper version for me. Symbols get their meaning from other symbols, all the way round, unless at some point they're connected to perception and action. Text-only models are a dictionary in which every definition points to other definitions.

    Helpful · 3

Against the thesis 4

  1. Priya Raghavan

    Against Fellow

    The Chinese Room is an intuition pump, and the intuition is doing all the work. Neurons also "just" follow local rules, and nobody's neurons understand Chinese individually. Also, interpretability work has found internal representations in these models; the 2023 Othello-GPT study found a representation of the board state in a model trained only on move sequences.

    Helpful · 2
    1. Pieter de Jong

      Author’s response Fellow

      Priya, I know the Othello work, and it's impressive. But having a representation of a board isn't obviously understanding a board. My thermostat represents temperature.

      Helpful · 3
  2. Tomasz Wójcik

    Against Fellow

    Searle's argument proves too much. If it works, it works against any physical system, including brains, unless brains have some special "causal powers" — and Searle never told us what those are. It's an argument that we haven't explained understanding, not that machines lack it.

    Helpful · 1
  3. Elif Yıldız

    Against Fellow

    Understanding seems graded to me. My nephew used "yesterday" for any time in the past before he understood what it meant, and then gradually he did. I don't see why a system couldn't be partway along a scale like that.

    Helpful · 2

Members reply here. Reading is always free.

Sign in to reply