Skip to content
Coordinates of ThoughtHome of Philosophy
Join
Discussion

Does a language model mean anything by what it writes?

Nur Aisyah Rahman Fellow
· 5 replies · 236 views
Artificial mindsAnalyticIndianMimamsaDaniel DennettJohn SearleKumārila BhaṭṭaPaul Grice#chinese-room#derived-intentionality#language-models#meaning
Follow

At work in Kuala Lumpur I evaluate text-generating systems for fairness. Last month a colleague said, half joking, that one of our models "meant" something offensive. Everyone laughed and then nobody could say whether that was false.

The usual philosophical framing seems to be this. Pictures, words on paper and machine outputs have only derived intentionality: they are about things because minds make and read them. Minds have intrinsic intentionality: they are about things in their own right. Searle's Chinese Room was meant to show that manipulating symbols by rules, however well, is not enough for intrinsic aboutness.

But I keep running into two complications. First, Dennett's line that having beliefs just is being well predicted by the intentional stance. If that's right, and the stance works on these systems, maybe they do mean things in a thin sense. Second, in my own field the practical question is responsibility. If a system says something harmful, saying "it meant nothing" seems to move all the responsibility to the people who built and deployed it. Maybe that's correct. But it's a strong claim.

I'm not asking whether these systems are conscious. I'm asking the narrower question: can a system that is not conscious be about things, and can it mean what it says?

LikeReply
Follow5 replies

Members reply here. Reading is always free.

Sign in to reply

5 replies

  1. Oskar Nyberg

    Fellow

    It helps to split "mean". Grice distinguished what a sentence means from what a speaker means by uttering it, and speaker meaning, for him, required intentions about the hearer's recognition. A model's outputs can certainly have sentence meaning: the words mean what they mean in English. Whether the system has speaker meaning is the contested part. Your colleague's joke was about speaker meaning; the offence was in the sentence meaning.

    Helpful · 1
  2. Govinda Prasad Mishra

    Fellow

    Let me add a perspective from the Pūrva Mīmāṃsā, which I teach. The Mīmāṃsakas held that the Veda has no author, human or divine, and yet is meaningful, because the relation between a word and its meaning is not created by anyone's intention. Kumārila defended this at length. So the idea that meaning requires a speaker who means it is not universal. Of course, they were defending scripture, not machines, and they would be horrified at the comparison. But the logical point stands: authorless meaning was argued for very seriously in India.

    Helpful · 2
    1. Nur Aisyah Rahman

      Fellow

      Govinda, that is fascinating, and it cuts both ways for me. If meaning can exist without a meaner, then the model's outputs can be meaningful without the model meaning anything, which is roughly Oskar's split. Responsibility then sits with whoever puts the words into the world.

      Helpful · 3
  3. Derek Hollis

    Fellow

    I'd resist the intentional-stance argument a little. The stance works on these systems for short stretches and then fails in ways it doesn't fail for people: the system "believes" one thing in one paragraph and the opposite in the next, with no sign of noticing. Dennett's own criterion is predictive success across the board. By that criterion the case is mixed, which is maybe the honest answer.

  4. Pieter de Jong

    Fellow

    I build these systems. My view: they carry a huge amount of information about the world, so in Dretske's sense their internal states are informative. Whether information becomes aboutness is the whole question of naturalising intentionality, and the machines didn't create that problem, they just made it impossible to ignore.

    Helpful · 1