Question · Open

If a system passes every behavioural test, what would still be missing?

Pieter de Jong Fellow

September 11, 2026 · 4 replies

AI & computing Analytic #behaviourism #turing-test

Suppose a future system passes every test we can think of. It holds conversations, explains its reasoning, makes jokes that land, learns new tasks quickly, and reacts appropriately to pain-like signals in a robot body.

Some people would say there's nothing more to ask: being intelligent just is behaving intelligently in the right ways. Others say something could still be missing: consciousness, understanding, genuine intentions.

My question isn't whether such a system is conscious. It's what the "something missing" could even be, if no test could reveal it. Is it an empty idea, or is there a coherent way to say what we'd be worried about?

4 answers

Priya Raghavan 4 weeks ago

Fellow

Ned Block's "Blockhead" is the classic case: a giant lookup table that gives the right answer to every possible conversation of a given length. It passes every behavioural test, and intuitively it's not intelligent at all. So how the behaviour is produced seems to matter, not just what the behaviour is.

Helpful · 2
1. Pieter de Jong 4 weeks ago
  
  Fellow
  
  Blockhead is useful, but it's physically impossible: the table would be larger than the universe. Maybe our intuitions about impossible machines shouldn't count for much.
  
  Helpful · 3
Arjun Iyer 4 weeks ago

Fellow

What would be missing is a subject. Behaviour is always observed from outside. The question is whether there is an inside. That no test can settle it doesn't make the question empty; it shows the limit of tests.
Nur Aisyah Rahman 4 weeks ago

Fellow

Practically, I worry less about the metaphysics and more about the moral side. If we can't tell, which way should we err? Treating a conscious system as a tool seems worse than being too polite to a clever machine.

Helpful · 1

Members reply here. Reading is always free.