by Haytham ElFadeel - hfadeelm@gmail.com
2014, updated 2019
In 2013, I found myself thinking about a deceptively simple question: What is understanding?
I was building a startup focused on automated knowledge extraction and question answering, and the question felt fundamental. What does it actually mean for a person - or a machine - to understand something?
I took a week off from my startup just to think about it. Here is what I came up with.
A bit of history
One of the dominant schools of psychology during the first half of the twentieth century was behaviorism.
Behaviorists argued that we could build a science of behavior without needing to know what was happening inside the brain. The brain could largely be treated as a black box: we could observe an animal's environment and its behavior - what it senses and what it does, its inputs and outputs.
Learning could then be studied through mechanisms such as conditioning, reward, and punishment, without having to deal with difficult subjective concepts such as hunger, fear, meaning, or understanding.
Behaviorism eventually declined as the dominant paradigm in psychology during the second half of the twentieth century. AI did not simply emerge from behaviorism - early AI was heavily influenced by logic, cognitive science, cybernetics, and other traditions - but a related question remained central: is producing the right behavior enough to constitute intelligence?
I have always been skeptical of that idea. I believe that trying to build AI entirely by reproducing the correct input-output behavior, without thinking about the internal representations and mechanisms that make that behavior possible, makes the problem substantially harder.
This brings us to one of the most famous thought experiments in the philosophy of AI.
In 1980, UC Berkeley philosopher John Searle introduced the Chinese Room argument. It goes roughly like this:
Suppose you have a room with a slot in one wall, and inside is an English-speaking person sitting at a desk. He has a large book of instructions for manipulating Chinese characters.Someone outside the room passes in a story and questions written entirely in Chinese. The person inside does not understand Chinese at all. He simply follows the instructions in the book: copying symbols, comparing them, rearranging them, and writing new symbols according to the rules.
Eventually, he produces answers and passes them back through the slot.
To a Chinese speaker outside the room, the answers may be completely correct—even insightful. From the outside, it appears that whoever is inside understands Chinese.
But who actually understood the story?
The person certainly did not. He was merely manipulating symbols according to rules. The rulebook itself does not understand anything either.
Searle's argument was that a digital computer is analogous to this room: the person is the processor, the rulebook is the program, and the scratch paper is memory. Correctly manipulating symbols is not, by itself, sufficient for understanding.
I disagree with Searle's conclusion that computers cannot understand. But I think the thought experiment exposes an important question:
What is missing between producing the correct answer and actually understanding why it is correct?
Understanding as compression
After spending a significant amount of time thinking about this, I arrived at the following view:
Understanding is a form of knowledge compression.
More precisely:
To understand something is to build a compact model that captures and exploits the structure of a problem or environment well enough to reason about it.
As someone who had been building automated knowledge extraction and question-answering systems, I could connect this idea to the Chinese Room fairly naturally.
Imagine two possible instruction books inside the room.
In the first case, the book contains an enormous lookup table:
Input 1 → Output 1
Input 2 → Output 2
Input 3 → Output 3
...and so on.
For every possible question, the correct answer is already explicitly stored somewhere in the book.
Such a system could behave intelligently, but the size of the book would have to grow roughly with the number of situations it might encounter. It has discovered essentially no underlying structure. I won’t call that understanding. Now imagine a very different book.
Instead of containing every possible question and answer, it contains a compact model of the Chinese language, concepts about the world, and rules for reasoning about them.
That relatively small model could generate answers to an enormous number of questions it had never encountered before.
Now something qualitatively different has happened.
The book no longer memorizes the mapping from every possible input to every possible output. It has captured structure.
On this view, the system as a whole - the book plus the machinery executing it - can reasonably be said to understand.
The important distinction is therefore not simply whether a system produces the correct output. It is how much structure it has discovered that allows it to produce that output.
This is what I mean by compression.
And by compression, I don't mean arbitrary data compression. A ZIP file can compress information, but nobody would say that a ZIP file understands its contents.
The relevant kind of compression is structured and usable compression: discovering regularities that allow a model to reconstruct, predict, infer, and generalize beyond the observations from which it was built.
A simple analogy
Think about fitting a curve to a set of data points.
Suppose you have five points.
You could construct a sufficiently flexible function with five degrees of freedom that passes perfectly through all five points.
It achieves zero training error.
But it may have learned essentially nothing about the process that generated those points. Give it a sixth point and it may make a terrible prediction.
Alternatively, perhaps all five observations can be explained approximately by:
y = 2x + 1
That tiny equation represents an enormous compression of the observations.
More importantly, it captures a regularity that allows us to predict what will happen at points we have never seen.
The compression is evidence that we have discovered something about the underlying structure.
This is why memorization and understanding feel fundamentally different.
A sufficiently large lookup table can memorize the world.
A model understands the world to the extent that it discovers a smaller set of structures that can explain it.
There is therefore a tradeoff. The best understanding is not simply the smallest model, nor simply the model that memorizes the most facts. It is a model that achieves high explanatory and predictive coverage with relatively little complexity.
Two other ways to think about understanding
When I originally wrote these thoughts, compression was the framework that made the most sense to me. Since then, I have encountered two other views of understanding that I think are important—and, rather than contradicting the compression view, they help complete it.
Understanding as grasping causal and explanatory structure
A major view in the philosophy of science is that understanding means grasping why something is the way it is. Knowing that something happens is different from understanding why it happens.
You might know that releasing an object causes it to fall. You understand more when you know how gravity explains the observation, how the relevant variables relate to one another, and what would happen if those variables changed.
Michael Strevens expresses a version of this idea very directly: scientific understanding requires grasping an explanation. Related accounts appear in the work of philosophers such as Stephen Grimm and Henk de Regt.
This adds something important to the compression view.
Not every compact representation is equally valuable. A particularly powerful representation captures the causal and explanatory dependencies of the world.
If I understand a car engine, I don't simply possess a compressed list of observations about engines. I have some model of how combustion creates pressure, how that pressure moves a piston, how the crankshaft converts that motion, and how changing one component affects the others.
The structure of the model matters.
Understanding as ability
Another view asks a more operational question:
What can someone who understands something actually do that someone who merely knows facts cannot?
If you really understand a concept, you should usually be able to:
- explain it in your own words;
- apply it to a situation you have not seen before;
- make predictions from it;
- identify when and why it does not apply;
- reason about counterfactuals - what would happen if things were different?
This idea appears in both philosophy and education research. Philosophers such as Alison Hills and Stephen Grimm, for example, connect understanding with possessing the cognitive abilities required to manipulate and apply what one knows.
This is appealing because understanding is not merely something stored in your head. It reveals itself through generalization.
Someone who memorized Newton's equations but cannot use them on a slightly unfamiliar mechanics problem knows the equations but does not understand classical mechanics very deeply.
Someone who understands them can take a new situation, construct the relevant abstraction, predict what will happen, and explain why.
Three sides of the same idea
I increasingly think these three views describe different sides of the same phenomenon.
Compression tells us something about the representation.
Understanding replaces a large number of disconnected observations with a smaller model that captures their common structure.
Explanation tells us what kind of structure matters.
The strongest models capture dependencies, mechanisms, causes, and relationships that explain why observations take the form they do.
Ability tells us how we can test whether the model is useful.
If the model represents genuine structure, we should be able to use it to predict unfamiliar situations, answer counterfactual questions, explain observations, and solve new problems.
Put together, I would now modify my original definition slightly:
Understanding is having a compact model that captures the important structure and dependencies of a domain well enough to support explanation, prediction, generalization, and counterfactual reasoning.
Or, even more compactly:
To understand something is to discover a representation that makes many things follow from a few things.
That is why understanding feels different from memorization.
When we memorize, every new fact adds another fact we have to store.
When we understand, many facts become consequences of a smaller number of ideas.
And perhaps that is one of the best indicators that understanding has actually occurred: the world becomes simpler without becoming less useful.