“[We believe] we understand more about the world than we actually do. It is often not until we are asked to actually explain a concept that we come face to face with our limited understanding of it.”

— The Decision Lab

example: tell me what a house is

Imagine an alien comes to Earth, and asks you to explain the concept of houses. You agree, and start in – eagerly at first – but slowly realize all kinds of gaps in your knowledge.

At what point in history did humans begin constructing houses?
What are the different kinds of houses?
How are houses built, and what are the common building materials?
How are the prices of houses determined, and what are the laws governing their construction, modification, and sale?

You realize that you actually know much less than you thought.

IOED + Yale Grad Students + Toilets

Leon Rozenblit and Frank Keil are a pair of psychologists who asked Yale graduate students to explain, in as much detail as possible, their understanding of common devices (eg: toilets), procedures (eg: how to tie a bow tie), and natural phenomena (eg: how earthquakes occur). The full list is hilarious, and I’ve put it at the bottom of this page.

The instructions

“For each [item], please describe all the details you know […], going from the first step to the last, and providing the causal connection between the steps. That is, your explanation should state precisely how each step causes the next step in one continuous chain from start to finish. In other words, for each phenomenon, try to tell as complete a story as you can, with no gaps.

If you find that your story does have gaps (i.e., you are not sure how the steps are connected) please write the word “GAP” in your description at that point, and then continue. Feel free to use labeled diagrams, or flow-charts to get your meaning across.

When you are done, please re-rate your knowledge of the phenomenon on a 1–7 scale.”

— Leonid Rozenblit & Frank Keil, The misunderstood limits of folk science: an illusion of explanatory depth

What happened In the Study?

Everyone overestimated their knowledge. You can read about it in this wonderful paper: The misunderstood limits of folk science: an illusion of explanatory depth.

IOED + Software Engineering

Software engineers are just as delusional as Yale grad students. When you’re part of an engineering team, it’s normal to become superficially familiar with a TON of concepts, and then feel like you know a lot more than you do (even if you don’t write code 🤣). If you’re a full-stack developer, you’ve likely heard of some or all of these concepts and tools:

Prototypal vs Class inheritance, ()=> and the binding of this
Concepts like tech debt, cylomatic complexity, and big O notation
Syntactic differences between TypeScript vs. JavaScript vs JavaScript ES5, ES6, ES7, ES8, ES9, ES10
The concept of transpiling, the use of polyfills and shims, and tools like Babel, Webpack, SASS, Grunt, Gulp, ESLint, and Prettier
CI/CD, the difference between CI and CD, the use of green/blue deployments, A/B testing, and feature flags
Containerization, Docker, Docker-Compose, Kubernetes, pods, clusters, and service meshes
Serverless applications, infrastructure as code, and ephemeral services

☝🏽 That is a lot of stuff, and it’s not even 20% of the surface area of what a typical engineering team talks about in a single sprint.

Making software in 2021 is complicated

Back in 1993, the pool boy could build a web site. That’s no longer the case. The task of building and releasing Software has become so complex and differentiated, one person cannot possibly know a lot about a lot. At best, you can know a lot about a low number of things, and have superficial knowledge of lots of adjacent topics. But in general, there are too many sub-disciplines within software engineering for a single person to know how to design, build, test, and deploy an application from scratch… without receiving massive amounts of help in the form of:

A tricked out code editor
Self-scaffolding frameworks with boilerplate code
CI and/or some kind of build server
Containerized services for testing, auth, db, file storage, notifications, queueing, etc.
Desktop and cloud-based automation tools
Github gists + Stack Overflow + a shit-load of Google searches
More 😢

It’s natural for people who work in software engineering to feel overwhelmed. One coping mechanism is to pretend you understand more than you do, rather than saying “I don’t know”. I don’t recommend this – I’ve tried it.

Reference Borrowing (to the rescue?)

There is a thing called reference borrowing – shorthand for Saul Kripke’s causal theory of reference – which basically says that when we use names and tokens in language, there’s no burden of proof that has to be surmounted. So we can feel comfortable saying things like…

Plato was the teacher of Aristotle
Mark Twain was a master storyteller
Einstein developed general and specific theories of relativity

… without having met any of the people above, or possessing a paper trail of documentation to underwrite our claims. We can meaningfully make all those statements without tying philosophers into knots – which happens more than you’d think. According to Kripke’s theory, when we use names like Plato, we’re participating in a causal chain of linguistic references that ultimately lead back to an initial event, at which a name and its meaning were bound by an original utterer. In essence, there exists some chain of events that connects our utterance of Plato was the teacher of Aristotle all the way back to the flesh and blood Plato himself, along with some person’s observation that, gee there goes the teacher of Aristotle.

Modern Cognitive Science and Psychology

Philip Fernbach and Steven Sloman have modernized and simplified Kripke’s ideas, and framed them in ways that I think resonate well with any internet-literate audience.

“Knowledge isn’t in my head or in your head. It’s shared. You know that the earth revolves around the sun. But can you rehearse the astronomical observations and calculations that led to that conclusion? You know that smoking causes cancer. But can you articulate what smoke does to our cells, how cancers form and why some kinds of smoke are more dangerous than others? We’re guessing no. Most of what you “know” — most of what anyone knows — about any topic is a placeholder for information stored elsewhere, in a long-forgotten textbook or in some expert’s head.”

— Philip Fernbach and Steven Sloman

The Evolutionary Biology Spin on Reference Borrowing

Steven Pinker has observed that one of humans’ greatest talents is the ability to “shape events in each other's brains with exquisite precision.” I’d say it’s the killer feature.

Also…

“Humans’ biggest advantage over other species is our ability to cooperate. Cooperation is difficult to establish and almost as difficult to sustain. For any individual, freeloading is always the best course of action. Reason developed not to enable us to solve abstract, logical problems or even to help us draw conclusions from unfamiliar data; rather, it developed to resolve the problems posed by living in collaborative groups.”

— Hugo Mercier and Dan Sperber, the Enigma of Reason (Harvard)

Language facilitates large-scale cooperation, which in turn alters the brain

Joseph Henrich has written an astounding book, tracking how specific cultural events like the Protestant Reformation altered our brain development.

Use it or lose it

The brains of London cab drivers are different in 2021 than they were in 1991 because the availability of cheap GPS technology has de-emphasized the need to develop enlarged hippocampuses.

Reference Borrowing is a feature, not a bug

The world of 2021 is orders of magnitude more complex than the world of 1981. We’re living in an age where we have no idea what set of skills and knowledge will be valuable 30 years from today.

When I think of the stuff my daughters have to know, just to get through their school day, I’m grateful that they don’t have to waste a single calorie of energy on unnecessary tasks like remembering to rewind vcr tapes before they return them to the store.

The Secret Sauce: Collective Fictions

Yuval Noah Harari says that humans are very good at efficiently deploying a wide array of collective fictions – eg: religion, money, laws, human rights, systems of credit, Facebook likes – that are completely imaginary, yet serve a social purpose in the furtherance of civilization’s “agenda”. Kind of like a gigantic game of make-believe that everyone plays, either knowingly or in blissful ignorance. I believe that reference borrowing is one of the premier tools in our repertoire that allows every man, woman, and child to get in on the action.

Reference Borrowing !== IOED

Reference borrowing is a facilitating tactic that allows language participants to engage in a wide array of discourses, confidently saying things like…

North Korea is a rogue state
Quantum computing is rad
The multiverse is real

… without having to earn multiple phd’s. And without making a mountain of preliminary clarifications, justifications, and caveats. People can just talk and say things, and hopefully good ideas will sprout forth. But this occurs with the presumption of objective factual accuracy: that at the endpoint of each speaker’s chain of causal references, lies a true proposition which could theoretically be validated by some mechanical procedure. Academic folks just call this intellectual honesty.

IOED does not do the same thing. It insists that it’s right, and is resistant to the introduction of inconvenient conflicting truths – an approach popularized by Kellyanne Conway and her alternative facts “school of thought”.

The Dunning Kruger Effect

In 1995, McArthur Wheeler walked into two Pittsburgh banks and robbed them in broad daylight, with no visible attempt at disguise. He was arrested later that night, less than an hour after videotapes of him taken from surveil-lance cameras were broadcast on the 11 o'clock news. When police later showed him the surveillance tapes, Mr. Wheeler stared in incredulity. “But I wore the juice,” he mumbled. Apparently, Mr. Wheeler was under the impression that rubbing one's face with lemon juice rendered it invisible to videotape cameras.
Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments

“The Dunning Kruger effect states that when you know only a little about a topic, you tend to think you know a lot”

— Productive Club

Self-assessments of competence, when performed by the incompetent, are grossly inflated. The reverse is true of highly competent people: they under-estimate their knowledge.

Can you identify the person responsible for these self-assessments of competence?

“I understand the tax laws better than almost anyone.”
“I know more about renewables than any human being on Earth.”
“I understand the power of Facebook maybe better than almost anybody.”
“Nobody knows more about debt. I'm like the king.”
“Nobody knows banking better than I do.”
“I understand money better than anybody.”
“Nobody knows more about trade than me.”
“Nobody in the history of this country has ever known so much about infrastructure as [me].”
“There's nobody bigger or better at the military than I am.”
“I know more about ISIS [the Islamic State militant group] than the generals do.”
“I know more about drones than anybody.”
“Nobody knows more about technology than me.”
“Nobody in the history of this country has ever known so much about infrastructure as [me].”
“I know a lot about wind.”
“I know more about renewables than any human being on Earth.”
“I know more about courts than any human being on Earth.”

Full List of IOED devices in the Rozenblit/Keil study

Below is the list of common devices that Rozenblit and Keil asked Yale graduate students to descibe in detail:

How a sewing machine works
How a flush toilet operates
How an LCD screen works
How a hydroelectric turbine changes water pressure into electricity
How a can opener works
How a car battery stores electricity
How a 35 mm camera (single-lens reflex camera) makes images on film
How a jet engine produces thrust
How a zipper works
How a self-winding watch runs without batteries
How a cellular phone works
How a microchip processes information
How a greenhouse works
How the U.S. Supreme Court determines the constitutionality of laws
How a fluorescent light works
How a photocopier makes copies
How a nuclear reactor produces electricity
How a car ignition system starts the engine
How a speedometer works
How the liver removes toxins from blood
How the heart pumps blood
How a car differential helps the car turn
How a water faucet controls water flow
How the presidential elections determine the next president
How a quartz watch keeps time
How steam central heating warms large buildings
How a VCR works
How a snare catches small animals
How a car’s gearbox works
How an incinerator works
How a cylinder lock opens with a key
How a television creates pictures
How a helicopter flies
How a ball-point pen writes
How a radio receiver works
How an electric motor changes electricity into movement
How a telephone transmits sound through wires
How piano keys make sounds
How a fireplace works
How a spray-bottle sprays liquids
How a solid-fuel rocket produces thrust
How a manual clutch works
How the aqualung (Scuba-gear) regulates air-pressure
How an Ethernet network allows computers to share files
How a computer mouse controls the pointer on a computer screen
How a transistor works
How a scanner captures images
How the brain coordinates behavior

Illusions of Explanatory Depth (IOED)