Noise
Why our judgments are so contradictory and how we can change that
You will learn:
Why you should always get a second opinion on critical medical diagnoses?
Why some court decisions by judges remind you of a lottery?
Why AI consistently makes better decisions than humans?
Why your forecasts and estimates can be all over the place?
How do you sort out this mess?
Summary
Noise is unwanted variability in judgments and decisions, especially repeated decisions. It is a measure of consistency in your decisions.
It's unexpectedly strong implications for businesses and entire industries that involve important and repeated decisions, such as insurance, medicine, and law.
Noise in human decisions is so strong that even computer models of humans perform better than the humans they're modeled after.
The noise can be contained using many of the same methods required for good decision-making.
Noise - The Book
One day I'll have his face tattooed on my right arm, that's for sure.
I'm talking about Daniel Kahneman - professor of psychology, behavioral economist, researcher, prolific author, and Nobel Prize-winning economist. Most of you probably know him through his famous book, Thinking Fast and Slow.
He and his co-authors (Olivier Sibony, Cass R. Sunstain) have a new book! It's called Noise: A Flaw in Human Judgment.
I'm a bit of a decision fanatic myself. When I started reading this book, I was excited, but also a little skeptical - along the lines of, "Sure, I'll read the book just because Kahneman is one of the authors, but I don't think there's anything surprising in it." Most books on behavioral economics these days are repetitive - they deal with the same psychological biases and mental models, just from slightly different angles. That's exactly what I expected here.
I wouldn't mention any of this if I didn't have a "But of course I was wrong!" in my pocket. This brilliant book brought up a topic that's always hovered in the air around us, but hasn't been seriously discussed or addressed. I'm talking about Noise.
I'm not going to tell you about the book again here. The goal of this article is to give you a taste of what Noise is and why it's so important so you've enough motivation to spend your hard-earned money on this wonderful if somewhat long, book.
What is Noise?
Noise is an undesirable variability of judgments about the same problem.
Now you know everything and we can tick off the matter... JK.
If you understand images better, here's the one that works best for me
Whenever you make a series of identical judgments, the noise is a variable deviation from the "correct" judgment - you're aiming for the bull's-eye, but hitting the target all over the place.
This affects every decision and judgment, not just the ones you make repeatedly. It's just more apparent in the repeated decisions.
Whenever a person makes a predictive or evaluative judgment about something, it contains Noise - and more than we think. That's, if that person had to make exactly the same judgment based on the same data, there's a very good chance that the judgment would be quite different. This is called Pattern Noise.
When a group of people who we expect to make identical judgments differ drastically in their judgments, this is called Level Noise.
Level and Patter Noise combine to create System Noise, which is the Achilles heel of many organizations that have similar cases judged by one or more judges over and over again. Medical decisions, investment decisions, forecasts of all kinds, personnel decisions, patent decisions, and more are prone to this type of error.
The noise arises from a lack of decision-making procedures and guidelines, a lack of data, or personal bias on the part of the individuals making a single decision. A unique combination of these factors creates a new answer, a new judgment, every time.
If that still sounds like something you can ignore, here are a few examples:
A noise audit at one insurance company found that while executives expect about 10% noise in underwriters' judgments, the actual result is 55%. That's, if one underwriter sets the premium at $9,500, another sets it at $16,700.
In Noise's study of 200+ U.S. judges, where they evaluated a series of identical cases, in the case where the average sentence was 7 years, the mean absolute difference between two arbitrary decisions is 3.8 years! Frankly, that looks more like a lottery.
Research confirms that Noise in the decision contributes as much to the overall decision error as statistical bias.
According to the research, which you can read in the book, humans SO BAD are able to deal with noise in their decisions and make consistent decisions, so almost any kind of decision automation based on data beats humans. That just blew my mind:
Models like linear regression that are trained on decision outcomes and input data beat humans in all cases. This isn't a surprise at all, but simply a fact. But read on...
Equally weighted models, that's, the average of the input data, perform worse than trained models (dah!) - but they still beat humans in most cases!
And now the thing that really shocked me - the model of you beats you! Once again, the model trained on your judgments and predictions tends to do better than you on future decisions. Kahneman says, "There is so much noise in judgment that a noise-free model of a judge achieves more accurate predictions than the actual judge does."
But even that doesn't stop there. In some studies, linear random models have outperformed humans.
"Their striking finding was that any linear model, when applied consistently to all cases, was likely to outdo human judges in predicting an outcome from the same information. In one of the three samples, 77% of the ten thousand randomly weighted linear models did better than the human experts. In the other two samples, 100% of the random models outperformed the humans. Or, to put it bluntly, it proved almost impossible in that study to generate a simple model that did worse than the experts did."
Hopefully, you now understand that this is a problem. As the authors note:
"We depend on the quality of professional judgments by underwriters, claims adjusters, and others. We assign each case to one expert, but we operate under the wrong assumption that another expert would produce a similar judgment."
And if you thought, "What the heck, I'll think about it when I deal with these organizations. Here's the bad news for you: Individual, one-time decisions suffer from Noise just as much as repetitive decisions in organizations. It's just a lot harder to measure.
If you make a decision that will never be repeated exactly (e.g., whether to marry a partner or make an investment) - your decision is still susceptible to Noise. This means that from a noise reduction perspective, you should think about those decisions and repetitive decisions that are made only once.
Before we move on, one more image here just to make sure the idea sticks in your mind:
How does one deal with this?
The good news is that the same noise reduction methods that work for businesses would work for you.
Another piece of good news is that the noise reduction methods are identical to traditional methods for making good decisions.
Let's take a look:
Focus on the Process, not the outcome. Most decisions aren't repeatable and not reviewable. That means you don't have a proper feedback loop and can't course correct based on the data. For this reason, it's the decision-making process that matters, not the outcome itself.
The psychological bias checklist is a good place to start. Part of the reason for the noise is the same biases I keep talking about on my blog.
Think statistically. Use probabilistic thinking, not Causal Thinking (or thinking in stories). This is a big topic that I'll talk about in a separate article.
Resist hasty intuitions. Do I even need to explain?
Use force ranking instead of an evaluative judgment. Research shows that you've a higher chance of achieving your goal if you force-rank your options instead of assigning them a specific number. This can be useful when you're evaluating or predicting something with numbers - chances of success, future costs, etc.
Aggregate and Average. Summarize several independent estimates/forecasts and balance them. Again, even if you do it yourself, you'll get better results on average.
Use Machine Learning. If possible, let the machines do the work. They'll be better at it.
Mediating Assessments Protocol (MAP). Use tools like MAP to make business decisions.
Hopefully, you now know why noise is a problem, how it can affect your business, and how to deal with it. I still recommend you read the book - it's one of those books worth taking the time to read.
Do you see the effects of Noise in your business decision-making?