Have you heard of the Dunning-Kruger effect? It’s an interesting psychological insight about how people’s actual knowledge and skill aligns with their perceived knowledge and skill.

People who are not very skilled at a particular task find it very hard to understand the level of skill needed to do that task well. As a consequence, they fail to recognise a truly skilled practitioner when they see one, and they tend to overestimate their own skill. As a people get better at a particular skill, they also get better at accurately estimating how good they are at it. This is true for a wide range of skills and knowledge areas, including grammar, logical thinking, and even humour. It was first described in a 1999 paper by Kruger and Dunning entitled Unskilled and Unaware of it.

Now, here is the problem.

You come across the Dunning-Kruger effect a lot on the internet. It seems to be everybody’s favourite cognitive bias. I certainly like it a lot. I think it is easy to like, because it basically suggests that people who think they are super smart are probably not as smart as they think they are, and this is a very comforting thought.

Unfortunately, it’s not only a comforting thought, but also a lazy thought. Just because most stupid people think they are smart does not mean that most of the people who think they are smart are actually stupid.

While I’m at it, there is a closely related statistical fallacy that often appears in this context. It is exemplified in this cartoon from Spiked Math Comics:

The argument goes like this: Most people think they are smarter than average, but only about half of all people can be smarter than average, so a lot of people must be wrong about being smart. This is, of course wrong, because the average and the median are not the same thing. Depending on how “being smart” is distributed, it might well be that most people are indeed smarter than average (just like most people actually have more arms and legs than average. Go do the math.)

So, much of the way in which we make intuitive sense of the Dunning-Kruger effect is actually statistically flawed. Still, the effect does of course exist, and it is understandably popular. So popular in fact, that you can find a number of graphs illustrating it on the internet. Here is my favourite (which I found on this blog, among other places):

Now, let’s have a look, shall we? First of all, the legend says: “Nobel Prize Psychology 2000”. This is indeed a legend. There is no Nobel Prize in Psychology. What Dunning and Kruger won is the Ig Nobel Prize, which is awesome, but not the same thing.

The plot shows confidence in what you know as a function of how much you know. The curve starts at zero (if you really know nothing, then you are obviously not confident in your knowledge). Then it skyrockets up, illustrating that people who know very little think they are super smart. As more knowledge is acquired, confidence starts going down again (presumably as people start knowing enough to grasp how little they know). Then, as you move towards expert mastery, your confidence goes up again, but never to the astronomical levels of the know-it-all beginner. Makes sense, right?

It’s also wrong. If you look at the actual data, it looks quite different. This is a plot from Kruger and Dunning’s actual paper (measuring confidence and actual skill in a test of grammar).

So the actual data looks more like … well, more like a straight line almost. It’s a bit hard to tell, because there are no error bars, but to me, it seems like everybody (regardless of actual ability) is just saying: “Yeah, I guess I’m doing alright. 60th to 70th percentile probably.” Of course, this means that less proficient people actually overestimate their skill level, but it also suggests that more proficient people underestimate their skill level. I don’t see evidence for the claim that metacognition improves, it is just that with increasing skill (up to a point) you get closer to the level you always thought you had. It is still an interesting finding, but it is far less dramatic than most people seem to think.

So let’s summarise: A lot of people like to cite the Dunning-Kruger effect, thereby displaying their knowledge of psychology (usually to prove some sort of point about people who think they are smart). In doing this, they often show questionable statistical understanding, use falsified plots, and reveal that they have not taken the half hour or so it takes to actually read the original research. But isn’t that often the case? People who know very little about a topic tend to overestimate their own expertise.

I just wish there was a name for that …