Accurate Confidence is a Trainable Measurable Skill

[sparr]

Without looking it up, who was born first, Mother Teresa or Ronald Reagan? Spoiler alert, they were born six months part, so very few people will know enough historical trivia to get this right other than by luck. It would be a coin flip for me. Everyone seems to be comfortable with the idea that you can seek out and learn more historical facts and other trivia, increasing your chances of getting questions like this correct. Now... How confident are you that your answer is correct? Very few people seem to be aware that you can learn to be better at this second question. Many people even deny that it is possible for one person to improve that skill or for someone to be better at it than someone else, let alone to measure it.

There are a few well known ways to demonstrate the existence of this skill and to measure it. A useful search term is "confidence calibration", which will lead you to various tests and assessments. An acquaintance of mine previously compiled a list of such exercises at https://www.lesswrong.com/posts/LdFbx9oqtKAAwtKF3/list-of-probability-calibration-exercises. 

One popular version ("confidence rating") asks you a series of questions with fill in the blank or multiple choice answers, then for each question how confident you are of your answer. For a fill in the blank question the confidence range might be 0-100%, while for a multiple choice or binary question the floor is usually equal to choosing an answer at random (e.g. 25% for 4 answers). If you have well calibrated confidence, the results might tell you that when you were 70% confident your answers were right 68% of the time. If you have poorly calibrated confidence, that result might be 50% or 90%.

Another version ("confidence interval") will set a particular target confidence (e.g. 80%) and ask you a series of questions with numerical answers (dates, weights, counts, etc). Instead of a single answer you give a range that you are 80% confident the answer falls in. At the end, if the true answer was in your range about 80% of the time, that's a good outcome, but if the true answer was in your range 50% or 95% of the time then there's a lot of room for improvement.

When someone without practice evaluating their level of confidence takes a confidence interval test targeting 90% success, they will usually succeed 30-60% of the time. When they take a confidence rating test, their confidence will correlate to their actual success rate similarly poorly. In my experience, a single demonstration of this outcome, alongside the results of someone with more calibrated confidence, will often suffice to convince them that there is something to this idea.

This skill extends beyond trivial examples and tests. It applies to far more impactful scenarios like planning professional projects, scheduling travel, buying stocks, signing a contract, negotiating a lawsuit settlement, etc. If you recognize this skill in yourself, it will allow you to make more effective decisions toward your goals. If you recognize this skill in others, it will better inform your reliance on their predictions and claims. Denying that this skill exists, or that you and other people can get or be better at it, will lead to less optimal outcomes in so many ways.

My goal in writing this post is to be able to link to it in the future when I encounter one of those people who don't believe this skill exists. Maybe at least a few of the people who aren't willing to take such a test themselves will instead be willing to have the nature of such tests explained to them, and a few of those will understand it enough to realize their mistake. Perhaps some of them will be interested enough in improving this skill that they will do some of the exercises and tests in the compiled list linked above.