This is based on a talk I gave at the Open Source Open Mic meetup, around Sep 2015. Slides from the talk.

Let’s say you’re walking down the street. You see someone walking towards you, and a question comes to mind:

barista-or-consultant

Barista or Consultant?

(A contrived case for sure, but humor me)

How would you answer this?

You could reason that this person’s clothes are quite representative of hipster fashion, so you might say that they are quite probably a barista.

But the representativeness heuristic isn’t so dependable in general.

Perhaps this is a better way to think about the probability of this person being a barista or a consultant:

two-dimensional-comparison

The y axis is an estimate of the share of the overall population, and the x axis represents how likely either is to be wearing the clothes that this person on the street is wearing.

If there are many more consultants overall in the world compared to baristas, it might be that this person is actually likelier to be a consultant!

This is Bayes’ Rule, in essence. Bayes’ Rule relates prior beliefs to future beliefs via evidence adjustment. Here’s the basic rule:

prior odds * evidence adjustment = posterior odds

You can get a lot more rigorous, but working with odds is easier and harder-to-mess-up.

Convert probability <-> odds like so: P=O/(1+O); O=P/(1-P)

A (slightly more) Real Problem

  • 1% of the population has cancer
  • 9.6% of healthy people test positive, 80% of people with cancer do
  • If Sally tests positive for cancer, what’s the probability she actually has it?

Can you solve this using the rule I provided above? Try it!

Usually, only around 15% of doctors get it right!

.

.

.

Ready for the answer?

.

Here it is:

  • Cancer:Healthy ratio is 1:99
  • Evidence adjustment: 80/100 : 9.6/100 = 80:9.6
  • 1:99 * 80:9.6 = 80:950.4
    • (roughly 1:12 odds of cancer, (1/12)/(1+1/12) => ~7.7% chance)

Takeaways

1. Remember the Base Rate!

If you just saw $event and are trying to decide whether it’s because $a or $b…

  • think about how likely $event is in case of $a or $b
  • but also, think about the base rate, how likely are $a and $b by themselves

2. How would things look if you were wrong?

Imagine your guess/theory is false; would the world look different?

weighing-evidence

Geeky Diversions!

What if you could add/subtract evidence?

Turns out… there’s Log Odds!

  • priors of 1:10 in favor of Consultants
    • => 10*log(1:10) => -10 dB
  • evidence adjustment: 5:1 in favor of Baristas
    • => 10*log(5:1) => ~7 dB

  • adjusted odds: -10 + 7 => 3 dB

  • and back to odds => 1/10^(3/10) ~> 0.5 => 1:2

But why bother?

  • Probability of 100% or 0% is not reachable, log odds make that obvious
    • 10 * log(10000000000000) is only 130 dB, and so on
    • 10 * log(1/10000000000000) is only -130 dB, and so on

As you go to higher/lower odds, the values increase/decrease more and more slowly.

Further Reading