What Type Of Error Am I Making?
In my life, I categorize my errors into three groups. I suppose I could use small, medium and large. And I could classify them as those I talk about, those I could talk about, and those I never talk about. But what I really do is sort them by how they make me feel. So I use emotional labels: stupid little errors, there-I-go-again errors and I-can’t-believe-I-did-that errors.
In science, there are only 2 kinds of errors: Type I and Type II. These are decision errors. That is, they are errors we make when we make a decision about the results we find. They are not errors in collecting the data (though that can happen too). Decision errors are what we do after we’ve analyzed the data.
A Type I error is deciding that your results are significantly different from chance, when in fact they are not. This is the equivalent of seeing things that aren’t there. A Type I error is the UFO of science. It is the Chicken Little of analysis: “I’ve collected the data, and I conclude that the sky is falling.”
Obviously, this is the worst kind of decision error you can make. It’s not good to find a false cure for cancer, psychosis, depression, and the common cold. Unfortunately, this is the kind of error we tend to make when we judge the universe based on our personal experience. We can find all kinds of significant findings in tea leaves, sand piles, and constellations of stars. Finding the pattern is fine. What isn’t good is to decide that the pattern we see is due a false causation: leaves change in fall because I sneeze; the sand at the beach is the result of Marians garbage dumping; or a new star appears when a person dies.
Type I error is jumping to conclusions of causation. Type II error is being blind to the truth. Although not as serious as Type I error, Type II error causes considerable distress. It delays progress and misleads people. It is not admitting that the world is round, not acknowledging that gravity impacts us, or not recognizing that skin color does not predict intelligence.
Type II error is scientific pretending. To counter it, we use replication. Replicating our findings allows us to hone our measurements. Over time, we get better at showing principles at work.
Type I error is scientific hallucination. The cure for it is to make our hypothesis falsifiable. That is, we design our experiment to prove something isn’t true. Science doesn’t prove things to be true, as much as it proves things aren’t true. To prove something true would require our testing every possible combination. Disproving only requires a single instance.
Proving only adds another brick to the theoretical structure. Disproving can fall the entire towering theory. “Monsters rule the universe” is hard to prove. But it is easy to test “There’s a monster under my bed.”
What science does is to state a hypothesis of no change. We assume that what we see is due to chance. And we only abandon that hypothesis when there is enough evidence.
We assume there is no difference between our wonder drug and getting an inert placebo. If there are small differences between the groups, we maintain our original hypothesis: no difference. If there are medium differences, we still hang on. We keep our hypothesis until we find “significant” statistical differences.
Type I error is rejecting our no-difference hypothesis (null hypothesis) when we shouldn’t. Type II error is accepting the null when we shouldn’t.