Look At This Chart Lie
An army of pollsters have fanned our across America to gauge the pulse of the nation on the latest issue of the day. Phones are ringing and average Joe is sharing his two cents about what should happen to Terri Schiavo's feeding tube.
What happens to the numbers from these phone calls can have an important impact on how people see this important issue. Because of this simple fact, new organizations have an obligations to present poll results in their proper context.
Today, CNN got it horribly wrong.
This graph leads the viewer to believe that Democrats are far more eager to remove Mrs. Schiavo's feeding tubes than Democrats or Independents. It also, as Media Matters for America points out, obscures the fact that majorities of all groups approved of the court's decision to remove the feeding tube.
To correct the error, Media Matters offered an improved chart:
I suggest, however, that it should be taken a step further. The essential problem with CNN's display of the data is corrected in the Media Matters chart. The y-axis is now correct and the reader is better able to appreciate the relative difference between 62% and 54%.
However, this data is useless because of the little text in the lower right hand corner. The sampling error of this poll is +/- 7% points!
Here is a chart I whipped up that reflects this fact:
In statistics this is what we call honesty. Any survey has a margin of error and to present a chart with a +/- 7 spread without displaying that fact is disingenuous. This survey shows no statistical difference between Republicans, Democrats, and Independents on this issue. They are within overlapping ranges of likelihood to feel that the court acted correctly.
The one thing this survey does reveal is that a near majority of all three groups supports the decision. Why isn't that the headline?
2 Comments
Leave a comment
Who is this guy?
I've just finished reading Edward Tufte's first two books on information design, The Visual Display of Quantitative Information and Envisioning Information. There are some pretty funny/sad examples of misrepresenting facts through bad information displays, but this one may be more egregious than any of his. And he's done a lot of research on bad graphical displays of information.
Right, on, Sam. Sad that CNN would publish something so blatantly misleading. Innumeracy or malign, I don't know, bad either way.
About the intervals, there is one more detail missing: the confidence value. Saying +/-7 points is actually meaningless without stating with what confidence that is being said. Are we 90% sure the real value is in a +/-7 interval? or 95% sure? I believe there may be a standard confidence for opinion polls that I am not aware of, but why not make it explicit?
This is not trivial. For example with 80% confidence the interval is much shorter, say +/-3 pts. With 98% confidence it would be very large, say +/-20 points. The interval is really a normal distribution (bell curve) of probability, I believe (think of it being sideways, centered on the estimate). This means that the real value is most likely not too far from the measured one, but as you get close to 100% confidence, the interval expands drastically out the tails of the bell curve... At 100% confidence the range of possibility is always from 0-100%!
Even for readers who don't understand exactly what that all this means, saying "+/-7 points with 95% certainty" is not THAT cryptic, and it hints at the fact that NO sampling is 100% which is the truth.
One final point is that even though the confindence intervals overlap, I'm not sure you can say that the differences are statistically insignificant. Not sure, guess there is an implicit confidence % here too. After all, the length of the intervals depends on confidence assigned to them. Because of the bell curve, the real value is probably not too far from the stated ones. It all depends on how sure we want to be :-) Oh, and CNN has fixed it's graph now! Nice blog, Sam!