Criteria for judging evidence

I spoke a bit in class today about how we can know if evidence (especially things like statistics or polls) are credible. I promised I'd outline those criteria here for you to peruse (and study for the upcoming quiz):

Sources for Evidence

• Is the source referenced? A newspaper or magazine should tell its readers where the evidence comes from.
  
• Is the source referenced by other people? Although it won't tell you much about the validity of specific sources, knowing that it's widely accepted can help (versus, for example, a claim by some scholar that is not widely accepted by his/her colleagues).
  
• Who else uses the source? If the data or figures are widely used (especially if by both sides of a debate), they are probably viewed as credible by many different people. If the data or figures are themselves a matter of debate, you should approach with caution.
  
• Does the source present its findings with transparency? A good source will tell you the methods of their data collection, and may even tell you about some potential limitations (being self-critical) of their data.
  
• Are multiple sources saying the same thing? Somewhat different from the above, you can look to see if several different, unrelated polling groups or think tanks came up with similar results in their surveys or studies. For example, if three different drug testing labs came up with the same results about a new medicine, the results are probably pretty convincing.
  
• Finally, you can look at the historical credibility of the source. Is the source known for being credible? Has the source been wrong in the past? How wrong?

Surveys/Polls

• Is the sample representative of the population? A good survey should include a good cross section of the population under study (if you're only studying women, it should still include differences by age, geography, race, religion, income, etc.).
  
• Is the sample random? The best way to to guarantee a representative sample is to make a random sample.
  
• What is the sample size? If a poll is referenced, it should tell us something about sample size. A good, random sample of around 1,200 is usually fairly accurate. Bigger is better, if you're looking at more & more variables.
  
• What were the questions? A good survey should have clearly worded, and unbiased questions. If the report can, it should mention the exact question used, and the possible answers available to respondents.
  
• What was the question order? Often, surveys include more than one question. What other questions where asked is important. What order they are asked in can significantly influence participant responses.
  
• Is the margin of error reported? All surveys and polls have a statistical margin of error. Is it 95%? Or 97%? How accurate do you want? If a poll says candidate A will win over B by 51% to 48%, but the margin of error is +/-3%, that is not useful to us (it could be 54% to 45% or 48% to 51% the other way).
  
• What date was the poll taken? Opinions can quickly change, it's important to know when a poll was taken. Because perhaps opinion might change one way or another based on events that happened after the poll.
  
• Is the poll widely used or accepted? As with any evidence, if the polling organization is widely respected and used, it's a good indication that maybe you can trust this poll.

Keep in mind that none of these things can tell you beyond a shadow of doubt whether the evidence is true or false. But it can give you a good idea of whether you should trust it or not.

Very often, in politics, we have to make decisions based on imperfect information. Mostly, because we either don't really know the situation. Or because we can't really predict the future (economic projections that go 50 years into the future are very, very difficult). And those of use who aren't experts in one field or other, have to rely on experts and what they tell us. But if we know a little about how to judge sources of evidence, we can protect ourselves from false information.