Skip to Main Content

Information Literacy

Introduction

Sometimes, instead of evaluating a single source (e.g., an article or a book), you want to evaluate a claim. For example, you may hear someone claim:

  • "Different people have different ways of learning (e.g., visual, auditory, or kinesthetic) and should be taught according to that 'learning style'." OR
  • "Children should be taught to read like adults read: memorizing and reading entire words and phrases at once, and not drilling on phonics."

Both of these are myths, not substantiated by robust scholarly research, but have been popularly believed.  So how do you know what to trust?

In Daniel T. Willingham's book When Can You Trust the Experts? (2012), he offers the following four-step process for evaluating scientific claims about education:

  1. Strip It and Flip It: Write the claim in its simplest form, devoid of emotion. Try flipping it: "90% effective" becomes "10% ineffective."
  2. Trace It: Find the origins of the claim, and pay attention to the qualifications and motivations of the person trying to persuade you. Mike Caulfield's SIFT strategy (2019) may be one way of tracing claims.
  3. Analyze It: Consider why you are being asked to believe something and evaluate the idea's scientific merit.
  4. Make Your Decision About Whether to Adopt It: Decide whether you will implement the change, based on the soundness of the evidence and its appropriateness for your context.

Strip It and Flip It

Willingham argues that you should start evaluating a claim by stripping it to its essentials by being very clear on the following (2012, p. 136):

  1. "precisely what Change [new curriculum, teaching strategy, software, plan, etc.] is being suggested
  2. precisely what outcome is promised as a consequence of that Change; and
  3. the probability that the promised outcome will actually happen if you undertake that Change"

You should also consider flipping the claims to ensure that you are not being biased by framing effects. For example, an action with an 85% pass rate implies a 15% fail rate.

Here are a few of the suggested actions that Willingham (2012) lists on page 164 and 165 for "stripping" and "flipping" a claim:

Suggested Action Why You're Doing This
Strip to the form "If I do X, then there is a Y percent chance that Z will happen." To get rid of emotional appeals, peripheral cues, and proferred analogies that may influence your belief. The scientific method is supposed to be evidence based and uninfluenced by these factors.
Consider whether the outcome (Z) has an inverse; if so, restate the stripped version of the claim using the inverse. To be sure that you appreciate all the consequences of the action -- for example, that a "85% percent pass rate" implies a "15% failure rate." We are subject to framing effects; we think something is better if the positive aspects are emphasized rather than the negative.
Evaluate whether the Change [new curriculum, teaching strategy, software, plan, etc.] is clear; "clear" means that you feel confident that you know how the Change will affect students' minds. To ensure that the Change is implemented as intended. Changes that sound good can go awry if they are not implemented in the classroom as intended or if students don't do what you're hoping they will do.
Evaluate whether the outcome (Z) is clear; "clear" means that there is some reasonably objective measure of whatever outcome you expect, how big the change in the outcome will be, and when it will happen. To be sure you will be able to tell whether or not the promised outcome is happening.

 

Trace It

It is important to trace the origin of a claim, particularly if the source cites other sources as the bases of the claims it makes.  Mike Caulfield's SIFT strategy may be helpful here:

  • Stop: When you start to read a source, stop and remind yourself to evaluate the claims using the SIFT method. In addition, once you begin to use the other three moves, if you find yourself getting overwhelmed, stop and remember your purpose for reading and fact-checking in the first place, and use that to determine how deep you need to evaluate the original source.
  • Investigate the Source: Evaluate the validity, relevance, and validity of the source. A very quick way to do this is by using Wikipedia to learn more about the author and/or publisher of the source! 
  • Find Trusted Coverage: If the source is not a valid source, look for other trusted reporting or analysis to see what they say about the claim. Ideally, look for multiple valid sources to see what the consensus about the claim seems to be.
  • Trace Claims, Quotes, and Media to the Original Context: Often, claims, quotes, and media cited in one source (particularly blog posts and other web pages) have been stripped of context. It can often be helpful to trace the claim, quote, or media back to the original source to see it in context.

 

Analyze It

Once you have stripped and traced the claim, you can try analyzing the claim itself.  Here are a few of the suggested actions that Willingham (2012) lists on page 205 for analyzing a claim:

Suggested Action Why You're Doing This
Compare the Change's predicted effects to your experience; but bear in mind whether the outcomes you're thinking about are ambiguous, and ask other people whether they have the same impression Your own accumulated experience may be valuable to you, but it is subject to misinterpretation and memory biases.
Imagine the opposite outcome for the Change that you predict. Sometimes when you imagine ways that an unexpected outcome could happen, it's easier to see that your expectation was shortsighted. It's a way of counteracting the confirmation bias.
Look up research on the Internet A logical argument in support of a Change is not the same thing as scientific data indicates that it works. You want to know whether such data exist. (p. 196)
Evaluate what was measured, what was compared, how many kids were tested, and how much the Change helped The first two items get at how relevant the research really is to your interests. The second two items get at how important the results are.

 

Make Your Decision About Whether to Adopt It

Just because a Change is scientifically sound doesn't mean you should adopt it. Here are a couple of the factors that Willingham (2012) suggests that you consider (p. 213):

 

Factor Example
Implementing a Change likely incurs a cost in time, energy, or other resources. Even if you believe that the promised benefits will accrue, you must weigh them against the anticipated costs. Reducing class size probably helps student achievement, but the benefit is modest. Meanwhile the cost is enormous because one not only needs more teachers (with salaries and benefits to be paid) but also more classrooms (increasing physical plant costs).
A Change may work as described, but may also have negative side effects. A friend of mine decided to homeschool her son. Her husband thought it was a bad idea, and it was the subject of several arguments, which ended when he said, "Fine, do it, but it's your thing. I'm staying out of it." My friend anticipated that homeschooling would be great for her son, but a strain on her marriage. She was right--her husband didn't stay out of it.

 

Additional Resources