Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Information Literacy

Introduction

In Daniel T. Willingham's book When Can You Trust the Experts? (2012), he offers the following four-step process for evaluating scientific claims about education:

  1. Strip It and Flip It: Write the claim in its simplest form, devoid of emotion. Try flipping it: "90% effective" becomes "10% ineffective."
  2. Trace It: Find the origins of the claim, and pay attention to the qualifications and motivations of the person trying to persuade you. Mike Caulfield's SIFT strategy (2019) may be one way of tracing claims.
  3. Analyze It: Consider why you are being asked to believe something and evaluate the idea's scientific merit.
  4. Make Your Decision About Whether to Adopt It: Decide whether you will implement the change, based on the soundness of the evidence and its appropriateness for your context.

Strip It and Flip It

Willingham argues that you should start evaluating a claim by stripping it to its essentials by being very clear on the following (2012, p. 136):

  1. "precisely what Change [new curriculum, teaching strategy, software, plan, etc.] is being suggested
  2. precisely what outcome is promised as a consequence of that Change; and
  3. the probability that the promised outcome will actually happen if you undertake that Change"

You should also consider flipping the claims to ensure that you are not being biased by framing effects. For example, an action with an 85% pass rate implies a 15% fail rate.

Here are a few of the suggested actions that Willingham (2012) lists on page 164 and 165 for "stripping" and "flipping" a claim:

Suggested Action Why You're Doing This
Strip to the form "If I do X, then there is a Y percent chance that Z will happen." To get rid of emotional appeals, peripheral cues, and proferred analogies that may influence your belief. The scientific method is supposed to be evidence based and uninfluenced by these factors.
Consider whether the outcome (Z) has an inverse; if so, restate the stripped version of the claim using the inverse. To be sure that you appreciate all the consequences of the action -- for example, that a "85% percent pass rate" implies a "15% failure rate." We are subject to framing effects; we think something is better if the positive aspects are emphasized rather than the negative.
Evaluate whether the Change [new curriculum, teaching strategy, software, plan, etc.] is clear; "clear" means that you feel confident that you know how the Change will affect students' minds. To ensure that the Change is implemented as intended. Changes that sound good can go awry if they are not implemented in the classroom as intended or if students don't do what you're hoping they will do.
Evaluate whether the outcome (Z) is clear; "clear" means that there is some reasonably objective measure of whatever outcome you expect, how big the change in the outcome will be, and when it will happen. To be sure you will be able to tell whether or not the promised outcome is happening.

 

Trace It

It is important to trace the origin of a claim, particularly if the source cites other sources as the bases of the claims it makes.  Mike Caulfield's SIFT strategy may be helpful here:

  • Stop: When you start to read a source, stop and remind yourself to evaluate the claims using the SIFT method. In addition, once you begin to use the other three moves, if you find yourself getting overwhelmed, stop and remember your purpose for reading and fact-checking in the first place, and use that to determine how deep you need to evaluate the original source.
  • Investigate the Source: Evaluate the validity, relevance, and validity of the source. A very quick way to do this is by using Wikipedia to learn more about the author and/or publisher of the source! 
  • Find Trusted Coverage: If the source is not a valid source, look for other trusted reporting or analysis to see what they say about the claim. Ideally, look for multiple valid sources to see what the consensus about the claim seems to be.
  • Trace Claims, Quotes, and Media to the Original Context: Often, claims, quotes, and media cited in one source (particularly blog posts and other web pages) have been stripped of context. It can often be helpful to trace the claim, quote, or media back to the original source to see it in context.

 

Analyze It

Make Your Decision About Whether to Adopt It