Social purpose organizations need better ways to determine the quality of the evidence they use for making decisions. They need the strongest possible evidence they can find to plan effective action, demonstrate the value of their activities to funders and stakeholders, and influence public policy.
Yet funders, policy makers, and managers face competing definitions of what constitutes quality evidence. And while using evidence of any kind to make decisions is better than using no evidence, existing definitions may cause leaders to overlook valuable information that would improve their chances of making truly effective decisions.
Many in the field are calling for higher standards for evidence, or “stronger evidence.” Prevailing evidence hierarchies include the US Department of Education’s What Works Clearinghouse and the US Department of Health and Human Services’ Teen Pregnancy Evidence Review. But those focus on finding good supporting evidence for whole programs, policies, or products. They don’t consider how well evidence addresses the many other questions important to those who want to improve existing programs or create new social initiatives.
This is an important consideration, because the best approach usually isn’t to replicate a whole program. Instead, local innovation is usually more beneficial. For example, consider an existing organization in a small city that helps low-income older adults live longer lives at home. Rather than dismantle their existing program and attempt to implement a program originating in a large urban area, they are more likely to innovate a new model of care that meets the unique needs of their community. For a successful innovation, they would need to consider multiple services and practices, the organizations and people involved, the operating environment, and what it takes to support lasting change in people’s lives. For that, they need to draw on evidence from many types of studies and sources.
One resource for assessing any type of study is the transdisciplinary research quality assessment framework, which considers many aspects of study quality, such as whether the researchers explicitly account for limitations, whether the findings are transferable to other contexts, and whether the researchers disclose their funding sources and other potential biases. Other resources provide guidance for assessing evidence from specific types of studies. This che
cklist, for example, focuses on assessing the strengths and weaknesses of a systematic review—a study that finds and synthesizes information from multiple studies as they relate to the questions at hand.