Challenge of measuring impact
Government, non-profits, and organizations fund research with the ultimate goal of improving health and society. Yet measuring these long-term outcomes can be challenging, and research impact has traditionally been tied to quantitative productivity, like bibliometrics and grant funding, and not necessarily the broader impacts of research. For years, researchers lacked a cohesive framework to measure their work in terms of lives saved, improvements to health, and cost savings.
To address these challenges and to speed the effort for a more ubiquitous measurement tool, the Institute of Clinical and Translational Sciences at Washington University in St. Louis created the Translational Science Benefits Model (TSBM) as a framework public health and clinical scientists can use to demonstrate the impact of their work in the real world. The framework was developed by a cross-disciplinary team including members of the Institute of Clinical and Translational Sciences (ICTS), Bernard Becker Medical Library at Washington University School of Medicine, and the Center for Public Health Systems Science at the Brown School at Washington University. By looking beyond quantitative counts of scientific productivity and visibility, the TSBM utilizes broader measures of real world benefits in four distinct domains: clinical, community, economic, and policy.
Douglas Luke, PhD, the Irving Louis Horowitz Professor in Social Policy at the Brown School at Washington University, director of CPHSS and ICTS Evaluation and Continuous Improvement Function lead, explains the landscape of evaluation that generated the TSBM. “There was a recognition that we needed to move beyond widget counting, which implies evaluation that focuses exclusively on simple counting of research publications and grants. We wanted to emphasize evaluation of how research actually benefits people. It would provide a framework for the whole reason we are doing this work in the first place; to enhance clinical and health sciences, but also to help move the results of science into the community, and embed translational science into healthcare systems, population health, and public health policy.”
The TSBM was first introduced to the broader scientific community in the September 8, 2017 issue of Clinical and Translational Science, “The Translational Science Benefits Model: A New Framework for Assessing the Health and Societal Benefits of Clinical and Translational Sciences”. Subsequent promotion by the Washington University TSBM team has increased the awareness of the model, particularly among the hub institutions associated with the NCATS Clinical and Translational Science Awards (CTSA) Program.
The TSBM in Action
From its inception, the TSBM was developed to be an active, working model, easily sharable and adaptable by other evaluators, translational scientists, and academic institutions. That intention is now becoming practice, notably among the CTSA community.
To disseminate the model, the TSBM team created a website to share how to use the model and feature case studies applying the TSBM to real-world research projects. Incorporated within the website are user-friendly tools like a Benefits Checklist that shows researchers how to track the impact of their own research over time and a Case Study Builder form, a fillable PDF to help researchers tell their own impact stories in easy-to-read, lay person language.
Word of mouth has also been essential to promoting the TSBM. Luke and the TSBM team have promoted the model via presentations, even going virtual with those efforts in light of the COVID pandemic. Through the CTSA Program Evaluators group, hubs have shared their unique experiences using the TSBM from incorporating it into funding programs to reforming how investigators and cores report on their work.
The TSBM team is also actively promoting how other CTSAs are using the model by showcasing their case studies on the TSBM website. Researchers at the Oregon Clinical and Translational Research Institute (OCTRI), a CTSA program hub institution, were early adopters of the model, utilizing the TSBM to create case studies on existing clinical research. These two real-world clinical projects are summarized in “SOLACE: An Early Warning System for Adolescent Suicidality” and “Reducing Diabetes Risk Factors”, currently featured on the TSBM website.
Adrienne Zell, PhD, OCTRI Assistant Director and Director of the Evaluation Core, recalls how they started using the TSBM. “We initially approached the Washington University team with interest in the case study format. Prior to the TSBM, OCTRI would summarize research with anecdotal stories. The model provided us with a framework to better organize these stories into case studies and to really think about what the ongoing benefits would be.”
The use of the case studies also made a significant impact on OCTRI’s community core. According to Zell, the community core is now using the model prospectively by looking at the indicators related to their research and then tracking them over time to see if the outcomes are realized. And OCTRI is currently expanding use of the model by including the TSBM indicators in their annual survey to funding awardees.
In tandem with OCTRI’s use of the model, interest has grown across the CTSA hub community. So much so that in November 2020, the CTSA Program Evaluators Group focused their meeting on sharing examples of how other CTSAs are currently using the TSBM.
Joseph McClernon, PhD, Director of Evaluation and Strategic Planning for the Duke Clinical and Translational Science Institute (CTSI), was one of five CTSA hubs that presented at the meeting. McClernon’s interest in the model was first piqued by the 2017 article in Clinical and Translational Science. Then, after a presentation by Luke and the TSBM team, they embraced the model and went to work on ways to use it in their evaluation efforts.
In 2020, the Duke CTSI asked applicants to the Multidisciplinary Vision Program (MVP) award to frame their translation plan around the TSBM. Applicants were asked to identify up to five benefit indicators where they most intend to have an impact. CTSI then mapped these indicators across the four primary TSBM domains. Ultimately, CTSI plans to have awardees revisit these indicators as part of their progress reporting to evaluate how close they came to making the impact they intentioned.
The model continues to gain a lot of traction with the Duke CTSI and is now a required translation plan element for all hub funding programs. In addition to the pilot award application and tracking process, CTSI is currently working on developing case studies using the TSBM methodology.
“A key attraction to utilizing the model is the work that the TSBM team has done to make it accessible,” says McClernon. “Everything from the journal article to the graphics, tools and templates and the accessibility of the team at WashU, made it easy for us to apply the TSBM to our needs here at Duke.”
The future of the TSBM
With the TSBM going viral among the CTSA community, more plans to share examples and experiences are underway. Interest from the evaluators’ group meeting has generated an organized TSBM working group within the Center for Leading Innovation and Collaboration (CLIC), the coordinating center for the CTSA program. This working group, called the Translational Sciences Benefits Model Research Group, plans to share hub experiences with the TSBM in hopes of broader adoption among CTSAs. Interest is strong with over 30 CTSA hubs in attendance at the group’s first meeting in May, 2021.
And, in anticipation of this increased interest, the TSBM team at Washington University continues to develop tools to assist others in adapting the model to their research. They are currently developing a Translating for Impact Toolkit, which will include a full suite of tools to help researchers plan, track and demonstrate the benefits of their research.
By using the TSBM indicators, CTSAs are able to truly understand their portfolio of research and the longer-term impacts of that researchJoseph McClernon, PhD, Director of Evaluation and Strategic Planning for the Duke Clinical and Translational Science Institute
Reflecting on the TSBM, McClernon summarizes, “Doug Luke and his team have hit on something that people in evaluation want; that is a framework for assessing, demonstrating and communicating impact. By using the TSBM indicators, CTSAs are able to truly understand their portfolio of research and the longer-term impacts of that research.”