Tracking and Evaluation Publications, Reports, Analyses and Presentations


  • Breaking down silos: mapping growth of cross-disciplinary collaboration in a translational science initiative.  PMCID: PMC4405428
  • Academic cross-pollination: The role of disciplinary affiliation in research collaboration.  PMCID: PMC4711942

2015 ICTS Stakeholder Interviews

In collaboration with the T&E Team, the Brown School Evaluation Center at Washington University conducted a qualitative evaluation of the ICTS in October 2015.  Twenty-five in-person interviews were conducted with key ICTS stakeholders, including KL2 graduates, division directors, department chairs, team science specialists, representatives from ICTS partner institutions, and ICTS directors.  The resulting confidential report summarized the strengths and challenges of the ICTS as described by the stakeholders as well as suggested strategies for the future, and was used to inform the application for the 2016 CTSA program.

2015 Member Engagement AnalysisMember Engagement

With nine years of past ICTS data, the T&E team is able to examine in much greater depth what the return on NCATS investment is in terms of scientific collaborations and productivity.  ICTS members can engage with ICTS in four main ways: use of core services, receipt of internal funding, participation as CRTC mentor or mentee, and leadership participation (Executive Committee, Operations Committee, or Core Directorship).  The T&E team is currently constructing a comprehensive dataset that combines information on all types of member engagements as well as productivity (publications, grant submissions, and awards) across the lifespan of the ICTS.  Preliminary analyses show that engaged ICTS members are submitting new grants at a much higher rate than non-engagement members.

2015 Collaboration Analysis

Overall, ICTS member collaborations on grants and publications are growing, as are interdisciplinary collaborations. For example, this figure and table show that the average degree of connection between members grew rapidly in the first 3 yearsMember Engagement

of the ICTS and has continued to grow even as our membership grows. The 744 ICTS members who were listed as Key Personnel on one or more NIH grant submissions in 2014 were listed, on average, with 7.3 other ICTS members that year.  Measures of modularity  showed increased between-discipline collaborations in relation to within-discipline collaborations over time in these data. These data demonstrate increases in the number, degree, and interdisciplinary nature in this basic measure of research collaboration (did researchers submit grants together?) Similar, though lagging trends are seen with grant awards and publications.

2014 Service User Survey

In five years of surveying the ICTS membership, investigators continue to report  the impact the ICTS and provided services has had on their scientific output. ICTS services contribute to their ability to conduct translational research, publish a manuscript and/or obtain additional funding. Responses to the Annual ICTS Member Survey not only provide feedback about personal satisfaction with services received, but also indicate the impact of the program.

Collaboration Survey Reports

To assess collaborative research among ICTS investigators, the Tracking & Evaluation Program developed the ICTS Research Collaboration Survey.  Distributed to all ICTS members in both 2011 and 2013, the 2013 report documents the comparison of responses for those two surveys.  (Also see the 2011 Executive Summary or Full Report)

2013 Community Survey Report

T&E is excited to report back to ICTS, Siteman Cancer Center and the Institute for Public Health researchers who responded to the survey about their collaborations with community researchers.  The report linked below summarizes the responses and the information shared by the identified community partners.  In short, approximately 4% of the investigators reported community research partnerships and 86 community organizations were identified. 2013 Community-Engaged Survey Report

2013 VIVO Conference Workshop

The ICTS Tracking &Evaluation team described development of strategic aims, implementation of tools for data collection and analyses, and ways to weave the resulting information into a description of impact.  As all CTSAs wrestle with exactly how to demonstrate program impact at multiple levels, the workshop attendees shared ideas and discussed options.  For more specifics, see the workshop PowerPoint slides.

2012 Social Network Analysis Presentation

By evaluating grant applications and publications by ICTS members, the ICTS Tracking & Evaluation team revealed that the average number of collaborations per member for both metrics increased over the first three years of the CTSA award.  To illustrate their findings, members of the team presented a poster entitled "Using Social Network Analysis to Evaluate Changes in Interdisciplinary Collaboration during a Clinical and Translational Science Award" at the April, 2012 Science of Team Science meeting.