NIH CSR Advisory Council Meeting Provides Peer Review Updates

By: Kamille Rasche and Naomi Charalambakis
Thursday, September 28, 2023
During its September 18 meeting, the Center for Scientific Review (CSR) Advisory Council shared several updates related to proposed changes to the National Institutes of Health (NIH) peer review criteria. Since 2020, CSR has been instrumental in facilitating discussions about ways to strengthen the peer review process that further emphasizes the scientific merit of proposals while reducing reviewer bias and administrative burden.

First, CSR presented the status and implementation timeline for proposed changes to peer review criteria for Research Project Grants (RPGs). The simplified peer review framework will reorganize the existing five review criteria into three factors: 1) Importance of the Research; 2) Rigor and Feasibility; and 3) Expertise and Resources. While only the first two factors will be scored, the third factor will still be considered in the overall impact score, prompting reviewers to use a drop-down menu to indicate whether the proposal sufficiently addresses these aspects (“appropriate”) or contains gaps (“gaps identified”).

After soliciting comments from the research community through a Request for Information (RFI), a trans-NIH implementation committee has been engaged in various aspects of the roll-out process, including finalizing the policy language, facilitating communication efforts, and developing training resources. CSR anticipates publishing a guide notice this fall notifying the public of the final language and its effective date. Currently, CSR expects all RPG applications (R01, R21, etc.) submitted for January 25, 2025, due dates to abide by the new peer review framework, emphasizing that a significant amount of training and outreach to applicants, reviewers, and NIH staff will occur in the interim.

Similar to updating RPG peer review criteria, CSR is finalizing a new peer review framework for National Research Service Award (NRSA) fellowship applications to ensure the review process is fair and equitable for all applicants. The proposed changes are intended to better focus reviewer attention on key aspects relevant to training and reduce inappropriate consideration of sponsor and institutional reputation. For example, the revised framework reduces the current five criteria to three factors by eliminating “Sponsor” and “Institutional Environment” as distinct measures. Additional changes include removing the requirement to submit grades and placing more emphasis on the training plan and preparedness rather than the sponsor's track record.

To inform the updated fellowship review framework, CSR posted a content analysis report of the feedback received from the associated RFI. CSR received 780 individual responses along with 30 from scientific societies and 30 from academic institutions. The report shows an overall positive response to the proposed changes with a push toward limiting reviewer biases and providing clearer instructions for applicants and reviewers. There was strong support for the three-factor structure, though there were conflicting opinions on not giving factor three (“Expertise and Resources”) a numeric score. This section is centered on the sponsor and trainee background, and it was proposed that keeping these topics unscored could prevent institutional biases. However, since this factor would still weigh on the overall score, some respondents worried it may allow biases to have a larger impact than if it were numerically scored along with the other factors. This divide in opinion was seen in both the individual respondents as well as in the societal and institutional responses.

Together, the responses outlined in the report fall closely in line with FASEB’s previous comments and concerns on the proposed changes. Specifically, other respondents also acknowledged ambiguity in the instructions for both reviewers and applicants, supporting a careful training process to ensure clarity. Both FASEB and other respondents supported training sessions for reviewers as well as a clearly written rubric with examples to improve transparency of each factor and final scores.

Implementation of the new fellowship review framework will be analogous to the RPG changes. CSR anticipates publishing a guide notice later this fall announcing the final language and expects the first NRSA submissions under the new framework to occur in 2025.  To ensure applicants and reviewers are aware and prepared for the changes, NIH plans to conduct extensive training and outreach through various formats, including webinars and the creation of new resources.