Created by on 02/14/2012

The National Science Board (NSB) recently released a report on the National Science Foundation’s (NSF’s) Merit Review Criteria, which includes revisions to the criteria by which peer reviewers are instructed to evaluate the merit of funding applications received by the agency. Reviewers are asked to evaluate all proposals against two standards: 1) intellectual merit, which encompasses the potential to advance knowledge; and 2) broader impacts, which encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes. The criteria themselves are not new; the NSB did, however, work to define them more clearly with the hope of providing the NSF community with a better understanding of each criterion and how they relate to one another.

The report states that the following elements should be considered in the review for each proposal:
  • the potential for the proposed activities to advance knowledge and understanding within its own field or across different fields, and to benefit society or advance desired societal outcomes
  • the extent to which the proposed activities suggest and explore creative, original, or potentially transformative concepts
  • whether the proposed plan is well-reasoned, well-organized, based on a sound rationale, and incorporates a mechanism to assess success
  • the qualifications of the individual, team, or institution
  • whether there are adequate resources available to investigators to carry out the proposed activities.
Applicants may satisfy the broader impacts criterion through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by, but complementary to, the project. The assessment and evaluation of projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects. For example, the NSB notes that if the size of the activity is limited, evaluation of that activity in isolation is not likely to be meaningful. Assessment, therefore, may best be done at a higher, more aggregated level than the individual project. The report states that NSF should provide guidance on when project level assessment would be appropriate, what broader impacts data are important for future assessment purposes, and when assessment at a program or institutional level would be more reasonable than individual project evaluations.