Altum recently hosted an insightful webinar titled “Unmasking Bias in Peer Review and Grant Funding.” We explored the pervasive issue of bias within the peer review process and its impact on grant funding. This event brought together experts:
- Jenna Hicks, Assistant Director at the Health Research Alliance
- Anne Hultgren, PhD, Executive Director, Arnold and Mabel Beckman Foundation
- Alison J. Hatt, PhD, Independent Consultant for the Advanced Light Source
The discussion focused on identifying different types of bias and presenting evidence-based strategies to mitigate these biases to ensure a fair and effective review process.
Understanding Bias in Peer Review
We kicked off the webinar with a detailed overview of bias, highlighting its various forms and the impact it has on decision-making. Biases are preferences, assumptions, beliefs, attitudes, and stereotypes people have towards or against particular characteristics, individuals, or groups. They can be conscious (explicit) or unconscious (implicit), and they significantly influence the fairness of the peer review process.
Biases are often triggered by cognitive shortcuts used to process information quickly, especially under high-stress situations. These biases can manifest in various ways, including gender, age, race, and institutional bias, affecting the evaluation of research proposals and grant applications.
Case Study: Blinding to Reduce Bias
Anne Hultgren, PhD, shared the results of a study conducted by the Arnold and Mabel Beckman Foundation. They implemented a blinding process in their peer review system to address institutional bias. Traditionally, the foundation’s review process was unblinded, meaning reviewers knew the identities of applicants. However, they noticed a recurring pattern of awarding grants to the same prestigious institutions year after year.
To mitigate this bias, the foundation introduced a blinding step where only the technical information of proposals was reviewed, without revealing the applicants’ identities. The results were significant. The probability of proposals from top institutions being advanced dropped, while proposals from less prestigious institutions saw a more equitable consideration. This change allowed for a more diverse range of applicants to receive funding. It aligned with Dr. Arnold Beckman’s vision of supporting young scientists without the clout to receive major grants.
Dual Anonymous Peer Review: A Broader Perspective
Alison J. Hatt, PhD, expanded on the theme of reducing bias by advocating for dual anonymous peer review, also known as double-blind review. This approach ensures that both the reviewers and the applicants remain anonymous to each other. Hatt cited the example of the Hubble Space Telescope, which implemented dual anonymous peer review in 2018. The transition led to a significant reduction in gender bias and an increase in the success rates of first-time principal investigators (PIs) and researchers from less prestigious institutions.
Hatt also highlighted the importance of improving the statistical integrity of proposal evaluations. This includes providing detailed rubrics to reviewers, using coarse-grained scoring scales, and offering training to ensure consistency and accuracy in scoring.
Additionally, she proposed the use of partial lotteries to rank closely or identically scored proposals, reducing the subjectivity and bias inherent in the final ranking process.
Key Takeaways and Recommendations
The webinar concluded with practical recommendations for organizations looking to address bias in their peer review processes:
- Implement Blinding: Introduce blinding steps in the review process to ensure that technical merit is evaluated without the influence of applicants’ identities.
- Adopt Dual Anonymous Review: Move towards dual anonymous peer review to minimize identity-based biases and ensure a fair evaluation of proposals.
- Provide Detailed Rubrics: Use detailed scoring rubrics with multiple criteria to guide reviewers and improve the precision and accuracy of scores.
- Offer Training: Provide training videos and resources to educate reviewers on applying rubrics and recognizing their own biases.
- Use Partial Lotteries: For closely scored proposals, use partial lotteries to assign ranks, ensuring a fair and bias-free selection process.
Altum remains committed to revolutionizing the research grant life cycle by fostering fair and equitable peer review processes. We thank our speakers and participants for their valuable insights and contributions to this important discussion.
For those who couldn’t attend, the recording of the webinar is available on-demand. Stay tuned for more updates and resources from Altum as we continue to address critical issues in grant funding and research evaluation.
