Dual-Anonymous Proposal WorkshopN. Reid (inr[at]stsci.edu)
In 2018, STScI implemented a dual-anonymous proposal system for the HST proposal review process. Under this system, not only are proposers unaware of the identity of the members on the review panel, but the reviewers do not have explicit knowledge of the proposal teams. The goal is to mitigate unconscious bias and allow reviewers to concentrate on the science rather than the scientist. Following the successful implementation in HST cycles 26 and 27, Paul Hertz, the Director of the NASA Astrophysics Division, directed all Astrophysics GO Programs to implement this process. As part of that process, STScI hosted a workshop to pass on the experiences in developing and implementing the process.
The workshop was held on 25th September 2019 and attended by representatives not only from NASA's GO programs, but also from a number of ground-based observatories. Stephanie Johnson (Leeds Business School, Colorado) provided an opening discussion of unconscious bias in review processes and mechanisms that can mitigate their impact. There were presentations on the HST review process, the steps taken to prepare the Hubble community for the implementation of dual anonymous reviews and the most recent results, while Arvind Parmer summarized analysis of the results from ESA's AO reviews. Jessica Kirk (Memphis University), who, with Professor Johnson, served as an external adviser, gave her perspective from observing several Telescope Allocation Committee (TAC) reviews. In particular, she noted that more than half of the in-panel discussions included references to team members prior to the implementation of the dual anonymous requirements.
Besides the presentations, the workshop included panel discussions led by representatives from the community, panel support staff and TAC members. Christina Ritchey (JPL) and Brian Williams (GSFC), on the first panel, were members of the Dual Anonymous Working Group and described the initial community polling for feedback on the concept. On the same panel, the Chair of the Space Telescope Users Committee (STUC), Rupali Chandar (U. Toledo), outlined the STUC's involvement in developing and refining the process. Those discussions led to a key addition to the review process, revealing the investigators after the ranking is finalized to enable verification that the team has the appropriate expertise.
Three STScI staff members, Svea Hernandez, Crystal Mannfolk and Greg Snyder, described their experiences as panel support scientists (PSS) and levelers at the HST Cycle 27 TAC review. They highlighted the importance of setting clear procedures from the outset, as well as establishing the roles and responsibilities, particularly for the levelers, charged with monitoring the panel discussions for potential breaches of anonymity. It is important to have good communications between the PSS, the leveler, and the panel chair that should be developed through pre-meeting practice sessions.
Priya Natarajan (Yale), Saurabh Jha (Rutgers) and Rupali Chandar provided their perspectives as reviewers on the Cycle 26 and 27 TACs. All three noted that reviews generally went smoothly. Indeed, the overall level of discussion was much more collegial and focused than many of their previous experiences. The team expertise description did provide a safety net to check for specific information, such as access to ground-based facilities, but in many (most) cases, the proposal itself was sufficient to give confidence in the team's ability to complete the project successfully. Finally, Molly Peeples (STScI) summarized some of the lessons we have learned from the initial TACs.
Following the talks, the participants split into breakout sessions to consider the potential challenges and advantages of applying the dual-anonymous approach to their time-allocation processes. Reconvening, the meeting came to consensus agreement in several areas:
- All observatories and missions agreed that there are no insuperable obstacles to implementing the dual-anonymous process for their reviews.
- There were concerns regarding the resources needed, notably the addition of levelers. This was a particular concern for the smaller NASA missions.
- There was some discussion of providing a central source for coordinating information on reviewer conflicts.
- There were suggestions for possible adjustments to the process, perhaps incorporating some measure of the past productivity of the proposal teams and giving additional considerations to proposals from student PIs.
These consensus conclusions were provided as feedback to Paul Hertz at NASA. The workshop participants also highlighted the importance of publicizing the process beyond the HST community. NASA has since hosted a number of community discussions, including presentations at the 2020 winter AAS meeting and on-line webinars discussing how the process is being applied to other missions.
Further information on the workshop content, including links to the presentation, is collected on the STScI Outerspace site.