Hubble Cycle 26 TAC and Anonymous Peer ReviewN. Reid (inr[at]stsci.edu)
Astronomers gathered in Baltimore recently to complete the annual Telescope Allocation Review for the Hubble Space Telescope. This year's Telescope Allocation Committee (TAC) differed in three important respects from previous years: first, the schedule is later by four months; second, the review focused on Medium and Large observing proposals and Archive Legacy programs; and, finally, all of the proposals were reviewed through a dual anonymous system. In this system, not only are proposers unaware of the identity of the reviewers, but the reviewers are also not made aware of the identities of investigators. To the best of our knowledge, this is the first occasion where a fully anonymized process has been employed in a large-scale proposal review in the physical sciences.
The timing and limited focus of this year’s TAC are "one-offs," triggered by the original Webb Cycle 1 GO review, which is now postponed. In contrast, the adoption of dual anonymous review is the culmination of several years of analyses that show consistently higher success rates for proposals led by male PIs. The offset persists despite changes in the proposal format designed to lower PI visibility in the review. The obvious concern is that these results reflect unconscious bias in the review process—bias that might extend beyond the simple male/female paradigm measured by our statistical analysis. The female PI success rate serves as the canary in the coal mine.
Following discussions with the Space Telescope Users Committee (STUC) and community members, the Institute Director brought in Professor Stefanie Johnson of the Leeds Business School, University of Colorado as a consultant. She and her graduate student, Jessica Kirk, sat in as observers in the Cycle 25 TAC review. Their analysis showed no discernible gender bias in the preliminary grades submitted prior to the TAC meeting, but the final results show the same offset as previous cycles. Reporting to the STUC in October 2017, they noted that almost 50% of the panel discussions included a focus on personnel that detracted attention from the proposed science; they recommended adopting a fully anonymous review process.
Following these discussions, the Director commissioned a Working Group (WG) to explore options, to consult with the community, and to develop guidelines for reviewers and proposers. The WG presented their report (https://outerspace.stsci.edu/display/APRWG) to the STUC at their April 2018 meeting, recommending implementation of a dual anonymous process for the scientific review of Hubble proposals. Specific recommendations included:
- The addition of "levelers" to the panel discussions, who are empowered to step in if a proposal discussion veered away from focusing on the science.
- In response to the key concern raised by the community, adding a (non-anonymized) "Team Expertise and Background" section, to be made available after the final ranking, to allow the TAC to assess whether the proposal team would be capable of executing the proposed investigation. If there were clear, sufficient deficiencies, the panel could flag a proposal for disqualification; panels would not have the option of recommending replacement proposals.
The WG guidelines on how to adapt proposals for anonymous review emphasize that the goal is not to actively conceal relevant research, but to allow the reviewers to focus on the science rather than the scientist. The WG report was endorsed by the STUC and the Director took the decision to implement the dual anonymous process for the Cycle 26 ΔTAC.
The ΔTAC was structured with four multi-disciplinary panels covering the Solar System, exoplanet and disks; stellar physics and stellar populations; AGN and IGM; and galaxies and cosmology. The panels review and rank all proposals in those science areas over the first two days of the TAC meeting, and the medium proposals are selected by the panels. The co-chair and chair from each panel form a merging super-TAC to select the large/treasury/legacy proposals on the final day.
The Cycle 26 ΔCall was issued in May 2018 with a proposal deadline of August 17. The response from the community was enthusiastic, perhaps partly stimulated by the elimination of the Webb Call: we received a total of 489 proposals, including 50 Legacy AR proposals, 335 Medium, 75 Large, and 21 Treasury GO proposals for a total of 25,775 orbits, an over-subscription of ~12:1. This represented a significant increase over expectations for each panel, and additional panelists were recruited to mitigate, as far as possible, the individual workload. Even more than usual, we owe a substantial debt of gratitude to the community members who gave their time to select Hubble's observing program.
The TAC, chaired by Professor Priya Natarajan (Yale University), met at the Institute from October 9–11. The orientation session, attended by observers from NASA and several other facilities, included extensive discussion of the dual anonymous process. The subsequent review sessions were intense, given the highly competitive proposal pool, but centered on the science. The vast majority of proposals were closely compliant with the anonymizing guidelines; a few minor cases were flagged for comment by panelists, but only one proposal was sufficiently egregious to be eliminated from consideration. There were comments from several panelists that it was almost liberating to be able to focus on "what" without regard to "who." No concerns were raised in the final stage regarding the team expertise for any of the highly ranked proposals.
The forty proposals recommended by the TAC and selected by the Director span a wide range of topics. The selected programs include a multi-wavelength, multi-epoch Target of Opportunity program for follow-up observations gravitational wave optical counterparts discovered in LIGO's O3 run; optical and UV imaging of nearby star-forming galaxies being observed by ALMA in the PHANGS survey; UV imaging of a subset of the CANDELS fields; and three programs that will use different techniques (Cepheids, RGB tip stars and gravitational lensing) to constrain the cosmic distance scale and H0. Hubble will map gaseous emission from comet 46P/Wirtanen at ~20 km resolution and continue to support Juno's measurements of Jupiter's magnetosphere; it will search for planetary-mass companions to young stars in Taurus and Serpens, test how flare activity in Proxima Centauri might affect planetary habitability in that system and conduct a detailed investigation of mass outflows in Betelgeuse through combined multi-epoch UV spectroscopy and radio observations; and Hubble will continue to probe the multiple populations in globular clusters and map the tangential motions of galaxies in the Local Group.
Statistically, 138 of the 489 submitted proposals (28%) were led by female PIs—as a comparison, in Cycle 25, female PIs led 46 of 167 Medium and Large proposals (27.5%). Twelve of the 40 proposals selected for execution are led by female PIs, a success rate of 8.7% (12/138); for male PIs, the success rate is 8.0% (28/351). This reverses the trend seen in the past 15 cycles. Specifically, in Cycle 25, 13% (6/46) of Medium and Large proposals submitted by female PIs were approved while 24% (29/121) of proposals submitted by male PIs were successful. Given the special circumstances involved in the Cycle 26 ΔTAC it would be premature to draw broad conclusions, but the results are encouraging.
The Cycle 26 TAC process and the results were discussed with the STUC at their November 2018 meeting, including contributions from Professors Johnson and Natarajan. Prof. Johnson commented that the panel discussions that she observed had a very different flavor from Cycle 25 and that the panels largely "self-policed" in monitoring the discussion, with minimal intervention required from the levelers. Professor Natarajan concurred, noting that all the panel discussions she observed had a strong science focus. The STUC endorsed the continuation of the dual anonymous process for Hubble proposals, with a recommendation that the Institute continue to monitor the situation.
"I am grateful to all involved in proposing and recommending the Cycle 26 science program, which will continue Hubble's rich tradition of broadening and deepening our understanding of the universe. In implementing a fully anonymous proposal review, I believe we have taken a positive step forward in ensuring that scientific merit is the primary focus of the review process," concluded STScI's Director, Ken Sembach.
Looking forward, Hubble will return to the standard cadence in Cycle 27, with the Call issued on January 14, 2019, the Phase Ⅰ deadline on April 5, and the TAC meeting during the week of June 9–13. The Cycle 27 Call will include all types of proposals (Small, Medium, Large, Treasury, Archive, Snap) that will be reviewed by the same set of topical panels as in Cycle 25. Proposals will be reviewed under the dual anonymous system.