STScI Logo

Hubble Space Telescope

Cycle 21 Peer Review Support Guidelines

Panel Support Orientation from March 21, 2013 Meeting

FileMaker Pro and Web-Reviewer Training from April 17, 2013 Meeting

Panel Support Checklist for Mock and Review Meetings: May 06, 2013

INS Contacts May 06, 2013

Canon Secure Print Instructions
for those Panels meeting in the Muller Building

Cycle 21 Peer Review Agenda

Frequently Asked Technical Questions During the Review

I will post any questions that come up in the Chat session that everyone might find interesting, but you won't have to scroll back and forth to find the answer in the iChat Session.

Introduction

The Hubble Space Telescope science peer review process has changed considerably in recent years so even if you have served previously as a support staff, please review this website carefully. The Panel members have their own website that will also be useful for your review, http://www.stsci.edu/hst/proposing/panel/. To minimize duplication, this website describes only those issues unique to the panel support duties.

The majority of the Cycle 21 science program will be recommended by a set of fourteen different Review Panels. These deal with the Small (1-34 orbits) and Medium (35-74 orbits) size GO proposals, SNAP proposals, AR and Theory proposals. The Chairs of the panels together with an overall Chair, and several at-large members form the Telescope Allocation Committee (TAC). The TAC reviews the recommended Medium proposals from the panels, the Large GO proposals (75 or more orbits), as well as the Treasury, and AR Legacy Programs. It also resolves any cross-panel issues. The STScI Director is responsible for the final allocation of HST observing time and funding.

Confidentiality and Conflicts of Interest

The proprietary nature of the proposals and related documents is taken extremely seriously by STScI. Proposals are the intellectual property of the PIs, and the sensitivity of the process is enhanced by the heavy over-subscription of HST time and the significant funding implications for US proposers. Review-related materials should not be left in the open when not in use, and should be locked away overnight or when otherwise inactive.

Panel membership is confidential except within a panel, until after the review. It is critically important that conflicts of interest are avoided during the selection process. See the Panel Member website for their conflicts of interest.

As a panel support member you have a conflict of interest with a proposal only if one of the following occurs: You leave the room for discussion, grading, voting and ranking if that proposal is near the cutoff. All HST Phase I proposals and documents related to the proposal review are strictly confidential.

Schedule

The Space Telescope Science Institute (STScI) will conduct the HST Cycle 21 Proposal Review May 13 - 18, 2013. The Review meetings will take place at the Space Telescope Science Institute and the Johns Hopkins University Bloomberg Building. Fourteen Review Panels will meet for 2 1/2 days each, and the overall TAC will meet for 2 days, immediately after the panel meetings.

Role of the Panel Support Staff (PSS)

The PSSs are intermediaries between the external reviewers and STScI. Their main tasks are to answer questions, to ensure that the panel meetings proceed smoothly, to record the results of the panel discussions, and to bring any critical issues promptly to the attention of Science Policies Group (SPG) and/or relevant STScI technical staff. The PSSs do not take part in the scientific discussions of the panel and do not vote, so that they have no conflicts of interest other than direct participation in a proposal.

The PSS performs a vital task. They represent STScI and provide continuity through their insight into the operations of HST, STScI and the review process. It is the PSSs who minimize the number of issues that 'slip through the cracks' and cause problems later on in the process. Their participation increases the quality of the process, which depends substantially on their scientific and technical expertise, commitment, and integrity.

Reviewer Assignments

Proposals and panelists are assigned to specific panels by SPG, and each proposal is assigned one primary reviewer and one secondary reviewer (using software to minimize conflicts of interest). Reviewers will be assigned by specific scientific subfield expertise. The responsibilities of these reviewers are described on their webpage.

Conflicts of interest for panelists are also defined on their webpage. The software finds (and avoids) only conflicts that can be identified by the name or the institutional affiliation of a reviewer; more subtle conflicts will be missed (e.g., ties as a close collaborator). Reviewers who feel that they have a conflict of interest on a proposal to which they have been assigned as a primary or secondary reviewer are instructed to contact their PSS. In such cases the PSS should reassign the proposal to another reviewer. The new reviewer must not have a conflict (please check explicitly with the potential new reviewer), and have specific expertise in the scientific area of the proposal.

The PSS may make multiple reviewer reassignments (or perform swaps), if necessary to keep a similar number of assignments for each reviewer (the balance does not have to be exact). Any reviewer reassignments should be communicated by the PSS via email to the reviewers involved, with a cc to blacker@stsci.edu. The Panel Chair should also be informed about reassignments; the PSS should ascertain directly from the Chair whether they prefer to be updated with each reassignment or to receive the final assignment list after all swaps are made (just before the review).

Proposal Content

The PSS should read (at a minimum) each proposal's abstract and observation summary in order to acquire an overall familiarity with the proposal content of his/her panel, to assist the Chairs in organizing the material and running the meetings, and to flag duplication, resource, and feasibility issues for further review by STScI specialists as appropriate (see below).

In Phase I there will not be a systematic technical review of all proposals, nor will there be a systematic computer-aided target/exposure duplication review. Such reviews are only done in Phase II for proposals that have been approved in Phase I. Nonetheless, both the PSS and the panelists should be alert to any such issues in Phase I, either before or during the panel meetings. Any such issues that are noted should be recorded in the database, and if appropriate, attempts must be made to resolve them. Cross-panel duplications will be dealt with only after preliminary rankings, and should be flagged by the Chairs. Any duplications (as defined in Section 5.2.1 of the Cycle 21 Call for Proposals) that are not specifically noted in the 'notification comments' of the database as being approved by the panel will be disallowed in Phase II.

If a PSS has a technical or scientific question(s) about a proposal, (e.g., proposed observations of Mercury, use of filters that don't exist on a given instrument, etc.), you should direct any questions about the scientific or technical feasibility of a proposal, to Jennifer Mack: mack@stsci.edu. In the subject field write "TAC question for SI", where SI should be replaced by the name of the Scientific Instrument (ACS/COS/FGS/STIS/WFC3) to which the question pertains.

If the question is not about a specific instrument but about scheduling or Policies, then please email Brett Blacker: blacker@stsci.edu. In either case, the body of the email should contain the proposal ID number, the name of the PI, and the question itself. We will forward your question to an appropriate expert and provide a response usually within two business days.

A written record of any noted technical feasibility issues must be kept for each proposal. These should be entered into the 'Technical Comments' section of the electronic database. These comments must be augmented with any resolutions (or lack thereof) that emerge from the panel deliberations.

Review Meetings

During the meetings, the PSS functions as an executive assistant to the Chair. They organize the materials, provide input on any technical issues noted before the meeting, coordinate real-time consultation with STScI specialists, record the grade votes and orbit allocations, operate the proposal database, and monitor the reviewer comments. More specifically, the principal activities during the meetings are as follows.
  1. Assist the Chair in the organization of the materials and discussion, and provide any other assistance requested by the Chair.
  2. Coordinate requests for consultations with STScI experts. A list of contacts and phone numbers for on-call experts will be available in each meeting room. A STScI specialist should be consulted on every significant feasibility or policy issue, and no proposal may be rejected on feasibility or policy grounds without such consultation. The reason for this is that we have had problems in prior cycles with peer reviewers making their own judgments that later turned out to be wrong. Any consultation must be recorded as a comment in the database.
  3. The PSS should keep careful notes on any controversies, conflicts, or other exceptional issues, and the resolutions of them recommended by the panels, for subsequent consideration during the Director's Review. These notes should be of a quality such that they would be intelligible to someone else (e.g., SPG staff responding months later to appeals from rejected proposers). An example of such information is keeping track of who leaves the room for each proposal, due to a conflict of interest. Another example would be noting potentially prejudicial or extra-scientific remarks about proposers.
  4. Although it may be difficult, the PSS should completely refrain from making any scientific, technical, scheduling or policy comments, even if asked to do so by the Chair.
  5. Monitor comments (see below) progress via the Web-based system. Ensure that all comments are finished and signed off by the primary reviewers before they leave the meeting. Ensure that Panel Chairs have been provided with that set of comments. Proof-read comments as time permits.
  6. The voting procedure is that the panel first agrees on a specific allocation to vote for; then, on this basis, each member gives a numerical grade to the proposal in the Web-based Review System. The PSS must enter the orbit allocations and the final grade into the electronic database. The importance of care and accuracy in this activity cannot be overemphasized, since it is the only record of the panel's intent.
  7. After the voting stage follows a ranking stage. The final product of each panel will be a ranked list for all GO, SNAP, AR and Theory proposals. Ranks should be entered into the database as agreed upon by the panel and as integers only. Grades should not be changed during the ranking stage.

Proposal Comments

Before the end of the TAC meeting on Saturday, the PSSs will be asked to proof-read and edit the proposal comments to make them suitable for dissemination to the proposers. We will ask the Chairs to proof-read the final comments as well. Use the following guidelines:
  1. All rejection letters should have a clear statement on why the proposal was rejected.
  2. Correct any obvious scientific or technical errors. However, PSSs must not add qualitatively new information.
  3. Correct grammatical and typographical errors.
  4. Eliminate all offensive, gratuitous or personal remarks.
  5. Remove any explicit identifications of other Cycle 21 proposals or proposers, reviewers, or STScI staff. References to specific GTO or prior-cycle GO approved programs are allowed.
  6. Only those comments that contain substantive, useful technical or scientific information should be sent. Comments that contain only insubstantial or superficial remarks should be removed, whether for approved or unapproved proposals (since they create an erroneous impression of the review itself).
  7. Remove remarks that do not make sense because the reviewers expected the proposal to be approved, while in the end it was not. However, it is okay if a rejected proposal has positive remarks that are useful; that possibility is covered in the notification letter as a consequence of the heavy over-subscription.
  8. Make sure you verify in the proposals, any references to Proposals that are over the page limits and correct comments as appropriate.
  9. Consult with SPG on any questionable cases.
  10. Inform SPG when a complete set of the final, edited comments are available.

Specific PSS Duties

Technical Information

As a valuable member of the Cycle 21 Peer Review process, you are a critical part of the system and we value your comments and suggestions, and welcome feedback throughout the process.

This page is maintained by Brett S. Blacker
Last updated: May 09, 2013.