Overview and New Features

The basic sequence of steps in the STScI pipeline system (also known as OPUS) is:

1. Assemble data received from HST into datasets.
2. Perform a standard level of calibration of the science data.
3. Store both the uncalibrated and calibrated datasets in the Archive and populate the Archive database catalog to support StarView queries. The pipeline must also handle exceptions (e.g., incomplete data) and perform a general data evaluation and quality control step. Final delivery of data to observers is accomplished by the data distribution mechanisms of the Archive system.

   The calibration step has several goals:

• Remove the known instrumental signatures (e.g., flat field and dark -current).
• Correct linear and physical units (e.g., gain and flux calibration).
• Flag degraded or suspect data values and provide estimates of the -statistical uncertainties of each pixel. While a calibration pipeline may not be able to provide the optimal calibration for a specific observation (which may, in fact, not become available until some time after the data were obtained and calibrated), the goal is to provide data calibrated to a level suitable for initial evaluation and analysis for all users. Further, observers frequently require a detailed understanding of the calibrations applied to their data and the ability to repeat, often with improved calibration products, the calibration process at their home institution. To support this last goal, the calibration software is available within the STSDAS system and the calibration reference files (e.g., flat fields) are available from the HST Archive via StarView.

Concept

An illustrative (partial) counter example to this procedure is the WFPC2 CRSPLIT proposal instruction. This results in two WFPC2 exposures from a single line on the exposure logsheet (the way in which observers specify commands for HST). However, the HST ground system treats a CRSPLIT as two distinct exposures which are commanded, processed, calibrated, and archived separately. The pipeline does not combine these two images (datasets) to create the single image without cosmic ray events which was the observer's original intention. Presently, the observers (and any future archival researchers) are left to perform this task on their own.

The second generation instruments present many instances in which the combination of data from two or more exposures is necessary to create a scientifically useful data product. Both NICMOS and STIS will need to combine exposures to remove cosmic rays and to improve flat fielding (by dithering or stepping). For NICMOS, the HST thermal background is expected to have significant temporal variations. Multiple exposures (dithered for small targets and offset onto blank sky-chopped-for larger targets) will be necessary to measure and remove this background. While this has been standard practice for ground based infrared observations and is the basis of essentially all existing infrared data reduction schemes, it is a new paradigm for the HST ground system.

Usage

An association is a set of one or more exposures along with an association table and, optionally, one or more products. We define the following terms:

• An exposure is the atomic unit of HST data.
• A dataset is a collection of files having a common rootname (first 9 -characters).
• A product is a dataset derived from one or more exposures. The first generation instruments all have a one-to-one correspondence between exposures and datasets. They do not have products. NICMOS and STIS use the association structure as a meta-dataset. Further, they use the information in multiple exposures during the calibration process to create products.

  From a high level, an association is a means of identifying a set of exposures as belonging together and being, in some sense, dependent upon one another. The association concept permits these exposures to be calibrated, archived, retrieved, and reprocessed (within OPUS or STSDAS) as a set rather than as individual objects. In one sense, this is a book-keeping operation which is being transferred from the observer to the HST data pipeline and archive.

  Associations are defined by optional parameters on a single exposure logsheet line. That is, there is a one-to-one correspondence between proposal logsheet lines and associations (although it is possible to have exposures which are not in associations).

  Observers may obtain one or more exposures at each of one or more positions on the sky using the NICMOS proposal grammar. Typically usage will be:

• To obtain a sequence of slightly offset exposures (dithering) to improve the flat fielding, avoid bad pixels and cosmic rays, and, for sufficiently compact targets, to remove the background illumination.
• Mapping of targets larger than the NICMOS detector's field of view.
• To obtain a sequence of observations in which the telescope is chopped between the target and one or more offset regions of (hopefully blank) sky. A set of pre-defined patterns are provided in the proposal instructions for these types of observations or a combination of both types. The Institute ground system will expand the observer's single line request into multiple exposures each with its own identifying name (IPPPSSOOT) and populate the necessary database relations to describe this association for the OPUS system.