NICMOS Instrument Handbook for Cycle 11

Overview

---

All data taken with NICMOS are automatically processed and calibrated by a suite of software programs known as the pipeline. The purpose of pipeline processing is to provide data products to observers and the HST Data Archive in a form suitable for most scientific analyses. Pipeline processing is also applied to engineering data and calibration data.

The basic sequence of steps in the STScI pipeline system (also known as OPUS) is:

http://www.stsci.edu/software/OPUS/

1. Assemble data received from HST into datasets.
2. Perform a standard level of calibration of the science data.
3. Store both the uncalibrated and calibrated datasets in the Archive and populate the Archive database catalog to support StarView queries. After implementation of OTFR, information from the calibrated data will be catalogued, but the calibrated files will not be stored.

The pipeline must also handle exceptions (e.g., incomplete data) and perform a general data evaluation and quality control step. Final delivery of data to observers is accomplished by the data distribution mechanisms of the Archive system.

The calibration step has several goals:

• Remove the known instrumental signatures (e.g., flat field and dark current).
• Correct the data for non-linear behavior and convert to physical units (e.g., gain and flux calibration).
• Flag degraded or suspect data values and provide estimates of the statistical uncertainties of each pixel.

While a calibration pipeline may not be able to provide the optimal calibration for a specific observation (the best calibration files may, in fact, not become available until some time after the data were obtained and calibrated), the goal is to provide data calibrated to a level suitable for initial evaluation and analysis for all users. Observers frequently require a detailed understanding of the calibrations applied to their data and the ability to repeat, often with improved calibration products, the calibration process at their home institution. Further, certain types of image artifacts can appear in NICMOS data, which require processing with specialized tools to remove. To support these goals, the calibration software is available within the IRAF/STSDAS system and the calibration reference files (e.g., flat fields) are available from the HST Archive via StarView so that observers have the ability to repeat and customize the calibration processing to meet the specific needs of individual observations.

Associations

To improve the utility of the pipeline processing for the second generation science instruments (NICMOS, STIS) and future instruments (ACS, COS, WF3) several significant changes were made to the structure of the calibration pipeline. The largest of these changes was to enable the combination of multiple observations during the calibration process. This permits the pipeline to both generate a combined product and to use calibrations obtained contemporaneously with the science observations. This capability is designed to support the cosmic ray event removal, mosaicing, and background subtraction for NICMOS observations. As discussed in Chapter 11, mechanisms exist for compactly requesting such observations in the Phase II proposal.

Concept

The basic element in the HST ground system has historically been the exposure. The first generation HST science instruments were commanded to generate single exposures, which result from a recognizably distinct sequence of commands to the instrument. This creates a flow of data which is assembled into a single dataset. Each dataset is given a unique 9 character identifier (an IPPPSSOOT in STScI terminology) and is processed by the pipeline, calibrated, and archived separately from all other datasets.

An illustrative (partial) counter example to this procedure is the WFPC2 CRSPLIT proposal instruction. This results in two WFPC2 exposures from a single line on the exposure logsheet (the way in which observers specify commands for HST). However, the HST ground system treats a CRSPLIT as two distinct exposures which are commanded, processed, calibrated, and archived separately. The pipeline does not combine these two images (datasets) to create the single image without cosmic ray events which may have been the observer's original intention. Currently, the observers (and any future archival researchers) are left to perform this task on their own.

The second generation instruments present many instances in which the combination of data from two or more exposures is necessary to create a scientifically useful data product. Both NICMOS and STIS need to combine exposures to remove cosmic rays and to improve flat fielding (by dithering). For NICMOS, the HST thermal background contributes a significant signal at wavelengths longward of 1.7 µm. Multiple exposures (dithered for small targets and offset onto blank sky-chopped-for extended targets) are necessary to measure and remove this background.

Usage

Associations exist to simplify the use of HST data by observers. This starts from the proposal phase, continues with a more complete calibration process than would be possible without associations, carries into the archiving and retrieval of associated data, and includes the use of HST data by observers within the IRAF/STSDAS system.

An association is a set of one or more exposures along with an association table and, optionally, one or more products. We define the following terms:

• An exposure is the atomic unit of HST data.
• A dataset is a collection of files having a common rootname (same IPPPSSOOT).
• A product is a dataset derived from one or more exposures.

The first generation instruments all had a one-to-one correspondence between exposures and datasets. They do not have products. NICMOS and STIS use the association structure as a meta-dataset. Further, they use the information in multiple exposures during the calibration process to create products.

From a high level, an association is a means of identifying a set of exposures as belonging together and being, in some sense, dependent upon one another. The association concept permits these exposures to be calibrated, archived, retrieved, and reprocessed (within OPUS or STSDAS) as a set rather than as individual objects. In one sense, this is a book-keeping operation which has been transferred from the observer to the HST data pipeline and archive.

Associations are defined by optional parameters on a single exposure logsheet line. That is, there is a one-to-one correspondence between proposal logsheet lines and associations (although it is possible to have exposures which are not in associations).

Observers may obtain one or more associations at each of one or more positions on the sky using the NICMOS proposal grammar. Typically usage will be:

• To obtain a sequence of slightly offset exposures (dithering) to improve the flat fielding, avoid bad pixels and cosmic rays, and, for sufficiently compact targets, to remove the thermal background signal.
• Mapping of targets larger than the NICMOS detector's field of view.
• To obtain a sequence of observations in which the telescope is chopped between the target and one or more offset regions of (hopefully blank) sky.
• To obtain multiple identical exposures of a target with a single exposure logshheet line (Number_of_Iterations).

A set of predefined patterns are provided in the proposal instructions for these types of observations or a combination of both types (Chapter 11). The Institute ground system will expand the observer's single line request into multiple exposures each with its own identifying name (IPPPSSOOT) and populate the necessary database relations to describe this association for the OPUS system.

Re-engineering

For the second generation science instruments the format of the data products from the pipeline is FITS (Flexible Image Transport System) files with image extensions. The IRAF/STSDAS system was modified to operate directly on these files. Each NICMOS image is expressed as a set of five image extensions representing the image, its variance, a bit encoded data quality map, the number of valid samples at each pixel, and the integration time at each pixel. This structure is used at all stages of the calibration process which permits the re-execution of selected elements of the pipeline without starting from the initial point. Finally, the calibration code itself is written in the C programing language (rather than IRAF's SPP language). This greatly simplifies the modification of the pipeline code by users and the development of new NICMOS specific data processing tasks. See Section NICMOS Data Products and the HST Data Handbook for more information regarding data products and structure.