Space Telescope Science Institute
help@stsci.edu
Table of Contents Previous Next Index Print


NICMOS Data Handbook > Chapter 2: Data Structures > 2.1 NICMOS Data Files

2.1 NICMOS Data Files
STScI automatically processes and calibrates all NICMOS data and archives the data files resulting from pipeline processing in FITS format. If you have retrieved NICMOS files from the Archive, you will notice that their names look like this:
The first part of the file name (n3w2a1wqm) is the rootname, identifying the dataset to which the file belongs, the second (cal) is the suffix, identifying the type of data the file contains, and the third (fits) indicates that this is a FITS format file. Chapter 2 of the "HST Data Handbook" shows how to access the data contained in NICMOS FITS files, while Appendix B of the "HST Data Handbook" explains how to decipher the rootnames of these files and explains why some of them are grouped into data associations. This section describes the files that constitute a NICMOS dataset.
2.1.1
Each file in a NICMOS dataset has a three-character suffix that uniquely identifies the file contents. The file name suffixes for NICMOS and the corresponding file contents are summarized below. Table 2.1 lists all of the files that the pipeline can produce. For some observing strategies not all of the processing steps are performed and only a subset of these files will be produced by the pipeline.
Table 2.1: NICMOS File Name Suffixes
-
Raw Science File (_raw):
This FITS file contains the raw image data received from the spacecraft. One file per exposure is created (a MULTIACCUM exposure is considered a single exposure irrespective of the number of samples specified).
-
Support File (_spt):
This FITS file contains supporting information about the observation, the spacecraft telemetry and engineering data from the instrument that was recorded at the time of the observation, including detector temperature measurements.
-
Association Table (_asn):
This file is a FITS binary table that contains the list of datasets making up an association.
-
Calibrated Science File (_cal):  
This FITS file contains the calibrated science data for an individual dataset, and is produced by the pipeline calibration task calnica (see Chapter 3). The input to calnica are the _raw images. For a MULTIACCUM exposure, this file contains a single science image formed by combining the data from all samples.
-
Intermediate Multiaccum Science File (_ima):
This FITS file is also produced by the pipeline task calnica and contains the calibrated science data for all samples of a MULTIACCUM dataset before the process of combining the individual readouts into a single image has occurred. This file is only produced for MULTIACCUM observations.
-
Mosaic Files (_mos):   
These FITS files contain the composite target and, for chopped pattern sequences, background region images constructed by the pipeline task calnicb for an associated set of observations (see Chapter 3). The input to calnicb are the calibrated _cal images from calnica and the _asn association table. Target images are co-added and background-subtracted. The value of the last character of the rootname is 0 for targets, and 1 to 8 for background images. These files are only produced for an associated set of observations.
-
Post-calibration Association Table (_asc):
This table is produced by the pipeline calibration task calnicb, and is the same as the association table _asn, with the addition of new columns which report the offsets between different images of the mosaic or chop pattern as calculated by calnicb, and the background levels computed for each image. This file is only produced for an associated set of observations.
-
Trailer File (_trl):
This FITS ASCII table contains a log of the pipeline calibration processing that was performed on individual datasets and mosaic products.
-
Processing Data Quality File (_pdq):
This FITS ASCII table provides quality information on the observation, mostly on pointing and guide star lock. Possible problems encountered, e.g., a loss of guide star lock or a guide star acquisition failure, are reported here.
-
Temperature Telemetry Table (_epc):
This FITS binary table is produced by the Engineering Data Processing System (EDPS) as an aid for use in calibration and temperature monitoring. Nine NICMOS Cooling System (NCS) mnemonics and five temperature sensors are bundled together into an STSDAS FITS table. (Available after OPUS 15.2)
2.1.2
The *_raw.fits, *_cal.fits, *_ima.fits, and *_mos.fits files are all defined as science data files, as they contain the images of interest for scientific analysis.
File Contents and Organization
The data for an individual NICMOS science readout consist of five arrays, each stored as a separate image extension in the FITS file. The five data arrays represent:
An error (ERR) array containing an estimate of the statistical uncertainties (corresponding to 1σ uncertainties) of the science data.
An array of bit-encoded data quality (DQ) flags representing known status or problem conditions of the science data.
An array containing the number of data samples (SAMP) that were used to compute each science image pixel value.
A grouping of the five data arrays for one science image is known as an image set or imset.
A science data file can contain one or more imsets. For example, an individual NICMOS exposure obtained with the ACCUM mode will generate a *_raw.fits file with one imset; an individual MULTIACCUM exposure with n readouts will generate a *_raw.fits file (and, after calibration, an *_ima.fits file) containing n + 1 imsets, including the zeroth readout. The *_cal.fits and *_mos.fits files always contain one imset.
 
The imsets of a MULTIACCUM exposure are stored in files from last to first. The result of the longest integration time (the last readout performed on-board) occurs first in the file (first imset), the readout before the last is the second imset, and so on; the zeroth readout is the last imset.
Although the five science, error, data quality, samples, and integration time arrays associated with each imset are stored in a single file, they are kept separate as five individual FITS image extensions within the file. The order of the images in the FITS files is listed in Table 2.2 and shown graphically in Figure 2.1 and Figure 2.2. The examples given in Table 2.2 and Figure 2.2 refer to a MULTIACCUM image (multiple imsets), while Figure 2.1 refers to ACCUM and BRIGHTOBJ images (one imset, namely extensions 1 through 5). Each extension comes with its own header, and each FITS file contains, in addition, a global or primary header (primary header-data unit or HDU).
The only contents of the primary HDU are header keywords. There are no image data in the primary HDU. The keywords in the primary header are termed global keywords because they apply to the data in all of the file extensions. The organization and location of header keywords is explained in detail later in the chapter.
Table 2.2: NICMOS Science Data File Contents
Figure 2.1: Data Format for ACCUM and BRIGHTOBJ Modes
Figure 2.2: MULTIACCUM Mode Data Format
The following sections explain the contents and origin of each of the five image arrays in each imset in more detail.
Science Image
This image contains the data from the detector readout. In ACCUM and BRIGHTOBJ modes the image received from the instrument is the result of subtracting the initial from the final readouts of the exposure. In MULTIACCUM mode the images received are the raw (unsubtracted) data corresponding to each detector readout. In raw datasets the science array is an integer (16-bit) image in units of DNs (counts). In calibrated datasets it is a floating-point image in units of DNs per second (count rates).
Error Image
The error image is a floating-point array containing an estimate of the statistical uncertainty associated with each corresponding science image pixel. This image is computed in the ground calibration pipeline task calnica as a combination of detector read noise and Poisson noise in the accumulated science image counts (see Chapter 3) and corresponds to 1σ uncertainties. For calibrated MULTIACCUM images (i.e., *_cal.fits files), the values of the error array are computed uncertainties in the count rates derived from the linear fit to counts vs. exposure time from the intermediate readouts.
Data Quality Image
This integer (unsigned 16-bit) array contains bit-encoded data quality flags indicating various status and problem conditions associated with corresponding pixels in the science image. Because the flag values are bit-encoded, a total of 16 simultaneous conditions can be associated with each pixel. Table 2.3 lists the flag values and their meanings.
Table 2.3: NICMOS Data Flag Values
0000 0000 0000 0000
0000 0000 0000 0001
0000 0000 0000 0010
0000 0000 0000 0100
0000 0000 0000 1000
0000 0000 0001 0000
0000 0000 0010 0000
0000 0000 0100 0000
0000 0000 1000 0000
0000 0001 0000 0000
0000 0010 0000 0000
0000 0100 0000 0000
0000 1000 0000 0000
0001 0000 0000 0000
0010 0000 0000 0000
0100 0000 0000 0000

1
Most significant bit is at left.

Number of Samples Image
The SAMP image is an integer (16-bit) array containing the total number of data samples that were used to compute the corresponding pixel values in the science image. For ACCUM and BRIGHTOBJ modes, the number of samples contributing to each pixel always has a value of 1 in the raw data file. For MULTIACCUM mode the sample values in the raw and intermediate data files are set to the number of readouts that contributed to the corresponding science image.
 
Because the number of samples in the raw images for MULTIACCUM, ACCUM and BRIGHTOBJ modes is the same value for all pixels of an imset, the image array is usually not created (to save on data volume), and the value of the sample is stored in the header keyword PIXVALUE in the SAMP image extension (see Table 2.5 below).
In MULTIACCUM calibrated data files (*_cal.fits) the SAMP array contains the total number of valid samples used to compute the final science image pixel value, obtained by combining the data from all the readouts and rejecting cosmic ray hits and saturated pixels. In this case the sample array may have different values at different pixel locations (less than or equal to the total number of samples in the MULTIACCUM sequence), depending on how many valid samples there are at each location.
In the mosaic images (*_mos.fits), the data in the SAMP array indicate the number of samples that were used from overlapping images to compute the final science image pixel value.
Integration Time Image
The TIME image is a floating-point array containing the effective integration time associated with each corresponding science image pixel value. These data are always computed in the ground calibration pipeline for recording in the raw data file.
 
For ACCUM and BRIGHTOBJ mode observations, each pixel has the same time value. For MULTIACCUM observations each pixel for a given readout has the same time value in the raw and intermediate data. In these cases, to save on data volume, the image array is not created and the value of the time is stored in the header keyword PIXVALUE in the TIME image extension (see Table 2.5).
In MULTIACCUM calibrated data files (*_cal.fits) the TIME array contains the combined exposure time of all the readouts that were used to compute the final science image pixel value, after rejection of cosmic ray and saturated pixels from the intermediate data. As in the case of the SAMP array, the TIME array can have different values at different pixel locations, depending on how many valid samples compose the final science image in each pixel.
In mosaic images (*_mos.fits), the TIME array values indicate the total effective exposure time for all the data from overlapping images that were used to compute the final science image pixel values.
2.1.3
The *_spt.fits, *_trl.fits, *_pdq.fits, *_asn.fits, and the *_asc.fits files are termed auxiliary data files. They contain supporting information on the observation, such as spacecraft telemetry and engineering data, assessment of the quality of the observation, calibration information, and information on the associations present in the observations.
Association Tables
The association tables, *_asn.fits and *_asc.fits, are FITS binary tables which are created when a particular observation generates an association of datasets (see the discussion of “Associations” in Appendix B of the "HST Data Handbook"). In particular, the *_asn.fits table is generated by OPUS1, and contains the list of datasets which make up the association (e.g., from a dither or chop pattern). The *_asn.fits tables are the inputs to the pipeline calnicb, which creates the mosaiced or background subtracted images (*_mos.fits files) from the input datasets. All the datasets must have been processed through the basic pipeline data reduction (calnica) before being processed through calnicb. In addition to the output science image(s), calnicb produces another association table (*_asc.fits), which has the same content as the *_asn.fits table, along with additional information on the offsets used by the pipeline for reconstructing the science image and background values computed for each image. For mosaics (dither patterns), there is only one final image produced, with file name *0_mos.fits. For chop patterns, in addition to the background-subtracted image of the target (*0_mos.fits), an image for each background position is produced; the file names of these background images are *1_mos.fits, *2_mos.fits,..., *8_mos.fits (a maximum of eight independent background positions is obtainable with the NICMOS patterns; see the NICMOS Instrument Handbook for details).
Support File
The support files, *_spt.fits, contain information about the observation and engineering data from the instrument and spacecraft that was recorded at the time of the observation. A support file can have multiple FITS image extensions within the same file; in the case of a MULTIACCUM observation there will be one extension for each readout (i.e., each imset) in the science data file. Each extension in the support file holds an integer (16-bit) image containing the data which populates the *_spt.fits header keyword values.
Trailer File
The trailer files, *_trl.fits, contain information on the calibration steps executed by the pipelines and diagnostics issued during the calibration. There is one *_trl.fits file produced for each dataset, and, in the case of associations, one *_trl file for each NICMOS product (i.e., each *_mos.fits file).
Processing Data Quality File
The processing data quality files, *_pdq.fits, contain general information summarizing the observation, a data quality assessment section, and a summary on the pointing and guide star lock. They state whether problems were encountered during the observations, and, in case they were, describe the nature of the problem. There is one *_pdq.fits file produced for each dataset, and, in case of associations, one *_pdq file for each NICMOS product (i.e., each *_mos.fits file).
Temperature Table
The temperature tables, *_epc.fits, contain a set of 14 mnemonics (sensors) extracted from the engineering telemetry. The time range of the tables are over the duration of the corresponding observations. The selected sensor used to monitor the temperature of the NICMOS detectors is NDWTMP11, the temperature at the NIC1 mounting cup. A listing of the individual table names and their location within the FITS table can be obtained by using the IRAF/STSDAS task catfits. (See NICMOS ISR 2003-008 for more details.) Mnemonics in the temperature table *_epc.fits, report temperature information within the NCS and five temperature sensors within the NICMOS dewar. Table 2.8 presents a listing of the 14 NCS and NICMOS telemetry temperature sensors.

1
Observation Support & Post Observation Data Processing Unified Systems


Table of Contents Previous Next Index Print