3.1 Pipeline Processing Overview
When STIS was in active operation, telemetry containing STIS science data was downlinked through a TDRSS satellite to a ground station in White Sands, NM. From there it was sent to Goddard Space Flight Center where the PACOR data capture facility collected the downlinked science data into telemetry "pod files". These pod files were then transmitted to STScI where they were saved to a permanent storage medium. The STScI ingest pipeline then unpacked the data, extracted keywords from the telemetry stream, reformatted the data, and repackaged them into raw, uncalibrated, but scientifically interpretable data files. These raw files were then processed by the calstis software to produce a variety of calibrated data files. The results of these procedures were then used to populate the databases that form the searchable archive catalog describing the individual instrument exposures. At this point, the raw and calibrated data files generated from the pod files were normally discarded. Only the pod files and the information placed in the archive databases were preserved.
Each time a user requests data from the Hubble Data Archive via the "On The Fly Reprocessing" (OTFR) system, the raw files are regenerated from the original pod files, and then re-calibrated. What is described in this chapter is calstis, the program that performs the calibration of STIS science data and is available to the community as part of the STSDAS package. The static STIS archive prepared as part of the STIS calibration closeout (see Section 1.5), will make raw and calibrated STIS data available without having to rerun OTFR and recreate all the files.
A more detailed description of the OTFR system as it applies to STIS and other HST instruments can be found in Swade et al. (2001, ASP Conf. Ser., Vol. 238, 295), available on-line at:
http://www.adass.org/adass/proceedings/adass00/P2-36/.
Conceptually, calstis is several pipelines in one, reflecting the complexity and diversity of STIS observing modes. Your STIS data will have been calibrated to different levels, depending on their nature:
- ACQs and ACQ/PEAKs are not calibrated by calstis; you will get only the raw data from observations taken in these modes.
- All other science data are processed through basic two-dimensional image reduction (available in IRAF as basic2d), which includes such things as bias subtraction, dark subtraction, flat fielding, and linearity correction. In the case of CCD CR-SPLIT or REPEATOBS data, your data will also be passed through cosmic ray rejection (available in IRAF as ocrreject).
- Data taken in TIME-TAG mode are available from the archive as both event streams (rootname_tag.fits binary tables), and as raw images equivalent to those produced for ACCUM mode observations. For TIME-TAG data, the accumulation into an image is done by the ground system, rather than onboard the spacecraft as is the case for ACCUM mode data. The calstis software as run in the pipeline uses these raw image files as input in either case, and does not distinguish between TIME-TAG and ACCUM mode data. The calstis pipeline software does not operate on the _tag event files; (see Section 5.6 for a discussion of how to analyze these files).
- For MAMA data, the input raw data format is 2048 × 2048 (so called high-res pixels), while the calibrated data are binned by the pipeline to 1024 × 1024 native format pixels (see Section 3.4.17).
- Spectral data that were taken using a sufficiently small aperture, and which were also taken together with a wavelength calibration spectrum, are then passed through spectroscopic reduction to produce flux and wavelength calibrated science data. For first order spectra modes, a two-dimensional rectified spectral image is produced, and for both echelle and first order modes, a one-dimensional, background subtracted spectrum is also produced. For first order spectral observations where the target was behind the fiducial bars of one of the long slits, only two-dimensional rectified spectra are produced.
- Spectral data taken with very large apertures are treated as slitless observations. For such data, as well as for data taken without contemporaneous wavecal observations, the presumption is that the target location along the dispersion direction is too uncertain to assign a reliable wavelength scale. No flux or wavelength calibrated spectra are produced by the pipeline for such observations. The definition of which apertures are treated as slitless varies depending on the grating in use.
See Chapter 2 for the naming conventions of the various input, intermediate, and output calibrated files.
As with the calibration pipelines for the other HST instruments, the specific operations that are performed during calibrations are controlled by calibration switches, which are stored in the image headers as KEYWORD=VALUE pairs. Any given step in the calibration process may require the application of zero, one, or more calibration reference files, the names of which are also found in the image header. The names of the keywords containing the switches and reference file names were introduced in the previous chapter; Section 3.3 will outline the role these keywords play in the data reduction, and description of the calibration steps are given in Section 3.4. The path your data files take through the pipeline is determined by the calibration switches set in the primary header of the _raw data, which in turn depends directly on the type of data you have.
A few other general comments are in order. It is important to note that some of the STIS calibration reference data are obtained contemporaneously with the science observations. These data may be used to refine the calibration process (as with the automatic wavecals), or may require you to replace a default calibration reference file with a contemporaneously obtained one, as in the case of a CCD near infrared (NIR) fringe flat. The details of how these contemporaneous calibration files are used in calstis can be found in Section 3.4. The STIS (and NICMOS) pipelines are also unusual in that they are re-entrant. That is, a user running calstis off-line may choose to reprocess STIS data partially, performing one or more of the intermediate steps without re-exercising the complete calstis pipeline, for instance to perform cosmic ray rejection or one dimensional spectral extraction. Refer to Section 3.5 for the mechanics (and restrictions) of this kind of processing. Finally, as with other HST pipelines, calstis propagates statistical errors and tracks data quality flags throughout the calibration process.
