3.1 Pipeline Processing Overview
All data taken by HST are run through the STScI calibration pipeline. This consists of two main software systems, the Operations Pipeline Unified System (OPUS) and the Data Archive and Distribution System (DADS). Raw spacecraft telemetry from Goddard Spaceflight Center is processed by OPUS into uncalibrated data in a step called Generic Conversion. CALACS is then run by OPUS to process the uncalibrated data, using specific ACS reference images and tables. DADS populates a database from these data which is accessible to users via StarView. DADS is responsible for distributing any data (both calibrated and uncalibrated) requested for download to the user. (See chapter 1 of the HST Introduction for more information.)
ACS will be the first instrument with the "On The Fly Reprocessing" (OTFR) system in place from the beginning of its observing program. OTFR was developed with ACS in mind, due to the large file size of the uncalibrated WFC (68.7MB) and calibrated WFC (168MB) images. Under the previous system, all OPUS data was archived and DADS would distribute the same version of each data file each time it was requested. Archiving ACS WFC images would pose a significant storage problem for DADS. OTFR solves this problem by reprocessing the raw spacecraft telemetry files through OPUS "on the fly" for distribution each time any data are requested. Calibrated ACS data are not required to be archived for later distribution. Only the much smaller raw telemetry files are archived, while the larger uncalibrated and calibrated data are deleted after distribution to the user. More details on OTFR are in the section 3.5.
The goal of the ACS pipeline is to provide data calibrated to a level suitable for initial evaluation and analysis for all users. Further, observers frequently require a detailed understanding of the calibrations applied to their data and the ability to repeat, often with improved calibration products, the calibration process at their home institution. Therefore, the CALACS package used in this pipeline can also be used to calibrate ACS data off-line and is available within the STSDAS system.
The calibration reference files (e.g. flatfields, bad pixel tables) are also available from the HST Archive. Each ACS observation requires calibrations suited to the particular observation mode used to take the data. Calibration data specific to the different modes of ACS operation are prepared and archived in the Calibration Database System (CDBS). The Generic Conversion, described above, queries CDBS to determine which reference files apply to the specific observation configuration and then processes the raw files into uncalibrated data. The input files for CALACS then contain keywords for each calibration step which record the name of the reference file appropriate for each observation.
