HST Data Handbook for WFPC2

3.1 Data Processing Pipelines

---

Up until 1999, HST observers received their data via tape and/or via Archive retrieval requests; the raw and calibrated data were copies of files that had been stored on the archive media and were the result of pipeline processing at the time the image was taken. In December 1999, the On-The-Fly Calibration (OTFC) system was released to the public. In response to a user request for data, OTFC retrieved raw files (.d0*,.q0*,.x0*,.q1*) from the archive, then calibrated and shipped the data to the requestor. The primary advantages of OTFC included the

• automatic application of improved calibration files and switches,
• use of most recent calibration software (allowing for rapid access to improved algorithms, new capabilities, and software fixes), and
• correction of header keywords, if needed.

An additional major benefit to OTFC was that only the uncalibrated data needed to be stored in the archive¹.

The On-The-Fly Reprocessing (OTFR) system replaced OTFC on May 16, 2001, a change transparent to the majority of HST archive users. Requests for data are submitted either via StarView or the WWW (see chapter 1 of the HST Introduction); freshly-processed raw and calibrated data are delivered². There is no need to explicitly ask for OTFR, all requests for WFPC2 data are now handled by the OTFR system.

The primary difference between the two on-the-fly systems is that OTFR begins earlier in the data path. OTFR starts with the original telemetry files ("POD" files) received from Goddard Space Flight Center and performs all pipeline processing steps, including data partitioning, fixing data dropouts, converting the partitioned data packets into raw science files, and calibrating the newly-created raw files. OTFC performed only the last pipeline processing step, calibration, on raw files stored in, and retrieved from, the archive.

An overview of the data flow for both systems is summarized in the table below. The benefits of the new OTFR system encompass the benefits of the OTFC system, plus OTFR data need fewer header corrections since most problems are fixed as part of the pre-calibration pipeline processing. Also, the OTFR system as a whole requires significantly less maintenance effort than OTFC. As of late 2001, both the POD files and raw pre-calibration WFPC2 data are still being archived; the raw files are not used for distribution but kept merely as a backup to the POD files. Future plans call for the eventual cessation of raw data archiving, after which only the POD files will be archived.

Table 3.1: Comparison of Dataflow in the On-The-Fly Systems

OTFC (December 1999 - May 2001)	OTFR (May 2001 - present)
Request for data is submitted to the archive via StarView or WWW interface. Archive responds with acknowledgement email.	Same as OTFC.
Raw files are retrieved from the HST archive and passed to the OTFC system. For WFPC2, the raw files are the d0, q0, q1, x0, and trl files.	POD file1 (original telemetry file) is retrieved from HST archive and passed to OTFR system. Pre-calibration OPUS processing is performed: data partitioning, fixing dropouts, generating raw files (d0, q0, q1, x0, and trl files).
Any problems in the header keywords are fixed by special lookup table.	Same as OTFC although the OPUS pre-calibration processing fixes the majority of keyword problems automatically (i.e., significantly fewer header corrections required in OTFR).
The best calibration files & switches are determined by a separate standalone task, and header keywords updated accordingly.	Not needed. The best calibration files & switches are set by the pre-calibration OPUS code (generic conversion).
Images are calibrated by STSDAS calwp2 and sent back to the archive system.	Same as OTFC.
Archive delivers raw + calibrated data and emails completion notification to the requestor.	Same as OTFC. Files contain a new keyword, PROCTIME, to record when the OTFR system processed the data.

1 For WFPC2, there is 1 image per POD file.

 
¹ As of June 2000, calibrated WFPC2 data are no longer archived.

² By default, OTFR currently delivers both raw and calibrated data. A future enhancement to the retrieval process will allow requests for subsets of files (e.g., only calibrated).