Wide Field and Planetary Camera 2 Instrument Handbook for Cycle 13

8.7 On-The-Fly Reprocessing Systems

---

The On-The-Fly Calibration (OTFC) system, publicly released in Dec. 1999, calibrated data at the time a user requested data from the archive. The advantages to using OTFC included the automatic application of improved calibration files and switches, use of most recent calibration software (allowing for rapid access to improved algorithms, new capabilities, and software fixes), and correction of header keywords if needed. An additional benefit is that only the uncalibrated data needs to be stored in the archive.

The On-The-Fly Reprocessing (OTFR) system replaced OTFC on May 16, 2001. The change will be transparent to most HST archive users. Requests for data will be submitted as usual via StarView or WWW; raw and freshly-calibrated data will be delivered. There will be no need to explicitly ask for OTFR: all requests for WFPC2 data will be handled by the OTFR system.

The primary difference between the two systems is that OTFR begins earlier in the data path. It uses the original telemetry files ("POD" files) received from Goddard Space Flight Center and performs all pipeline processing steps; OTFC performed only the last pipeline processing step (calibration), on raw files retrieved from the archive. An overview of the data flow for both systems is summarized in the table below. The benefits of the new OTFR system encompass the benefits in the OTFC system; in addition, OTFR data will need fewer header corrections (most problems will be fixed as part of the pre-calibration pipeline processing) and the system as a whole will require significantly less maintenance effort than OTFC.

UCHCOORD and OTFR.

Improved knowledge of the detector plate scales and chip rotations, as well as changes in reference pixel locations, have resulted in periodic changes to the pointing parameters, especially early in the instrument's lifetime. These header parameters, which define the mapping between the pixel and world coordinate systems, can be updated using the STSDAS task UCHCOORD. The keywords affected include the reference pixel locations (crpix*), the values of the world coordinate system at the reference location (crval*), the partial derivatives of the world coordinate system with respect to the pixel coordinates (cd*), and the orientation of the chip (orientat).

Prior to OTFR (released to the public on May 16, 2001), observers requiring the most up-to-date pointing information in their science image headers ran UCHCOORD on their calibrated images. Since the implementation of OTFR in May 2001, all WFPC2 data retrieved from the archive, regardless of its observation date, has had these corrections applied automatically before being delivered, thus we have discouraged running the UCHCOORD task on OTFR data since it is no longer needed (as described in WFPC2 STAN 45, March 2001 by Baggett, Hsu & Gonzaga). It was expected that modifications to the UCHCOORD task enabled it to discriminate between old and new OTFR data, such that it would not perform astrometric corrections on any OTFR data, if it were inadvertently run (for example, Section 4.3.3 in WFPC2 Data Handbook, Version 4.0, 2002, S. Baggett et al.).

However, it has recently become apparent that the UCHCOORD task may still modify the astrometric information in image headers even when the data are retrieved via OTFR, since the keyword expected by UCHCOORD ("PROCDATE") does not match the keyword provided by OTFR ("PROCTIME"). We thus caution observers that it is likely that the UCHCOORD task has always incorrectly modified astrometric information in images retrieved via OTFR if it were run on such data.

Now that this issue has been identified, we have implemented the correction to it in the UCHCOORD task, and this corrected version will be available in the next official version of STSDAS, scheduled for September, 2003. We remind users that it is no longer necessary to run the UCHCOORD task on any WFPC2 data that is retrieved via OTFR, and we recommend that any old WFPC2 data should rather be re-retrieved via OTFR since many other calibrations are also improved. However, the new version of UCHCOORD will now correctly check whether or not the images have been processed through OTFR in case it is inadvertently run on such data, and will no longer modify the header astrometric keywords in such cases.

Table 8.1: Comparison of Dataflow in On-The-Fly Systems

OTFC	OTFR
Request for data is submitted to the archive via StarView or WWW interface; archive responds with acknowledgement email.	Same as OTFC.
Raw files are retrieved from the HST archive and passed to the OTFC system. For WFPC2, the raw files include d0, q0, q1, x0, and trl files.	POD file (original telemetry file) is retrieved from HST archive and passed to OTFR system. For WFPC2, there is typically 1 POD file for each image. Pre-calibration OPUS processing is performed: data partitioning, data editing, and generic conversion; these steps generate the raw files (d0, q0, q1, x0, and trl files).
Any problems in the header keywords are fixed by special lookup table.	Same as OTFC although the OPUS pre-calibration processing will have fixed the majority of keyword problems automatically (i.e., significantly fewer header corrections required in OTFR).
The best calibration files & switches are determined by separate standalone task, and header keywords updated accordingly.	Not needed. The best calibration files & switches are set by the pre-calibration OPUS code (generic conversion).
Images are calibrated by STSDAS calxxx module and sent back to the archive system.	Same as OTFC.
Archive delivers raw + calibrated data and emails completion notification to the requestor.	Same as OTFC.