3.1 Pipeline Processing, OTFR, and the HST Archive

NICMOS data are retrieved from the HST Archive using “On-The-Fly Reprocessing” (OTFR), a method also implemented for ACS, STIS and WFPC2, and planned for future instruments (e.g. COS and WFC3). OTFR goes back to the original telemetry (the “POD files”) to reconstruct FITS files, and then processes these raw FITS data through the most up-to-date version of the OPUS pipeline software, using the latest and best reference files available at the time of retrieval from the Archive, regardless of when the observations were conducted. As with all HST instruments, the pipeline software and reference files have evolved and improved with time as our knowledge of the instrument and its calibration have developed. OTFR ensures that users will receive the best standard processing available at the time they request NICMOS data from the Archive, and makes it easy to recalibrate data taken earlier in the mission lifetime.

All of the steps performed by the pipeline are recorded in the trailer file for your dataset (*_trl.fits). The main steps performed by the pipeline are:

1.	The data are partitioned (separated into individual files, e.g., engineering and science data are separated).

2.	The data are edited, if necessary, to insert fill values in place of missing data.

3.	The data are evaluated to determine if there are discrepancies between a subset of the planned and executed observational parameters.

4.	A list of calibration reference files to be used in the calibration of the data is created based on the executed observational parameters. This step does not generate comments in the NICMOS trailer file.

5.	The raw data are converted to a generic (FITS) format and the header keyword values are populated (known as generic conversion).

6.	The raw data is used to calculate the detector temperature at the time of the observation using the CalTempFromBias software.

7.	The raw data are calibrated using a standard set of NICMOS calibration programs (calnica and calnicb).

8.	If applicable, the task runcalsaa is run

The OTFR processing steps that involve STSDAS routines are described in Figure 3.1.

Figure 3.1: Processing Flow for OTFR

The calibration software used by the pipeline (steps 6-8 above) is exactly the same as that provided within STSDAS (see Section 3.2). The calibration files and tables used are taken from the Calibration Data Base System (CDBS) at STScI and are the most up-to-date versions available at the time the data are requested and processed by OTFR. However, calibration reference files and software have been improved with time, which is a reason why you may wish to reprocess your data. For example, beginning 17 December 2008 (OPUS version 2008_a and later), the data retrieved using OTFR uses temperature dependent reference files in the calibration step. Also, prior to 26 September 2001, the HST archive did not use the OTFR, so if you received NICMOS data as a GO or from the HST Archive before that date, the reference files and software used may be outdated. Therefore if you require best possible calibrations, you should check the retrieval status of your data and possibly consider recalibration. This is further described in Section 3.5