HST Data Handbook for WFPC2


3.4 Recalibration

With the advent of the On-The-Fly Reprocessing (OTFR) system in May 2001, all WFPC2 data requested from the Archive are automatically and completely reprocessed from the original telemetry file, using the most up-to-date software and calibration files. As such, the need for observers to manually recalibrate WFPC2 images has been greatly reduced. If recalibration is desired (e.g., due to an improvement in a reference file), the data can be re-retrieved from the Archive.

3.4.1 Why and When to Re-retrieve data through OTFR

The primary reason to re-request or reprocess WFPC2 data is the availability of improved or more up-to-date reference files and tables, especially darks, flatfields, synphot component tables, and improvements to the pipeline software, such as the calibration task calwp2. In this section, we provide a brief summary of some relevant changes to the pipeline task and related elements as of today (November, 2001). Details on changes to the pipeline software are maintained in the WFPC2 History Memo (section 5). A quantitative account of the differences between various generations of reference files is given in chapter 4.

Reference Files

Improved reference files will necessarily lag the science images in time. For example, the best dark reference file for a given WFPC2 observation will generally not be available until a few weeks after the science observation is taken. The time is required to retrieve the dark frames from the Archive, generate a new reference file, deliver the file to CDBS, and install the file for use in OTFR. Generally, there is one dark reference file per week, a combination of the dark frames taken that week and the current super dark.

Other reference files, such as the bias and flatfield, are more stable and thus are updated at less frequent intervals than the weekly darks. Bias reference files are generated roughly on an annual basis from a large set of frames taken over the course of one year. These new bias reference files become retroactive, applicable to any science data taken during the year covered by the individual bias frames used to generate the reference file. Superdarks are also generated about once per year; the input dark frames are processed with the superbias covering that year. The flatfields are typically updated less frequently than biases and darks, usually only once every few years. The mask file, A-to-D lookup table, and shutter shading files were created early in the mission of WFPC2 and have not changed since then. To check the quality of the reference files applied to your science observations, check the PEDIGREE and DESCRIP keywords in the HISTORY comments at the end of the header of the calibrated image (.c0h). For more details, refer to the reference file header itself, particularly the HISTORY records.



Prior to the public release of OTFC in 2000, the calibrated data originally delivered to the principle investigator and to the Archive had been processed through the pipeline within a few days of the observation. At that time, the available dark reference file normally did not contain the most up-to-date information about warm pixels, which change on a weekly timescale (see section 3.5.1). Thus, pre-OTFC calibrated data were always processed with an out-of-date dark, and the correction of warm pixels could be improved by manually recalibrating with a more appropriate dark file or by using the STSDAS task warmpix. In this era of OTFR (see section 3.1) however, reprocessing with updated reference files can be accomplished by simply requesting the data from the Archive at least a few weeks after the science observations were taken.

Since August 1, 1996, the dark reference files used in the pipeline have been generated by combining an appropriate superdark (stack of 120 individual dark frames, taken over the course of about one year) with the information on warm pixels from one week's worth of darks (stack of 5 darks taken over less than two days). Thus, a new pipeline dark reference file exists for every week in which a set of five long dark frames were taken; the primary difference from week to week is the warm pixel population, since the superdark contribution remains the same over a year.

Prior to August 1996, pipeline dark reference files had been generated from a stack of only ten dark frames taken during a two week interval. Thus, very deep observations taken before August 1996 could benefit from recalibration with an improved dark reference file. For observations longer than about 20,000 seconds (10,000 seconds in the UV and in narrow-band filters), the noise in the dark frame is a significant contributor to the total noise in the final calibrated image. For this reason, if the science results are limited by the noise, observers are advised to manually recalibrate their images with a superdark, since this will significantly reduce the noise associated with dark subtraction. Note that it is also advisable to use an appropriate superbias as well as correct for warm pxiels independently (e.g., by a post-pipeline processing task such as warmpix, discussed in section 3.5.1). Some early superdarks are listed in the WWW Reference File Memo and are available from the Archive; others are listed in the IDT Reference File Memo. Also, the individual dark frames are always available from the Archive, should observers wish to generate their own custom superdark.



Very early observations, those processed before March 9, 1994, used interim flatfields based on pre-launch data. These flatfields did not have good large-scale properties (peak errors were about 10%) and thus WFPC2 observations processed before that date should be re-retrieved through OTFR (the processing date can be determined from the trailer file). After March 1994, in-flight flatfields were used. Their quality has steadily improved, and those currently in the pipeline are believed to be good to about 0.3% on small scales, and 1% or less on large scales. See section 4.2 for a detailed discussion of the differences between various generations of flatfields.

If you are in doubt about the quality of the flatfielding in your observations, check the PEDIGREE and DESCRIP keywords of the flatfield file, also reported (after December 1994) in the HISTORY comments at the end of the header of the calibrated image. If PEDIGREE is GROUND, the data will need to be recalibrated. If the PEDIGREE is INFLIGHT, the flatfield was obtained from on-orbit data, and the DESCRIP keyword gives some information on its quality. INFLIGHT flatfields are of sufficient quality for most scientific goals, but for especially demanding applications and data with very high signal-to-noise ratio, it may be advisable to recalibrate with the most recent flatfields.

Photometric Tables

The photometric component tables are used by synphot to determine the photometric calibration header parameters, namely PHOTFLAM and PHOTPLAM (see chapter 5). These component tables have been updated several times, most recently on May 16, 1997, in order to contain the most up-to-date information on the throughput of WFPC2. If the photometric tables have changed, users can either re-retrieve their observations from the Archive or run synphot directly to determine the header parameters (see section 3.3.2). Note that to run synphot directly, observers must have the most up-to-date synphot tables installed at their local site; as a group, these tables are part of the standard STSDAS installation which is described in appendix A. Individual tables may be retrieved via the STScI ftp sites:

Users can also use the alternate methods given in section 5.1 to photometrically calibrate their observations.

Pipeline Calibration Task calwp2

Another possible reason for recalibration is to use a more recent version of the calibration pipeline task calwp2. This task has seen several minor revisions, usually to add information to processed data; the current version, as of this writing, is Only three of the calwp2 revisions since WFPC2 operations began actually affect the calibrated data. The first two were changes in how the bias level is computed. Starting in March 8, 1994 (version, columns 3 through 8 of the overscan data were no longer used because they could be affected by the image background; beginning on May 4, 1994 (version, separate bias levels were computed for even and odd columns, resulting in a slightly better image flatness for about 1% of all WFPC2 images (see WFPC2 ISR 97-04).

The third update (version, January 3, 1997) corrects a bug introduced in December 1994 ( in the calibration of WFPC2 single-chip, two-chip, or three-chip observations that do not include the PC (about 1% of all archived observations). We recommend that observers in possession of observations taken and processed between December 1994 and January 1997, re-request their data from the Archive.

3.4.2 Running calwp2 manually

As of May 2001, all WFPC2 data requested from the archive are processed through the On-The-Fly Reprocessing (OTFR) system, using the most up-to-date reference files and software. As such, recalibrating WFPC2 images can be done by simply re-retrieving the data via OTFR. However, occasionally, users may wish to recalibrate manually, for example, to make use of a non-standard dark or flatfield. In these cases, calwp2 will need to be run from the IRAF/STSDAS command line. To recalibrate, the raw data and all reference files must be retrieved, the headers must be updated with the desired corrections to be performed, and the reference file names must be changed as needed. This section outlines in detail the necessary steps required to manually recalibrate WFPC2 data.

Retrieve necessary files

In order to recalibrate a WFPC2 observation, you need to retrieve the dataset as well as all of the reference files and tables needed for calibration. Since standard pipeline processing uses those files listed by StarView as the best reference files, see chapter 1 of the HST Introduction for a description of how to obtain the appropriate reference files from the STScI Archive. We suggest copying the raw data files and the required reference files and tables to a subdirectory used for recalibration as this will preserve all original files. Before beginning, any calibrated files (e.g., .c0h/.c0d, .c1h/.c1d, .c3t, .cgr) must be removed from the recalibration directory, since calwp2 will not overwrite pre-existing calibrated products.

Edit calibration switches and reference files

The next step in recalibrating WFPC2 data is to set the calibration switches and reference keywords in the header of your raw data file (.d0h). These switches determine which calibration steps will be performed and which reference files will be used at each step in the process. The calibration header keywords in a dataset can be changed using the hedit or chcalpar task in the STSDAS hst_calib.ctools package. The hedit task provides more detailed control over individual keywords and is preferred by some users experienced with calibration of WFPC2; though simpler to use, each switch and file must be changed individually.

The chcalpar task takes a single input parameter-the name(s) of the image files to be edited. When chcalpar starts, it automatically determines the instrument used to produce that image and opens one of several parameter sets (pset) which it loads with the current values of the header keywords. The WFPC2 pset is named ckwwfp2. Typing ckwwfp2, as a task name, at the cl> prompt will also edit this pset.

A detailed description of the steps involved in changing header keywords follows:

  1. Start the chcalpar task, specifying the image(s) for which you want to change keyword values. If you specify more than one image, say by using wildcards, the task will take initial keyword values only from the first image. For example, you could change keywords for all WFPC2 raw science images in the current directory (with initial values from the first image), using the following command:
    wf> chcalpar u*.d0h

  2. When chcalpar starts, you will be placed in epar-the IRAF parameter editor, and will be able to edit the parameter set of calibration keywords. Change the values of any calibration switches, reference files or tables to the values you wish to use for recalibrating your data. Remember that no processing has been done on the raw datasets. Therefore, even if you wish to correct, for instance, only the flatfielding, you will need to redo the bias and dark current subtraction as well; hence, the switches for all these steps must be set to PERFORM. Note that only the keywords you actually type in will be substituted.
  3. Exit the editor by typing :q two times (the first :q to exit the pset editor, the second to exit the task). The task will ask if you wish to accept the current settings. If you type "y", the settings are saved and you will return to the IRAF prompt. If you type "n", you will be placed back in the editor to re-define the settings. If you type "a", you will return to the IRAF prompt and any changes will be discarded. For additional examples of updating the calibration keywords, check the on-line help by typing help chcalpar.

Set directory pointers

The calibration reference file names in the header of the raw data (i.e., the .d0h file) are typically preceded by five characters (e.g., uref$ for calibration images and utab$ for calibration tables). These are pointers to the subdirectory where the reference and .x0h/.q1h raw data files are located. Before calibrating your data, you will need to set these pointers. For WFPC2 data, you would use something like the following:
to> set uref = "/your/hstdata/caldir/"
to> set mtab = "/your/hstdata/caldir/"
to> set ucal = "/your/hstdata/rawdir/"


where caldir is the subdirectory for the reference files and rawdir is the subdirectory for the uncalibrated images.

To set the uref in VMS, one would use:
to> set uref = "DISK$SHARE:[HSTDATA.CALDIR]"


where HSTDATA.CALDIR is the directory where you have stored the calibration reference files and tables.

Execute calwp2

Once you have correctly changed the values of the calibration keywords in the header of the raw data file, you are ready to recalibrate your data. The WFPC2 calibration software, calwp2, is run by typing the name of the task followed by the rootname of the observation dataset. For example, to recalibrate the dataset u0w10e02t and write the log of the results to the file calwp2.log (rather than to the screen), type:
wf> calwp2 u0w10e02t "" > calwp2.log


Note that calwp2 will not overwrite an existing calibrated file. If you run the task in the directory where you already have calibrated data, you will need to specify a different output file name, for example:
wf> calwp2 u00ug201t wfpc_out > wfpc.log

For more information about how these routines work, use the on-line help by typing help calwp2.

A note about calculating the absolute sensitivity for WFPC2

If you set DOPHOTOM=OMIT before running calwp2, then the values of inverse sensitivity (PHOTFLAM), pivot wavelength (PHOTPLAM), RMS bandwidth (PHOTBW), zeropoint (PHOTZPT), and observation mode (PHOTMODE) will not be written to the header of the recalibrated data file. Remember that the DOPHOTOM calibration step does not alter the values of the data (which are always counts or data numbers in the calibrated file), but only writes the information necessary to convert counts to flux in the header of the file. Therefore, unless you wish to recalculate the absolute sensitivity for your observation (e.g., because a more recent estimate of the throughput exists for your observing mode), there is no need to recompute these values and you can simply use the keyword values from your original calibrated file and apply them to your recalibrated data. However, new estimates of WFPC2 transmission and absolute sensitivity were obtained in September 1995, May 1996, and May 1997. If your data were processed in the pipeline before May 1997, you may wish to re-create the absolute sensitivity parameters using the latest version of synphot, which contains tables based on the most recent photometric calibration of WFPC2; see Creation of Photometry Keywords, for an example of this computation.

If you wish to recalculate the absolute sensitivity, set DOPHOTOM=YES in the .d0h file before running calwp2, or alternately, use the tasks in the synphot package of STSDAS. The synphot section in section 3.3.4 of the the HST Introduction has more information about how to use this package. To calculate the absolute sensitivity, calwp2 and the synphot tasks use a series of component lookup and throughput tables. These tables are not part of STSDAS itself, but are part of the synphot dataset, which can easily be installed at your home site (see appendix section A.3.2 for information about how to do this). A more detailed discussion of photometric calibration can be found in section 5.2.

The most recent synphot tables must be in place in order to recalculate absolute sensitivity for WFPC2 data using calwp2 or synphot (see chapter 3 of the HST Introduction).

Space Telescope Science Institute
Voice: (410) 338-1082