HST Data Handbook for WFPC2


3.5 Improving the Pipeline Calibration

The individual calibrated images produced by the standard pipeline processing are, in most respects, as good as our knowledge of the instrument can make them. The usefulness of post-pipeline calibration is, in general, limited to three areas: improving the correction of pixels with elevated dark current (warm pixels), which are known to vary with time; employing a correction flatfield or alternate flatfield; and removing cosmic rays by comparing multiple images of the same field.

The treatment of warm pixels and cosmic rays can be quite different in the case of dithered data. This case is discussed in section 5.5; the present discussion refers to co-aligned data only.

3.5.1 Warm Pixels

Figure 3.4 shows a section of a PC image of a stellar field where cosmic rays have been removed through comparison of successive images. Nonetheless, individual bright pixels are clearly visible throughout the field.

Figure 3.4: PC Image of Stellar Field Showing Warm Pixels

These bright pixels are warm (or hot) pixels, i.e., pixels with an elevated dark current. The vast majority of WFPC2 pixels have a total dark current of about 0.005 e-/s-1 (including the dark glow, discussed in section 4.3.2). However, at any given time there are a few thousand pixels in each CCD, called warm pixels, with a dark current greater than 0.02 e-/s-1, up to several e-/s-1 in a few cases (see figure 3.5). Some of these pixels are permanently warm, but most become warm during the course of the month, probably as a consequence of on-orbit bombardment by heavy nuclei. STIS, another instrument currently aboard HST that uses CCDs, exhibits a similar behavior. Most warm pixels return to normal after the CCDs are brought to room temperature for a few hours.

Will Warm Pixels Hurt my Science?

The impact of warm pixels on the scientific results obtained from WFPC2 images depends on a number of factors: the exposure length, the number of objects, and the science goals. If the principal goal of the program is to acquire morphological information on well-resolved targets, warm pixels are usually not a serious concern since they are easily recognizable. If the goal is accurate photometry of point sources, the probability that uncorrected warm pixels will influence the measurement at a given level can be computed on the basis of the distribution of warm pixels (see figure 3.5). In general, warm pixels are a concern in two cases: accurate photometry of faint sources in crowded fields, where warm pixels can easily be confused with cores of faint sources; and aperture photometry with very large apertures and/or of extended objects. In the latter case, warm pixels cause a positive tail in the count distribution that is not included in the background determination, but-depending on the software used-could be included in the integrated source flux, which then will be positively biased.

Figure 3.5: Distribution of Dark Current for Warm Pixels

Repairing warm pixels

Decontaminations, during which the instrument is warmed up to about 22o C for a period of six hours, are performed about once per month. These procedures are required in order to remove the UV-blocking contaminants which gradually build-up on the CCD windows (thereby restoring the UV throughput) as well as fix warm pixels. These pixels, with dark current levels of 0.02 electrons per sec or higher, generally appear at a rate of about 30 pixels per detector per day. A decontamination procedure typically anneals about 80% of the new warm pixels that arise during the month. Of those pixels that are not fixed, about half are fixed after two or three additional decontamination procedures. After that, the rate of correction decreases; longer decontaminations do not appear to improve the fraction of pixels fixed. For more detailed information, see the WFPC2 Instrument Handbook.

Because of the time variability of warm pixels, the standard pipeline dark correction (generally, one dark reference file per week) may not deal with them adequately. Even dark frames taken within a day of the observation will contain some warm pixels that vary significantly from those in the science observation. There are several ways to improve the correction for pixels which are known to be warm or which have varied near the time of the observations: flag the pixels in some way; use the STSDAS task, warmpix; or generate a custom dark reference file and recalibrate manually. These methods will be outlined in the following sections.


Identifying and flagging warm pixels

The first method to treat warm pixels is to identify and flag them. Depending on the software used, the flagged pixels can either be ignored (PSF fitting software generally allows this) or be interpolated from nearby pixels (for software that requires a valid value for all pixels, such as most aperture photometry tasks). The identification of warm pixels can be accomplished by taking advantage of the fact that they are the only WFPC2 feature to extend across only one pixel; both cosmic rays and photons, in the form of point sources, involve more than one pixel. The IRAF task cosmicrays, written originally to remove single-pixel cosmic rays in ground-based data, has been used with some success to identify warm pixels in WFPC2 data. Identification of warm pixels is also possible using information from dark frames taken before and after the observations were executed, as described below.


Subtracting warm pixels via warmpix

The second option is to attempt subtraction of the warm pixel dark current that existed at the time of the observations. This has the advantage that the information that exists in the measured signal in the pixel is retained, but it does require independent timely information on the dark current. WFPC2 takes about five long dark frames every week, thus information on warm pixels is available with a time resolution of about one week. The STSDAS task warmpix, will flag and/or correct warm pixels using an input table, available on the WWW, containing the locations and dark count rates for warm pixels that existed around the time of the science observation. Each table typically spans the time interval between decontamination procedures, with information derived from dark files taken at several epochs (roughly once per week) within that period. This procedure will generally fix 90% to 95% of the warm pixels found in typical user data, though there are some uncertainties in the results due to the intrinsic variability of warm pixels and the time span between darks. The steps necessary to run warmpix are summarized below.

  1. Obtain the relevant warm pixel tables from the WWW. Each table name reflects its applicability dates; for example, the table named applies to all observations between December 14, 1995 and January 11, 1996.

    These tables are in Unix-compressed format. On some systems, the retrieved table will not have the .Z extension, but they still need to be renamed to add the .Z extension and uncompressed by the Unix task uncompress. Please contact the Help Desk at if you have a non-Unix system or if you encounter difficulties in retrieving the tables.

  2. Retrieve the calibration reference files used for dark subtraction and flatfielding from the Archive, see section 1.1 of the HST Introduction. The filenames are recorded in the science header keywords DARKFILE and FLATFILE, respectively.
  3. Redefine the IRAF variable uref$ to point to the directory where the dark and flatfield files are stored. This step is required for warmpix to undo the dark current subtraction performed in the pipeline and substitute its own. If warmpix cannot find these reference files, it will not be able to correct the dark current subtraction and will flag all pixels as being uncorrectable.
  4. Run warmpix to correct and/or flag warm pixels. There are a number of user-adjustable parameters to decide which pixels should be fixed and which should be flagged as uncorrectable. Please see the online warmpix STSDAS help file for more details (type "help warmpix" at the iraf prompt).

At the end of this process, pixels that exceed the user-defined thresholds either will be corrected for the dark current measured in the darks, linearly interpolated to the date of the observation, or flagged as uncorrectable. Specifically, if the pixel has a high or extremely variable dark count rate, warmpix will not change the pixel value in the science image but the data quality file will have its 10th least significant bit set to indicate that it is a "bad" or irrepairable pixel (i.e. value of 512, logically OR'ed with the other calibration reference data quality file flags for that pixel). If the pixel has a moderate dark count rate, warmpix will fix the pixel in the image and insert a value determined from an extrapolation of the warm pixel's value between two epochs that cover the time of observation for the image. In addition, the data quality file will have the value 1024 logically OR'ed with the other data quality files. Pixels with low dark count rates are not modified by warmpix.

Non-STSDAS tasks generally ignore the data quality files, and thus may not properly use the information indicating which pixels need to be rejected. Users should propagate this information by the appropriate method, which will depend on the specifics of the task.


Generating a custom dark reference file using "daily darks"

In July 1997, a calibration program was begun to obtain up to three dark frames every day, to allow for better warm pixel corrections. These darks, also referred to as "daily darks", are relatively short (1000 seconds) so that the observations can fit into almost any occultation period, making automatic scheduling feasible. In addition, the priority of the daily darks is low; they are taken only when there is no other requirement for that specific occultation period, so daily coverage is not guaranteed. Observers should be aware that only the standard (1800 second) darks, taken at the rate of five per week, are used in generating the pipeline darks, superdarks, and warm pixel tables. The daily darks are available without a proprietary period to the GO community via the Archive.

These daily darks should be used if very accurate identification of warm pixels is needed. Many observers develop their own software to make use of daily darks to improve the warm pixel correction. An alternative, however, is to use canned IRAF scripts available from STScI to generate a custom dark reference file for use in manually recalibrating their science images (see ISR 01-01 and Addendum 01-08). These reports provide detailed instructions on how to use the scripts to generate a custom dark; section 3.4.2 provides details on how to manually rerun calwp2.

3.5.2 Alternate Flatfields


Correction Flatfields

As of December 2001, the noise characteristics of the WF chips in the standard WFPC2 pipeline flatfield reference files are such that the signal-to-noise achievable in the final calibrated science images is not limited by the flatfield. However, all PC1 flatfields (and some WF flatfields in the UV) have less than ideal noise properties; improving these flatfields can improve the resulting signal-to-noise in the calibrated images.

A set of "correction" flatfields, designed to be applied after normal pipeline processing (i.e., OTFR), have been developed in order to help reduce the flatfield noise. In particular, highly-exposed science images (>20,000 e-/pixel, or 2860 and 1335 DN/pixel for gains 7 and 15, respectively) will show significant noise reduction, especially in the PC, if the new correction flatfields are used. Science images in some of the UV filters will show significant improvement as well, even at lower exposure levels. The correction flats are available from the Archive and have been generated so that they merely need to be multiplied into the calibrated (including flatfielded) science images. Please see ISR 01-07 for the details and names of these correction flats, as well as some caveats regarding their application to science images. Note that these correction flatfields are not, as of November 2001, in the OTFR pipeline, though they may be incorporated for selected filters in the future.


Flatfielding Linear Ramp Filter Images

Images observed with the linear ramp filters (LRFs) are, by design, currently not flatfielded in the OTFR pipeline. The calibrated science headers and trailers will indicate that the flatfield used was a "dummy" (i.e., 1 everywhere) and that the correction was effectively skipped. This is the case in the LRF image example below. The raw file (.d0h) has the FLATCORR set to perform; however, based upon the flatfield reference file PEDIGREE, calwp2 flags the correction step as SKIPPED in the calibrated image (.c0h).
cl> hedit u*d0h flat* .
u5ly020cr.d0h,FLATCORR = PERFORM
u5ly020cr.d0h,FLATFILE = uref$f4i1559cu.r4h
u5ly020cr.d0h,FLATDFIL = uref$f4i1559cu.b4h

cl> hedit uref$f4i1559cu.r4h pedigree,descrip .
uref$f4i1559cu.r4h,PEDIGREE = "DUMMY 18/04/1995"
uref$f4i1559cu.r4h,DESCRIP = "All pixels set to value of 1."

cl> hedit u*c0h flat* .
u5ly020cr.c0h,FLATCORR = SKIPPED
u5ly020cr.c0h,FLATFILE = uref$f4i1559cu.r4h
u5ly020cr.c0h,FLATDFIL = uref$f4i1559cu.b4h

cl> page *trl | grep FLAT


A single pipeline flatfield is difficult to generate for the LRFs, primarily due to the lack of an accurate spectrum for the external and internal flatfield light sources (i.e., observations of the bright Earth or images taken with the internal WFPC2 VISFLAT lamp). The color of the Earth can vary considerably, depending upon the feature observed (land, sea, or clouds). The color of the internal VISFLAT lamp is known to vary as a function of the position in the field of view, the total lamp ``on'' time, and the total number of times the lamp has been cycled on and off. Furthermore, since the linear ramp filters are far from the focal plane (the OTA beam has a diameter of approximately 33 arcseconds at the filter), any dust spots and other small imperfections in the filter have essentially no effect on the data. Any large-scale variations in the filter are contained in the filter transmission curves and are corrected during photometric calibration. Moreover, some of the ramps have pinholes and if LRF flats were to be made, they would unnecessarily degrade the science data.

Observers with LRF data are advised to check what flatfield has been applied to their data and if necessary, select an existing narrow band flatfield reference file close in wavelength to their LRF science observation (the header parameter LRFWAVE records the wavelength of the image) and manually recalibrate using calwp2. Since there are no narrow band filters near 8000 , the best alternative at these wavelengths is to use the F791W flatfield reference file. The WWW Reference File memo or the Archive's StarView can be used to peruse the flatfields available in neighboring wavelength regimes.

3.5.3 Removing Cosmic Rays from Co-aligned Images

WFPC2 images typically contain a large number of cosmic ray events, which are caused by the interaction of galactic cosmic rays and protons from the Earth's radiation belt with the CCD. Hits occur at an average rate of about 1.8 events s-1 per CCD (1.2 s-1 cm-2), with an overall variation in rate of 60% (peak-to-peak) depending upon geomagnetic latitude and position with respect to the South Atlantic Anomaly (also see the WFPC2 Instrument Handbook, v. 6.0, pages 51-52).

Unlike events seen on the ground, most WFPC2 cosmic ray events deposit a significant amount of charge in several pixels; the average number of pixels affected is 6, with a peak signal of 1500 e- per pixel and a few tens of electrons per pixel at the edges. About 3% of the pixels, or 20,000 pixels per CCD, will be affected by cosmic rays in a long exposure (1800 seconds). Figure 3.6, shows the impact of cosmic rays in an 800 second exposure with WFPC2. The area shown is about 1/16th of one chip (a 200x200 region); pixels affected by cosmic rays are shown in black and unaffected pixels are shown in white. A typical long WFPC2 exposure (2000 seconds) would have about 2.5 times as many pixels corrupted by cosmic rays.

Cosmic rays are noticeable even for very short exposures. The WFPC2 electronics allow activities to be started only at one-minute intervals; thus, a minimum-length exposure will collect at least one minute's (the interval between camera reset and readout) worth of cosmic rays, and will be affected by about a hundred of them per CCD.

As a result of the undersampling of the WFPC2 PSF by the WF and PC pixels, it is very difficult to differentiate stars from cosmic rays using a single exposure. If multiple co-aligned images are available, cosmic rays can be removed simply and reliably by comparing the flux in the same pixel in different images, assuming that any differences well above the noise are positive deviations due to cosmic rays. STSDAS tasks such as crrej and gcombine can identify and correct pixels affected by cosmic rays in such images. (The task crrej has been significantly improved since the previous edition of the HST Data Handbook and is now the recommended choice.) If the images are shifted by an integral number of pixels, they can be realigned using a task such as imshift. Because each CCD is oriented differently on the sky, this operation will need to be done on one group at a time, using the pixel shift appropriate for each CCD in turn.

Another consequence of the undersampling of WFPC2 pixels is that small pointing shifts will cause measurable differences between images at the same nominal pointing. These differences are especially noticeable in PC data, where offsets of only 10 mas can cause a difference between successive images of 10% or more near the edges of stellar PSFs. We recommend that users allow for such differences by using the multiplicative noise term included in the noise model of the cosmic ray rejection task (scalenoise for crrej and snoise for gcombine). For typical pointing uncertainties, a multiplicative noise of 10% is adequate (note, this is specified as 10 for scalenoise and 0.1 for snoise). It is also strongly recommended that an image mask be generated for each image, in order to determine if an undue concentration of cosmic rays are identified near point sources-usually an indication that the cores of point sources are mistaken for cosmic rays. Detailed explanations of crrej and gcombine can be found in the on-line help.

Figure 3.6: WF Exposure Showing Pixels Affected by Cosmic Rays

Because sub-pixel dithering strategies are now very common, the image combination tasks in the drizzle package (see section 5.5) include a script that can remove cosmic rays from images taken at multiple pointings.

How many Images for Proper CR Rejection?

Cosmic rays are so numerous in WFPC2 data that double hits are not uncommon. For example, the combination of two 2000 second images will typically contain about 500 pixels per CCD that are affected by cosmic rays in both images; in most of these cases, the hit will be marginal in one of the two images. If the science goals require a high level of cosmic ray rejection, it will be desirable to conduct a more stringent test in pixels adjacent to detected cosmic rays (see the parameter radius in crrej). A better solution would be to break the observation into more than two exposures during the planning stage; the WFPC2 Exposure Time Calculator gives specific recommendations on the number of exposures necessary as a function of the number of pixels lost. In general, three exposures are sufficient for non-stringent programs, and four exposures for any program. The Exposure Time Calculator is available via the WWW.

3.5.4 Charge Traps

There are about 30 pixels in WFPC2 which do not transfer charge efficiently during readout, producing artifacts that are often quite noticeable. Typically, charge is delayed into successive pixels, producing a streak above the defective pixel. In the worst cases, the entire column above the pixel can be rendered useless. On blank sky, these traps will tend to produce a dark streak. However, when a bright object or cosmic ray is read through them, a bright streak will be produced. Figure 3.7 shows examples of both of these effects. (Note that these "macroscopic" charge traps are different from the much smaller traps believed to be responsible for the charge transfer effect discussed under Charge Transfer Efficiency in section 5.2.2.)

The images in figure 3.7 show streaks (a) in the background sky and, (b) stellar images produced by charge traps in the WFPC2. Individual traps have been cataloged and their identifying numbers are shown.

Figure 3.7: Streaks in a) Background Sky, and b) Stars

Bright tails have been measured on images taken both before and after the April 23, 1994 cool down and indicate that the behavior of the traps has been quite constant with time; fortunately, there is no evidence for the formation of new traps since the ground system testing in May 1993. The charge delay in each of the traps is well characterized by a simple exponential decay which varies in strength and spatial scale from trap to trap.

The positions of the traps, as well as those of pixels immediately above the traps, are marked in the .c1h data quality files with the value of 2, indicating a chip defect. Obviously, these pixels will be defective even in images of sources of uniform surface brightness. However, after August 1995, the entire column above traps has been flagged with the value of 256, which indicates a "Questionable Pixel." An object with sharp features (such as a star) will leave a trail should it fall on any of these pixels.

In cases where a bright streak is produced by a cosmic ray, standard cosmic ray removal techniques will usually remove both the streak and the cosmic ray. However, in cases where an object of interest has been affected, the user must be more careful. While standard techniques such as wfixup will interpolate across affected pixels and produce an acceptable cosmetic result, interpolation can bias both photometry and astrometry. In cases where accurate reconstruction of the true image is important, modelling of the charge transfer is required. For further information on charge traps, including the measured parameters of the larger traps, users should consult WFPC2 ISR 95-03, available on the WFPC2 web pages.

Space Telescope Science Institute
Voice: (410) 338-1082