File Formats for WFPC2 Data (12 Aug 2009)
Reprocessing of the WFPC2 archive is nearly complete and as of August 3, 2009, 97% of WFPC2 data sets have been reprocessed and re-ingested into MAST (the HST Data archive). The remaining 3% are primarily images taken through the Linear Ramp Filters (LRFs) or polarizers, images taken through multiple filters, or low-bias WF4 images, which will be processed again with improved reference files.
Because most WFPC2 data have now been reprocessed, MAST has discontinued On-The-Fly Reprocessing (OTFR) for WFPC2 data and will supply calibrated images directly from a static archive. By default, these files will be delivered in multi-extension FITS (MEF) format, rather than the Waivered FITS (WFITS) format previously used for WFPC2 data. (Note that MEF format data ends with 'm.fits' while WFITS format data ends with 'f.fits') Users who still prefer to work with WFITS images can request this format when retrieving WFPC2 data using the web-based Archive Search Form. After you click the button to submit your archive request, you will be taken to the Retrieval Options page where, about 2/3 of the way down, you will see 'Deliver WFPC2 datasets in Waivered FITS format'. Click the button next to this option and you will receive your data in WFITS format.
Users who wish to recalibrate their data using calwp2 may use either WFITS files (converted to GEIS format on their home machines using the IRAF/STSDAS task, strfits) or MEF data files; directions for recalibrating data in MEF format are given in the WFPC2 MEF Recalibration document (PDF 110KB).
Final Calibration of WFPC2 (11 Nov 2008)
In preparation for the decommissioning of WFPC2, the WFPC2 team has undertaken a series of reference-file updates and software enhancements to the WFPC2 calibration pipeline:
WF4 Anomaly: Since 2002, a temperature-dependent reduction in the gain has plagued images obtained with the WF4 detector. Characterized by low or zero bias levels, faint horizontal streaks, and low photometry, the WF4 anomaly is thought to be caused by a failing amplifier in the WF4 signal-processing electronics. Software to correct for the anomaly has been added to CALWP2 and the associated reference files are now in place. The error contributed by the WF4 corrections on the photometry is 1 to 2 percent, which is comparable to all other error sources, so the effect on the total error is not large. A separate routine to remove the horizontal streaks that often appear in low-bias frames will be included in the next STSDAS release. The program, called "wfpc2destreak," is a stand-alone routine and not part of the pipeline. It can also remove the bias variations that occasionally appear in the other CCDs.
CTE Keywords: Three new keywords have been added to the WFPC2 file headers. CTE_1E2, CTE_1E3, and CTE_1E4 contain estimates of the charge-transfer losses (in units of delta mag) for a star at x = 400, y = 400 with intensities of 100, 1000, and 10,000 electrons, respectively. The program uses Andrew Dolphin's (2004) recipe, which is a function of star brightness, observation date, and background level. Note that the pipeline only populates these header keywords. No changes are made to the pixel values themselves.
UV contamination & QE changes with time: We have reanalyzed the calibration observations obtained to track the variation in UV throughput as contaminants condense on the cold CCD windows; and are later removed by periodic decontamination procedures. The result is an updated SYNPHOT contamination table. A new header keyword, ZP_CORR, gives the offset to PHOTFLAM needed to account for both short-term variations in UV throughput and long-term changes in the detector quantum efficiency as a function of the observation epoch and filter. Users will no longer have to run SYNPHOT to compute these corrections.
Photometric zeropoints: We are re-evaluating the zeropoints at the end of WFPC2's mission using new observations of standard stars and standard fields. We will update the SYNPHOT filter throughput tables and detector efficiency tables as needed.
Flat fields for broad-band filters: We evaluated the accuracy of the flat fields for broadband filters by a new technique: using the Earth illuminated by the full Moon as a diffuse light source. We found the existing flats to be quite accurate and plan no further updates to the flat fields.
Filter red leaks: We measured the leak component directly for the first time on-orbit by crossing the UV and blue filters with other filters and observing a bright standard star. We will use the results to update the SYNPHOT throughput tables.
Narrow-band filters: We made and analyzed new observations to assess the wavelength and photometric stability of the narrow-band filters over the 15-year WFPC2 mission. We will update the SYNPHOT throughput tables as needed.
Linear ramp filters: We made a new series of photometric calibrations that will produce substantially more accurate photometry in many wavelength ranges. We will incorporate the results in new SYNPHOT tables.
Polarizers: We have made new observations to reevaluate the calibrations of the polarizers near the mission end. The analysis of these data is pending.
Geometric distortion and astrometry: We reevaluated distortion corrections and generated new, time-dependent corrections for the offsets between the four CCD chips over the entire WFPC2 mission. We addressed the 34-row effect, where every 34th pixel row of the CCD is about 3% too narrow due to a manufacturing error. These have resulted in new geometry tables (IDCTAB, OFFTAB, and DGEOFILE) used for "drizzling" data onto a corrected grid. In addition, we added a new keyword, VAFACTOR, to the header of each image; it describes the effects of velocity aberration due to Hubble's orbital motion.
Bias and dark calibrations: We have generated new super-bias and super-dark reference files for the last few years of the WFPC2 mission. We are studying the long-term history of both the bias and dark calibrations.
WFPC2 Reprocessing and the Static Archive
At present, WFPC2 images are calibrated, or processed, each time they are requested from the Hubble archive, thus insuring use of the most recent calibration files and software. Once final versions of the calibration files and software are in place, this dynamic approach will no longer be necessary. The WFPC2 calibration pipeline will be run one last time for each image, and the results stored in a static archive. Subsequent data requests will be fulfilled from this archive, speeding delivery and reducing costs. Reprocessing of the WFPC2 data has already begun and should be completed early in 2009.
Data files will be archived in both the waiver FITS (WFITS) format traditionally used by WFPC2 and in the standard multi-extension FITS (MEF) format used by ACS and other recent HST instruments. Waiver FITS files will end in "f" as they do now (e.g., u9wj600im_c0f.fits), while MEF format files will end in "m" (e.g., u9wj600im_c0m.fits). Both the raw data and calibration reference files will continue to be available in the archive, should users wish to perform their own calibrations.
While reprocessing is under way, requests for data sets that have
already been reprocessed will be fulfilled using the new static
archive. For these data sets, files will be returned in WFITS format;
MEF files may be requested by entering the desired extension (c0m)
at the bottom of the MAST Retrieval Options page. Data sets that
have not yet been reprocessed will be calibrated using the standard
pipeline; for these data sets, only WFITS files are available. Once
reprocessing is complete, on-the-fly reprocessing will be discontinued,
and calibrated data will be supplied directly from the static
archive. You can monitor the status of the reprocessing effort
using this web page:
PyDrizzled Images from MAST
Users requesting WFPC2 data from the MAST archive now receive an additional data product, a quick-look mosaic of the images from all four detectors. The new files, labeled *drz.fits, are produced by the program PyDrizzle. The images are geometrically corrected and have a plate scale of 0.1" per pixel. Because PyDrizzle is run on each WFPC2 exposure independently, the program cannot perform sub-pixel sampling or cosmic-ray rejection. DRZ files are thus intended as quick-look images, rather than as science data products. DRZ files utilize a multi-extension FITS format, rather than the waiver FITS format of traditional WFPC2 data products.
While most WFPC2 image files are in units of counts, the units of WFPC2 drizzled images retrieved from MAST are counts per second. Always check the value of the header keyword BUNIT to be sure of the units. Note that the keyword PHOTFLAM in the DRZ file is taken from the first extension (usually the PC chip) of the calibrated data file.