Space Telescope Science Institute
WFC3 Data Handbook 2.1 May 2011
help@stsci.edu
Table of Contents Previous Next Index Print


WFC3 Data Handbook > Chapter 3: WFC3 Data Calibration > 3.4 Description of Calibration Steps

3.4
The calwf3 pipeline consists of four individual calibration tasks: wf3ccd, wf32d, wf3ir, and wf3rej. These tasks are diagrammed in Figure 3.1. calwf3 is responsible for controlling the processing rather than actually calibrating the data. The individual tasks apply the desired calibration steps to the data and create the output products, including the trailer files, which record a processing log.
In the following four sections, we describe each calwf3 task, give a detailed description of the calibration steps performed within each task, and give a brief description of the reference files used for each step.
calwf3 can be run on a single input raw file or an asn table listing the members of an association. When processing an association, calwf3 retrieves calibration switch and reference file keyword settings from the first image listed in the asn table. calwf3 does not accept a user-defined list of input images on the command line (e.g., *_raw.fits” to process all raw files in the current directory). The wf3ccd, wf32d, and wf3ir tasks, on the other hand, will accept such user-defined input file lists, but they will not accept an association table (asn) as input.
3.4.1
This routine contains the initial processing steps for all WFC3 UVIS channel data. These steps are listed in operational order in Table 3.4. The calibration switch keywords and reference file keywords that are used for these steps are listed in Figure 3.2. Only those steps with switch values of “PERFORM” in the input _raw.fits files will be executed. Each such switch value will be set to “COMPLETE” in the corresponding output files.
Input to wf3ccd is an image list or single image that is either automatically called by calwf3 or input directly by the user. wf3ccd processes each image in the input list one at a time, using the header keywords to determine which calibration steps are to be performed and which calibration reference files to use in each step. It also processes the image data from both CCD chips contained in the input file. Upon completion of wf3ccd, the overscan regions will be trimmed from the image and the blv_tmp output image is created. A description of each step follows.
Table 3.4: wf3ccd processing steps.
UVIS Error Array Initialization
-
-
-
Reference File: CCDTAB (*_ccd.fits)
First, the image error array is initialized. The function examines the ERR extension of the input data to determine the state of the array. Input raw images delivered by OPUS Generic Conversion contain a null (empty) ERR array, defined by the keywords NPIX1, NPIX2 and PIXVALUE, where PIXVALUE=0. If the ERR array has already been expanded and contains values other than zero, then this function does nothing. Otherwise, it will initialize the ERR array by assigning pixel values based on a simple noise model.
The noise model uses the science (SCI) array and for each pixel calculates the error value σ (in units of DN):
The CCDTAB reference file, the CCD Characteristics Table, contains the bias, gain, and readnoise values for each CCD amplifier quadrant used in this calculation. The table contains one row for each configuration that can be used during readout, which is uniquely identified by the list of amplifiers (CCDAMP), the particular chip being read out (CCDCHIP), the commanded gain (CCDGAIN), the commanded bias offset level (CCDOFST), and the pixel bin size (BINAXIS). These commanded values are used to find the table row that matches the characteristics of the image that is being processed and reads each amplifier’s characteristics, including readnoise (READNSE), A-to-D gain (ATODGN), and mean bias level (CCDBIAS).
UVIS Data Quality Array Initialization
-
-
-
Reference Files: BPIXTAB (*_bpx.fits), CCDTAB (*_ccd.fits)
DQICORR initializes the data quality (DQ) array by reading a table of known bad pixels for the detector, stored in the “Bad Pixel” reference table (BPIXTAB). The types of bad pixels that can be flagged are listed in Table 2.5.
The DQ array may already have been populated with some values to flag pixels affected by telemetry problems during downlink. Other DQ values will only be marked during further processing (such as cosmic-ray rejection). This function also checks pixel values in the SCI extension for saturation, using the value of the SATURATE column in the CCD parameters table (CCDTAB). Any SCI array pixel value that is greater than the SATURATE value will be assigned the appropriate flag value in the DQ array. This function also checks for SCI array pixel values that have reached the limit of the detector’s 16-bit A-to-D converters, flagging any pixel with a value > 65534 DN with the “A-to-D saturation” DQ value.
DQICORR combines the DQ flags from preprocessing, BPIXTAB, and saturation tests into a single result for the particular observation. These values are combined using a bit-wise logical “OR” operation for each pixel. Thus, if a single pixel is affected by two DQ flags, those flag values will be added in the final DQ array. This array then becomes a mask of all pixels that had some problem coming into the calibrations, so that the calibration processing steps can ignore bad pixels during processing.
The BPIXTAB reference file maintains a record of the x, y position and DQ value for all known bad pixels in each CCD chip for a given time period.
UVIS A-to-D Conversion Correction
-
-
-
Reference File: ATODTAB (*_a2d.fits)
An analog-to-digital conversion correction is applied if the CCD electronic circuitry, which performs the analog-to-digital conversion, is biased toward the assignment of certain DN values. WFC3 ground test results show that this correction is not currently needed, so the ATODCORR switch is currently always set to “OMIT” so that this function is not performed.
UVIS Bias Level Correction
-
-
Header Keywords Updated: BIASLEV[ABCD], MEANBLEV
-
Reference File: OSCNTAB (*_osc.fits)
BLEVCORR fits the bias level in the CCD overscan regions and subtracts it from the image data. The boundaries of the overscan regions are taken from the OSCNTAB reference file. With these regions defined, the serial and parallel virtual overscans are analyzed to produce a two-dimensional linear fit to the bias level. The overscan level for each row of the input image is measured within the serial virtual overscan region, utilizing sigma-clipping to reject anomalous values (e.g., cosmic-ray hits that occur in the overscan) and a straight line is fit as a function of image line number. The same procedure is followed for the parallel overscan, resulting in a straight line fit as a function of image column number. The parallel fit is computed in the form of a correction to be added to the serial fit result, in order to remove any gradient that may exist along the x-axis direction of the image. The serial fit and the parallel correction to it are then evaluated at the coordinates of each pixel and the computed bias value is subtracted from the pixel. This is done independently for each region of the image that was read out by one of the four CCD amplifiers. The mean bias value determined for each of the amplifier quadrants is recorded in the primary header keywords BIASLEV[ABCD] and the overall mean bias value is computed and written to the output SCI extension header as MEANBLEV.
UVIS subarray images do not include virtual overscan, therefore the serial physical overscan will be used - if present - to perform the bias subtraction. If a subarray image does not include the physical overscan region of the detector, then the bias level cannot be determined. In this case a default value (CCDBIAS from the CCD parameters table) will be subtracted instead and a warning message is written to the processing trailer file.
The full bias level-subtracted image is retained in memory until the completion of all the processing steps in wf3ccd. The overscan regions will not be trimmed until the image is written to disk at the completion of wf3ccd.
The OSCNTAB reference file (Overscan Region Table) describes the overscan regions for each chip along with the regions to be used for determining the actual bias level of the observation. Each row corresponds to a specific configuration, given by the CCD amplifier, chip, and binning factor used. The OSCNTAB columns BIASSECTAn and BIASSECTBn give the range of image columns to be used for determining the bias level in the leading and trailing regions, respectively, of the serial physical overscan regions, while columns BIASSECTCn and BIASSECTDn give the range of columns to be used for determining the bias level from the serial virtual overscan regions. The parallel virtual overscan regions are defined in the OSCNTAB in the VXn and VYn columns.
To determine which overscan regions were actually used for measuring the bias level, check the OSCNTAB reference file. Users may modify the overscan region definitions in the reference table for manual calibration, but the TRIMXn and TRIMYn values must not be changed.
UVIS Bias Image Subtraction
-
-
-
Reference File: BIASFILE (*_bia.fits)
BIASCORR subtracts the superbias reference image. The reference image, BIASFILE, must have the same values of DETECTOR, CCDAMP, CCDGAIN, and BINAXISi as the image being processed. The dimensions of the science image are used to distinguish between full- and sub-array images. Because the bias image is already overscan-subtracted, it will have a mean pixel value of less than one.
Dark counts accumulate for an additional time beyond the exposure time, due to the time required to read out the detector, and this portion of the dark current is subtracted along with the bias. This is described further in the section on Dark Image Subtraction.
The BIASFILE has the same dimensions as a full-size science image complete with overscan regions (4206 2070 per chip for an unbinned image). Only after the completion of wf3ccd are the science images trimmed to their final calibrated size (4096 2051 per chip for an unbinned image). A BIASFILE with a binning factor that matches the science data must be used. For sub-array images, however, it is not necessary to use a matching sub-array BIASFILE. calwf3 will extract the matching region from the full-size BIASFILE and apply it to the sub-array input image.
UVIS Post-flash Subtraction
-
-
-
Reference File: FLSHFILE (*_fls.fits)
WFC3 has a post-flash capability to provide a means of mitigating the effects of Charge Transfer Efficiency (CTE) degradation.
This function subtracts the post-flash reference image, FLSHFILE, from the science image. This file has the same dimensions as a full-size science image complete with overscan regions. The appropriate FLSHFILE has matching values of the following keywords from the image header: DETECTOR, CCDAMP, CCDGAIN, FLASHCUR, BINAXISi, and SHUTRPOS.
The success of the post-flash operation during the exposure is first verified by checking the keyword FLASHSTA. If any problems were encountered, a comment will be added to the history comments in the SCI extension header. The FLSHFILE is renormalized to the appropriate post-flash current level (LOW, MED, HIGH), given by the FLASHCUR keyword, and the flash duration (FLASHDUR) and is then subtracted from the science image. The mean value of the scaled post-flash image is written to the output SCI extension header in the keyword MEANFLSH.
At this time, the post-flash hardware capability is not enabled. Therefore, FLSHCORR is always set to “OMIT” and this step is skipped.
wf3ccd Final Output
-
Header Keywords Updated: CRPIX[1,2], LTV[1,2], SIZAXIS[1, 2]
If BLEVCORR was performed, the overscan regions are trimmed from the image when it is written out to the blv_tmp file. Otherwise, the full image array is written out. The keywords CRPIXi, LTVi, and SIZAXISi are updated in the output image extensions to reflect the offset of the image origin and the reduction in image size due to removing the overscan. The OSCNTAB reference table columns TRIMXn give the number of columns to trim off the beginning, end, and middle of each line (the serial physical and virtual overscan regions), while the TRIMYn columns give the number of rows to trim off the top and bottom of each column (the parallel virtual overscan region) when the overscan-trimmed image is written to disk.
If multiple images from a CR-SPLIT or REPEAT-OBS set are being processed, the blv_tmp files are sent to the wf3rej task to be combined. The resulting combined image (crj_tmp) is then sent to wf32d for final calibration. If multiple images are not being combined, the blv_tmp files are sent directly to wf32d for final calibration.
3.4.2
The wf32d primary functions are listed in Table 3.5 and include dark current subtraction, flat-fielding, and photometric keyword calculations. The calibration switch and reference file keywords used by these steps are listed in Figure 3.3. Only those steps with a switch value of “PERFORM” in the input files will be executed, after which the switch value will be set to “COMPLETE” in the corresponding output files.
wf32d contains the same ERR and DQ array initialization functions used in wf3ccd, but wf32d will check to ensure that these functions are not performed twice on the data. Calibration switches in the image header control the performance of the remaining calibration functions.
Table 3.5: The functions performed in wf32d (in operational order).
Apply a simple noise model, if not done in wf3ccd
Initialize data quality array, if not done in wf3ccd
UVIS Error Array Initialization
-
-
-
Reference File: CCDTAB (*_ccd.fits)
wf32d first checks to see if the image ERR array has already been populated, indicating that previous processing has been performed. If not, wf32d performs the same initialization as described for wf3ccd. If the input image has already been processed this step is skipped and no changes are made to the ERR array.
UVIS Data Quality Array Initialization
-
-
-
Reference Files: BPIXTAB (*_bpx.fits), CCDTAB (*_ccd.fits)
If the DQICORR header keyword switch is set to “COMPLETE”, this step will be skipped. Otherwise, the same initialization will be performed as described for wf3ccd.
UVIS Dark Image Subtraction
-
-
-
Reference File: DARKFILE (*_drk.fits)
This function is responsible for subtracting the dark current image from the input image. The dark image (in units of electrons/sec) is multiplied by the exposure time and divided by the gain before subtracting. The dark reference file, DARKFILE, is read in line-by-line and subtracted from the input image in memory. The mean dark value is computed from the scaled dark image and used to update the MEANDARK keyword in the SCI image header. The dark reference file will be updated frequently and will allow the tracking of hot pixels over time.
“Dark time” is simply the exposure time; it does not include the idle time since the last flushing of the chip or the readout time. Any dark accumulation during readout time is included automatically in the BIASFILE.
The reference file for dark subtraction, DARKFILE, is selected based on the values of the keywords DETECTOR, CCDAMP, and BINAXISi in the image header. The dark correction is applied after the overscan regions are trimmed from the input science image. As for the BIASFILE, calwf3 requires the binning factors of the DARKFILE and science image to match.
Sub-array science images use the same reference file as a full-sized DARKFILE. calwf3 simply extracts the appropriate region from the reference file and applies it to the sub-array input image.
UVIS Flat-Field Correction
-
-
-
Reference Files: PFLTFILE (*_pfl.fits), LFLTFILE (*_lfl.fits), DFLTFILE (*_dfl.fits)
This routine corrects for pixel-to-pixel and large-scale sensitivity variations across the detector by dividing the overscan-trimmed and dark-subtracted science image by a flat-field image. When performing the flat-field correction, calwf3 also multiplies by the gain so that the calibrated data will now be in units of electrons.
Because of geometric distortion effects, the area of the sky seen by different pixels is not constant and therefore observations of a constant surface brightness object will have counts per pixel that vary over the detector, even if every pixel were to have the same intrinsic sensitivity. In order to produce images that appear uniform for uniform illumination, the same counts per pixel variation across the field is left in place in the flat-field images, so that when a science image is divided by the flat it makes an implicit correction for the distortion. A consequence of this procedure is that two point-source objects of equal brightness will not have the same total counts after the flat-fielding step. Thus, point source photometry extracted from a flat-fielded image must be multiplied by the effective pixel area map. This correction is automatically included in pipeline processing by MultiDrizzle, which uses the geometric distortion solution to correct all pixels to equal areas. Photometry is therefore correct for both point and extended sources in drizzled images.
Up to three separate flat-field reference files can be applied: the pixel-to-pixel flat-field file (PFLTFILE), the low-order flat-field file (LFLTFILE), and the delta flat-field file (DFLTFILE). The PFLTFILE is a pixel-to-pixel flat-field correction file containing the small-scale flat-field variations. Unlike the other flat fields, the PFLTFILE is always used in the calibration pipeline. The LFLTFILE is a low-order flat that corrects for any large-scale sensitivity variations across each detector. This file can be stored as a binned image, which is then expanded when being applied by calwf3. Finally, the DFLTFILE is a delta-flat containing any needed changes to the small-scale PFLTFILE.
If the LFLTFILE and DFLTFILE are not specified in the SCI header, only the PFLTFILE is used for the flat-field correction. If two or more reference files are specified, they are read in line-by-line and multiplied together to form a combined flat-field correction image.
All flat-field reference images must have detector, amplifier, filter, and binning modes that match the observation. A sub-array science image uses the same reference file as a full-size image; calwf3 extracts the appropriate region from the reference file and applies it to the sub-array input image.
UVIS Shutter Shading Correction
-
-
-
Reference File: SHADFILE (*_shd.fits)
The SHADCORR routine applies the shutter shading correction image (SHADFILE) to the science data. This corrects the input image for the differential exposure time across the detector caused by the amount of time it takes for the shutter to completely open and close, which is a potentially significant effect only for images with very short exposure times (less than ~5 seconds).
Pixels are corrected based on the exposure time using the relation:
.
The SHADFILE is selected using the DETECTOR keyword in the input science image. This reference file is normally binned, because the correction varies slowly across the image.
The shutter shading correction can be applied either during wf32d processing for single exposures or during cosmic-ray rejection in wf3rej for CR-SPLIT and REPEAT-OBS exposures.
WFC3 tests have shown that the shutter shading effect is insignificant (<1%), even for the shortest allowed UVIS exposure time of 0.5 seconds (see WFC3 ISR 2007-17). Therefore this step is currently always set to OMIT in calwf3 pipeline processing.
UVIS Photometry Keyword Calculation
-
-
Header Keywords Updated: PHOTMODE, PHOTFLAM, PHOTFNU, PHOTZPT, PHOTPLAM, PHOTBW
-
Reference Files: GRAPHTAB (*_tmg.fits), COMPTAB (*_tmc.fits)
Before photometry can be derived from WFC3 observations, a transformation to absolute flux units must be done. calwf3 follows the WFPC2 and ACS methodology for calculating the photometry keywords in the calibration pipeline. The calibration reference files, GRAPHTAB and COMPTAB, point to the synphot tables containing the latest WFC3 component throughputs (please refer to Section 5.2.1 for details on how to set up environment variables that give the location of these files on your system). These tables contain the throughput as a function of wavelength for the various WFC3 detector and filter combinations. Using synphot allows the WFC3 team to maintain the latest throughput files in synphot to keep calwf3 up to date. For further discussion of synphot, refer to Chapter 3 of the Introduction to the HST Data Handbooks.
During this process the keyword PHOTMODE is built to reflect the configuration of the instrument for the exposure (e.g., “WFC3,UVIS1,F814W”). calwf3 then uses the PHOTMODE string with synphot to compute the total throughput for this instrument mode, based on the optics and filter throughputs and the detector QE. From that information, it computes values for the following photometry keywords.
PHOTFLAM: the inverse sensitivity in units of erg cm-2 A-1 electron-1
PHOTFNU: the inverse sensitivity in units of Jy sec electron-1
PHOTZPT: the Space Telescope magnitude zero point
PHOTPLAM: the bandpass pivot wavelength
PHOTBW: the bandpass RMS width
Users who wish to convert calibrated images (which are in units of electrons) to flux units can simply divide the image by the exposure time and then multiply by the PHOTFLAM keyword value. Drizzled (drz) images are already in units of electrons per second and therefore only need to be multiplied by the PHOTFLAM value to obtain flux units.
UVIS Image Statistics Calculation
-
-
Header Keywords Updated: NGOODPIX, GOODMIN, GOODMAX, GOODMEAN, SNRMIN, SNRMAX, SNRMEAN
-
This routine computes the minimum, mean, and maximum, as well as the minimum, mean, and maximum signal-to-noise ratio (the ratio of the SCI and ERR pixel values) for data values that are flagged as “good” in the data quality array. These quantities are updated in the SCI image header. The minimum, mean, and maximum statistics are also computed for the ERR array.
3.4.3
This routine contains all the instrumental calibration steps for WFC3 IR channel images. The steps are listed in operational order in Table 3.6. The calibration switch and reference file keywords used by these steps are listed in Figure 3.4. Only those steps with a switch value of “PERFORM” in the _raw.fits files will be executed, after which the switch value will be set to “COMPLETE” in the corresponding output files.
Input to wf3ir is an image list or single image that is either automatically called by calwf3 or input directly by the user. wf3ir processes each image in the input list one at a time, using the header keywords to determine which calibration steps are to be performed and which calibration reference files to use in each step.
The process begins working with the raw IR image file, which contains all of the non-destructive readouts for an exposure. Most of the calibration steps are applied independently to each readout. For example, the DQICORR, NLINCORR, and FLATCORR steps apply the same bad pixel flags, non-linearity correction coefficients, and flat-field image, respectively, to each readout. The CRCORR step, on the other hand, which attempts to remove the effects of cosmic rays, utilizes the values from all readouts of individual pixel simultaneously. Detailed descriptions of each step are provided in the following sections.
All steps up through UNITCORR are applied to an in-memory image stack that contains all the readouts. The CRCORR step produces an additional single image that gives the best-fit count rate for each pixel. The remaining steps in the process - FLATCORR and image statistics - are then applied to the full stack of readouts and to the single image produced by CRCORR.
Upon completion of wf3ir, two output files are produced. The Intermediate MultiAccum (ima) file, which contains the full stack of calibrated readouts, and the final calibrated image (flt) file, which is the single image produced by CRCORR (with subsequent flat-fielding applied). The flt file has the reference pixel regions trimmed from the image, so that it is appropriate to use in further processing, such as MultiDrizzle.(Note: although the flt images are normally flat-fielded, this is only the case if the flat-fielding step FLATCORR is performed.)
Note that the image data associated with the non-destructive readouts of an IR exposure are stored in reverse time order in the input raw and output ima files (see Section 2.2.2). The last readout of the exposure is therefore stored in imset 1 (e.g., [sci,1]), while the first readout is in imset NSAMP (e.g.,[sci,16]for NSAMP=16). It is useful in this context to think of the exposure data literally accumulating “from the bottom up” in the file.
Table 3.6: wf3ir processing steps.
IR Data Quality Array Initialization
-
-
-
Reference Files: BPIXTAB (*_bpx.fits)
DQICORR populates the data quality (DQ) array in all IR readouts by reading a table of known bad pixels for the detector, stored in the “Bad Pixel” reference table (BPIXTAB). The types of bad pixels that can be flagged are listed in Table 2.5.
The DQ array may already have been populated with some values to flag pixels that were affected by telemetry problems during downlink. Other DQ values will only be marked during further processing (such as cosmic-ray rejection).
This function also checks to see if the HST Take Data Flag (TDF) went down during any readout, as recorded in the TDFTRANS header keyword. If so, then all pixels in the affected readouts are flagged as bad, which will prevent them from being used to compute a final image value in the CRCORR step.
The reference file for data quality initialization, BPIXTAB, is selected based on the value of the DETECTOR keyword only.
IR Zero-Read Signal Correction
-
-
-
Reference Files: DARKFILE (*_drk.fits), NLINFILE (*_lin.fits)
At the beginning of an IR observation the detector pixels are reset to the bias level and then read out to record that bias level. An interval of approximately 2.9 seconds elapses between the time each pixel is reset and then read. Because the IR channel does not have a shutter, signal from external sources starts to accumulate during that 2.9 second interval. When the initial (or “zeroth”) read is later subtracted from subsequent readouts, any signal in the zeroth read will also be subtracted. Because linearity correction and saturation checking (described below) both depend on the absolute signal level in a pixel at the time it was read, the signal in the zeroth read from bright sources can be large enough to lead to inaccurate linearity corrections, as well as the failure to detect saturation conditions, in the NLINCORR calibration step.
The ZSIGCORR step is used to estimate the amount of source signal in the zeroth read and to supply this estimate to the NLINCORR step for linearity corrections and saturation checking. ZSIGCORR estimates the signal in the zeroth read by first measuring the signal in each pixel between the zeroth and first reads, and then scaling that signal to the effective exposure time of the zeroth read (nominally 2.9 seconds). Pixels that have an estimated zeroth read signal greater than 5 times their estimated uncertainty (noise) value are assumed to contain detectable signal; those below this threshold are ignored. The estimated zeroth read signal is then passed, as an in-memory image, to the NLINCORR step, which accounts for that signal when applying linearity corrections and saturation checking on the zeroth-read subtracted images.
Note that this technique will not work well for pixels covered by targets that are so bright that the signal is already beginning to saturate in either the zeroth or first readouts, because then it is difficult to accurately estimate the zeroth-read signal. ZSIGCORR therefore checks for saturation in the zeroth and first read images and flags those pixels.
Pixels that are determined to have detectable signal in the zeroth read are flagged in the DQ arrays of the output ima file with a data quality value of 2048.
The reference files used in this step, DARKFILE and NLINFILE, are selected from CDBS based on the values of the DETECTOR, CCDAMP, CCDGAIN, SAMP_SEQ, and SUBTYPE keywords. The DARKFILE file is used to subtract dark current from the first-minus-zero read difference image before using it to estimate incoming signal levels and the NLINFILE is used to perform saturation checking.
IR Bias Level Correction From Reference Pixels
-
-
-
Reference Files: OSCNTAB (*_osc.fits)
BLEVCORR uses the reference pixels located around the perimeter of the IR detector to track and remove changes in the bias level that occur during an exposure. For each raw readout, the mean signal level of the reference pixels is computed and subtracted from the image, and recorded in the MEANBLEV keyword in the SCI header of each readout.
The reference pixels located at the ends of each image row are used in this computation. Reference pixels are also located along the bottom and top of the image, but these have been found to be less reliable and are not used. As with the UVIS overscan correction, the boundaries of the reference pixel regions that are used in the computation are defined in the OSCNTAB reference table, in the BIASSECT* columns. The BIASSECTA[1,2] values indicate the starting and ending column numbers for the reference pixels on the left edge of the image, and the BIASSECTB[1,2] give the values for the right side of the image.
The reference pixel regions are maintained throughout the remainder of processing, but are usually ignored or skipped over in the actual application of calibration algorithms. They are left in place in the calibrated data stored in the ima file at the end of processing, but are trimmed from the flt image file.
The reference file for bias level correction, OSCNTAB, is selected from CDBS based on the value of the DETECTOR keyword only.
IR Zero-read Image Subtraction
-
-
-
ZOFFCORR subtracts the zeroth read from all readouts in the exposure, including the zeroth read itself, resulting in a zero-read image that is exactly zero in the remainder of processing. The zeroth-read image is propagated through the remaining processing steps and included in the output products, so that a complete history of error estimates and data quality (DQ) flags is preserved.
Note: In interpreting the IR Intermediate MultiAccum (ima) file, it is important to remember the file does not represent differences in adjacent reads, but always the difference between a given readout and the zero read. The signal rate recorded in each SCI extension of the ima file represents the average flux between that particular readout and the zero read.
IR Error Array Initialization
-
-
-
Reference File: CCDTAB (*_ccd.fits)
This step computes an estimate of the errors associated with the raw science data based on a noise model for the detector. Currently the noise model is a simple combination of detector read noise and Poisson noise in the signal, such that:
where the read noise is in units of electrons, gain is the analog-to-digital conversion gain factor (in electrons per DN) and counts is the signal in a science image pixel in units of DN. The detector read noise and gain are read from the CCDTAB reference file and use separate values for the particular amplifier quadrant with which each pixel is associated.
Throughout the remaining calibration steps the ERR image is processed in lock-step with the science (SCI) image, getting updated as appropriate. Errors are propagated through combination in quadrature. The ERR array for the final calibrated “_flt” image is populated by the CRCORR step, based on the calculated uncertainty of the count rate fit to the MultiAccum samples (see below for details).
The CCDTAB reference file used in this step is selected based on the value of the DETECTOR keyword only.
IR Non-Linearity Correction
-
-
-
Reference File: NLINFILE (*_lin.fits)
NLINCORR corrects the integrated counts in the science images for the non-linear response of the detector and flags pixels that go into saturation. The observed response of the detector can be conveniently represented by two regimes:
At low and intermediate signal levels the detector response deviates from the incident flux in a way that is correctable using the following expression:
where c1, c2, c3, and c4 are the correction coefficients, F is the uncorrected flux (in DN) and Fc is the corrected flux. The current form of the correction uses a third-order polynomial, as shown here, but the algorithm can handle an arbitrary number of coefficients. The number of coefficients and error terms are given by the values of the NCOEF and NERR keywords in the header of the NLINFILE.
At high signal levels—as saturation sets in—the response becomes highly non-linear and is not correctable to a scientifically useful degree.
This step uses the NLINFILE reference file, which includes a set of images containing the cn correction coefficients and their variances at each pixel. The [NODE,1] image extension in the NLINFILE gives the saturation value for each pixel, in units of DN. Each pixel that has an input value below its defined saturation level is corrected according to the equation above. Pixels at or above their saturation values receive no correction and are flagged as saturated in the DQ array for the readout. Any pixel flagged as saturated in a given readout is also automatically flagged as saturated in all subsequent readouts.
As mentioned in the description of the ZSIGCORR routine, the estimated amount of signal in the zeroth read of the exposure is temporarily added back into the signal of each pixel during the NLINCORR step, before the pixel is checked for saturation or receives the linearity correction. Once the correction has been applied, the zero read signal is again removed. This process only occurs if the ZSIGCORR step is turned on during processing.
The NLINFILE reference files is selected based on the value of the DETECTOR keyword only.
IR Dark Image Subtraction
-
-
-
Reference File: DARKFILE (*_drk.fits)
DARKCORR subtracts the detector dark current from the science data. Due to potential non-linearities in some of the signal components, such as reset-related effects in the first one or two reads of an exposure, the dark current subtraction is not applied by simply scaling a generic reference dark image to the exposure time and then subtracting it. Instead, a library of dark current images is maintained that includes darks taken in each of the available predefined MultiAccum sample sequences, as well as the available sub-array readout modes. The MultiAccum dark reference file is subtracted read-by-read from the stack of science image readouts. Thus there is an exact match in the timings and other characteristics of the dark image that is subtracted from each science readout.
The DARKFILE reference file must have the same values for the DETECTOR, CCDAMP, CCDGAIN, SAMP_SEQ, and SUBTYPE keywords as the science image.
IR Photometry Keyword Calculation
-
-
Header Keywords Updated: PHOTMODE, PHOTFLAM, PHOTFNU, PHOTZPT, PHOTPLAM, PHOTBW
-
Reference Files: GRAPHTAB (*_tmg.fits), COMPTAB (*_tmc.fits)
Before photometry can be derived from WFC3 observations, a transformation to absolute flux units must be done. calwf3 follows the WFPC2 and ACS methodology for calculating the photometry keywords in the calibration pipeline. The calibration reference files, GRAPHTAB and COMPTAB, point to the synphot tables containing the latest WFC3 component throughputs (please refer to Section 5.2.1 for details on how to set up environment variables that give the location of these files on your system). These tables contain the throughput as a function of wavelength for the various WFC3 detector and filter combinations. Using synphot allows the WFC3 team to maintain the latest throughput files in synphot to keep calwf3 up to date. For further discussion of synphot, refer to Chapter 3 of the Introduction to the HST Data Handbooks.
During this process the keyword PHOTMODE is built to reflect the configuration of the instrument for the exposure (e.g., “WFC3,IR,F160W”). calwf3 then uses the PHOTMODE string with synphot to compute the total throughput for this instrument mode, based on the optics and filter throughputs and the detector QE. From that information, it computes values for the following photometry keywords.
PHOTFLAM: the inverse sensitivity in units of erg cm-2 A-1 electron-1
PHOTFNU: the inverse sensitivity in units of Jy sec electron-1
PHOTZPT: the Space Telescope magnitude zero point
PHOTPLAM: the bandpass pivot wavelength
PHOTBW: the bandpass RMS width
Users who wish to convert calibrated IR images (which are in units of electrons per second) to flux units can simply multiply the image by the PHOTFNU keyword value.
IR Unit Conversion
-
-
-
This function simply converts the science data from a time-integrated signal to a signal rate, by dividing the science (SCI) and error (ERR) image arrays for each readout by the exposure time (TIME) image data. No reference file is needed.
Usually, the final units will be electrons per second, but if certain steps in the standard processing are omitted, the final units in the ima and flt files may be electrons, counts, or counts per second, where counts refers to the digitized signal from the FPA. The BUNIT keyword in the output files will always reflect the units of the data.
IR Up-The-Ramp Fitting and Cosmic-Ray Identification
-
-
-
CRCORR combines the data from all readouts into a single image and in the process identifies and flags pixels suspected of containing cosmic-ray (CR) hits. The data from all readouts are analyzed pixel-by-pixel, iteratively computing a linear fit to the accumulating counts-versus-exposure time relation. Samples flagged as bad in the DQ arrays, such as when saturation occurs midway through the exposure, are rejected from the fitting process. CR hits are identified by searching for outliers from the fit results. The rejection threshold is set by the value in the “CRSIGMAS” column of the Cosmic-Ray Rejection parameters reference table (CRREJTAB), which currently has a default value of 4σ. When a CR hit is detected, a linear fit is then performed independently for the sets of readouts before and after the hit. Those fitting results are then again checked for outliers. This process is iterated until no new samples are rejected. Pixel samples identified as containing a CR hit are flagged in the DQ arrays of the intermediate MultiAccum (ima) file, with a DQ value of 8192. The pixel values in the SCI and ERR images of the ima file, however, are left unchanged.
Once all outliers have been identified, a final count rate value, and its uncertainty, are determined for each pixel by computing the weighted mean of the slopes of each segment of non-flagged samples. The result of this operation is stored as a single imset in the output flt file. In the flt file the SCI array contains the final slope computed for each pixel, the ERR array contains the estimated uncertainty in the slope, the SAMP array contains the total number of non-flagged samples used to compute the slope, and the TIME array contains the total exposure time of those samples.
Pixels for which there are no unflagged samples, e.g., permanently hot or cold pixels, still get a slope computed, which is recorded in the SCI array of the output flt file, but they will also have their DQ flags recorded in the DQ array of the flt file. Users should therefore be careful to always check the flt file DQ arrays to help determine whether a given SCI image value is trustworthy for subsequent analysis.
IR Flat-field Image Correction
-
-
-
Reference Files: PFLTFILE (*_pfl.fits), LFLTFILE (*_lfl.fits), DFLTFILE (*_dfl.fits)
FLATCORR corrects for pixel-to-pixel and large-scale sensitivity variations across the detector by dividing the science images by one or more flat-field images. A combined flat is created within calwf3 using up to three flat-field reference files: the pixel-to-pixel flat (PFLTFILE), the low-order flat (LFLTFILE), and the delta flat (DFLTFILE). FLATCORR also multiplies the science data by the detector gain so that the calibrated data will be in units of electrons per second (or electrons if UNITCORR is not performed).
The PFLTFILE is a pixel-to-pixel flat-field correction file containing the small-scale flat-field variations. The PFLTFILE is always used in the calibration pipeline, while the other two flats are optional. The LFLTFILE is a low-order flat that corrects for any large-scale sensitivity variations across the detector. This file can be stored as a binned image, which is then expanded when being applied by calwf3. Finally, the DFLTFILE is a delta-flat containing any needed changes to the small-scale PFLTFILE.
If the LFLTFILE and DFLTFILE are not specified in the SCI header, only the PFLTFILE is used for the flat-field correction. If two or more reference files are specified, they are read in and multiplied together to form a combined flat-field correction image.
The flat-field correction is applied to all readouts of the calibrated IR MultiAccum stack, as well as the single image produced by the CRCORR function.
All flat-field reference images are chosen from CDBS based on the DETECTOR, CCDAMP, and FILTER used for the observation. A sub-array science image uses the same reference file(s) as a full-size image; calwf3 extracts the appropriate region from the reference file(s) and applies it to the sub-array input image.
See the discussion of this step in “UVIS Flat-Field Image Correction” for information regarding corrections for geometric distortion.
IR Image Statistics Calculation
-
-
Header Keywords Updated: NGOODPIX, GOODMIN, GOODMAX, GOODMEAN, SNRMIN, SNRMAX, SNRMEAN
-
This routine computes the minimum, mean, and maximum, as well as the minimum, mean, and maximum signal-to-noise ratio (the ratio of the SCI and ERR pixel values) for data values that are flagged as “good” in the data quality array. These quantities are updated in the SCI image headers. The minimum, mean, and maximum statistics are also computed for the ERR arrays.
This operation is performed for every readout in the calibrated MultiAccum stack, as well as the final (CRCORR-produced) calibrated image.
3.4.4
Header Switch: CRCORR (UVIS), RPTCORR (IR)
Header Keywords Updated: BADINPDQ, CRMASK, CRRADIUS, CRSIGMAS, CRTHRESH, EXPEND, EXPSTART, EXPTIME, INITGUES, MEANEXP, NCOMBINE, REJ_RATE, ROOTNAME, SCALENSE, SKYSUB, SKYSUM
Reference File: CRREJTAB (*_crr.fits)
wf3rej, the cosmic-ray rejection and image combination task in calwf3, combines CR-SPLIT or REPEAT-OBS exposures into a single image, first detecting and then replacing flagged pixels. The task uses the same statistical detection algorithm developed for ACS (acsrej), STIS (ocrrej), and WFPC2 data (crrej), providing a well-tested and robust procedure.
First, wf3rej temporarily removes the sky background from each input image (if requested via the SKYSUB parameter in the CRREJTAB), usually computed using the mode of each image. Sky subtraction is performed before any statistical checks are made for cosmic rays. Next, wf3rej constructs an initial comparison image from each sky-subtracted exposure. This comparison image can either be a median- or minimum-value sky-subtracted image constructed from all the input images, and it represents the “initial guess” of a cosmic-ray free image. The comparison image serves as the basis for determining the statistical deviation of each pixel within the input images.
A detection threshold is then calculated for each pixel based on the comparison image.
where:
-
-
noise is the readnoise in DN squared and gain is the e/DN of the amplifier used to read the pixel,
-
scale is the scale factor for the noise model,
-
-
value is the pixel value (in DN) from the median or minimum combined comparison image.
The actual detection criterion for a cosmic ray is determined as:
where:
-
pixn is the pixel value from input image n,
-
skyn is the sky background of image n, and
-
median is the median or minimum pixel value from the comparison image.
If , the pixel is flagged as a cosmic ray in the input image’s DQ array and is ignored when images are summed together. Surrounding pixels within some expansion radius (CRRADIUS) are marked as “SPILL” pixels and are given less stringent detection thresholds.
When all input images have been processed, the values of the non-rejected pixels are summed over all input images. Each pixel in the summed output array is then scaled by the total exposure time:
where:
-
-
-
T is the total exposure time (regardless of whether all input images were used for that particular pixel). This corresponds to the value recorded in the header keywords TEXPTIME and EXPTIME.
The following keywords are also derived from the variables in this equation:
-
TEXPTIME = EXPTIME = T
-
-
REJ_RATE = averaged over all pixels
-
The remaining keywords EXPSTART, EXPEND are updated based on the values corresponding to the first and last input images, respectively.
In summary, the cosmic ray rejection task sums all non-rejected pixel values, computes the true exposure time for that pixel, and scales the sum to correspond to the total exposure time. The final scaled, cleaned pixel is written to the comparison image to be used for the next iteration. This process is then repeated with increasingly stringent detection thresholds, as specified by CRSIGMAS.
Cosmic Ray Rejection Table
wf3rej uses the Cosmic Ray Rejection parameter table (CRREJTAB) to determine the number of iterations for cosmic-ray rejection, the sigma levels to use for each iteration, and the spill radius to use during detection. This allows the rejection process to be tuned to each detector and observation, with suitable defaults being applied during pipeline processing. Observers may fine-tune the cosmic-ray rejection parameters when manually reprocessing data with wf3rej by editing the CRREJTAB.
The CRREJTAB reference file contains the basic parameters necessary for performing cosmic-ray rejection. The column names and default values for the CRREJTAB are given in Table 3.7. The appropriate row is selected based on the chip being processed (CCDCHIP), the number of images into which the exposure was split (CR-SPLIT), and the exposure time of each CR-SPLIT image (MEANEXP). If an exact match is not found for the exposure time, the table row with the closest value is used. If the CR-SPLIT value of the input images exceeds the values in the table, the table row with the largest CR-SPLIT value will be used. The sky fitting algorithm is controlled by the parameter SKYSUB, which can have values of “mode”, “mean” or “none”. The “initial guess” image is created using the median or minimum value of the input exposures, as specified by the value of INITGUES.
Cosmic-ray detection requires the specification of a threshold above which a pixel value is considered a cosmic ray. This threshold was defined above as and uses the sigma rejection thresholds . These sigmas correspond to the CRSIGMAS column values in the CRREJTAB file. SCALENSE is a multiplicative term (in percent) for the noise model and is given as scale in the threshold equation above. This term can be useful when the pointing of the telescope has changed by a small fraction of a pixel between images. Under such circumstances, the undersampling of the image by the detector will cause stars to be mistakenly rejected as cosmic rays if a scale noise term is not included. This is a crude but effective step taken to satisfy the maxim of “do no harm”. However, for cases in which there have been no image-to-image offsets or the image is locally well-sampled, this will unduly bias against rejecting cosmic rays.
Pixels within a given radius, CRRADIUS, of a cosmic ray will also be treated as cosmic rays. A less stringent rejection threshold, CRTHRESH, can be used for detecting pixels adjacent to a cosmic ray. As for CRSIGMAS, CRTHRESH is also given as a sigma value. If CRTHRESH is exceeded, pixels within the defined radius of the cosmic ray will also be flagged. All pixels determined to be affected by a cosmic ray will have their DQ values set to 8192, as described in Table 2.5.
Table 3.7: Columns in cosmic-ray rejection parameter table.

Table of Contents Previous Next Index Print