WFC3 Data Handbook v.4
help@stsci.edu

Table of Contents Previous Next Index PDF


WFC3 Data Handbook > Chapter 3: WFC3 Data Calibration > 3.4 Pipeline Tasks

3.4
The following section contains information regarding the 5 high-level tasks that are called by the main program calwf3, namely wf3cte, wf3ccd, wf32d, wf3ir, wf3rej. These tasks can be called individually by the users from either the command line or within a python interpreter. They generally are “umbrella” tasks that can perform multiple data processing steps. However the user can control the processing flow (and limit it to individual steps) by properly setting the header keyword switches, as detailed in Sections 3.2 (UVIS) and 3.3 (IR).
3.4.1 wf3cte
This routine is used to correct for Charge Transfer Efficiency (CTE) losses that cause electrons that are generated on the UVIS channel to be trapped by detector defects during readout. More details on the CTE phenomenon are given in Chapter 6, see also the WFC3 CTE webpage.
The PCTETAB reference file contains extensions of calibration images and tables of parameters used during the CTE correction stage. The header of this file also contains parameters for the CTE correction algorithm. These parameters, as well as others defaults used by wf3cte to correct the data are stored in the output image headers. Users who wish to use other settings for the CTE correction algorithm can adjust the pertinent keywords in their dataset header and calwf3 will give them preference, see WFC ISR 2016-02 for more information.
wf3cte performs the CTE correction on raw data files. If the calibration step keyword PCTECORR is set to PERFORM then the CTE correction will be applied to the dataset. Some caveats for its use:
CTE corrections can ONLY be performed on raw data which has not been calibrated in any way.
Data which have BLEVCORR, BIASCORR or DARKCORR set to COMPLETE will be rejected.
The CTE correction step in is implemented only for FULL FRAME images in calwf3 v3.3, but v3.4 also corrects for CTE for the subarray apertures indicated in Table 3.5 (the calwf3 version used to process any data is stored in the header keyword "CAL_VER"). The primary distinction is that these apertures have physical overscan pixels included which are used to calculate a secondary bias subtraction for the image before the CTE is measured; a future version of calwf3 may enable CTE corrections for the remaining subarrays which don’t have physical overscan pixels, but is still being validated by the WFC3 science team.
Table 3.5: Sub-array modes for which CTE correction is enabled in calwf3 version 3.4. For sub-arrays not listed here, a workaround is available.
UVIS1-2K2A-SUB
UVIS1-2K2B-SUB
UVIS2-2K2C-SUB
UVIS2-2K2D-SUB
UVIS2-C1K1C-SUB
UVIS2-C512C-SUB
UVIS2-C512D-SUB
UVIS1-C512A-SUB
UVIS1-C512B-SUB
UVIS1-2K4-SUB
UVIS-QUAD-SUB
UVIS2-2K4-SUB
Calling the wf3cte executable produces a FITS file with the extension "_rac" appended. This contains only the CTE corrected data, no other calibrations have been performed.
Running wf3cte from a python terminal
In Python without TEAL:
from wfc3tools import wf3cte
wf3cte(filename)
In Python with TEAL:
from stsci.tools import teal
from wfc3tools import wf3cte
teal.teal('wf3cte')
Displaying output from wf3cte in a Jupyter Notebook
When calling wf3cte from a Jupyter notebook, informational text output from the underlying wf3cte.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3cte. As output is read from the underlying program, the wf3cte Python wrapper will call log_func with the contents of each line (print is an obvious choice for a log function, but this also provides a way to connect wf3cte to the Python logging system by passing the logging.debug function or similar).
If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.
Input Parameters for the Python interface
a single filename (iaa012wdq_raw.fits)
Command Line Options for the wf3cte executable
wf3cte.e input [-options]
Where the options include:
Basic Steps In The CTE Correction
The reference bias image named in the BIACFILE header keyword is subtracted from the data
Parameters from the CTE parameter table, referenced in the PCTETAB header keyword, are read and stored
The data are reformatted: each quadrant is rotated such that the readout amp is located at the lower left of the array. The reoriented four quadrants are then arranged into a single 8412 2070 image (including the overscan pixels) with amps CDAB in that order. In this format, the pixels are all parallel-shifted down, then serial-shifted to the left
The PCTETAB and Algorithm Parameters
The following are new primary header keywords which will be updated in the data headers during the wf3cte step. They are also specified in the PCTETAB reference file.
The PCTETAB reference file has 4 extensions, two tables and two images:
Filename: zcv2057mi_cte.fits
No. Name Type Cards Dimensions Format
0 PRIMARY PrimaryHDU 21 ()
1 QPROF BinTableHDU 16 999R x 4C ['i', 'j', 'e', '20A']
The first extension lists the charge-trap levels, the columns are respectively the trap number, the charge-packet size it applies to (in electrons), and the size of the trap (also in electrons).
The second extension contains the CTE scalings as a function of column number. There are 5 columns, each with 8412 elements. The first column contains the integer column number in the amp readout-aligned large array. The other columns contain the CTE scaling appropriate for that column at the 512th, 1024th, 1536th and 2048th rows respectively.
The third extension contains the differential CTE trail profile as a function of charge level in the form of an image
The fourth extension contains the cumulative CTE trail profile as a function of charge level, also in the form of an image.
Output Files
If the user is running the separate wf3cte.e step a _rac.fits file will be produced. This is the same as a _raw.fits file except the CTE correction has been applied to the data.
If the PCTECORR step is set to PERFORM:
when the _raw.fits file enters calwf3, then no intermediate _rac.fits file will be saved, unless you specify the -s flag, which instructs calwf3.e to save all intermediate files.
the calwf3 pipeline will produce both CTE calibrated product and non-CTE calibrated products. The CTE products have a ‘c’ at the end of their extension name, such as _blc, _rac, _crc, _flc, and the non-CTE calibrated products contain the familiar : _blv, _crj, _flt.
Note: The CTE correction step uses all available CPUs (while the rest of the calibration steps do not).
3.4.2 wf3ccd
This routine performs the initial processing steps for all the WFC3 UVIS channel data. These steps are:
DQICORR - initializing the data quality array
ATODCORR - perform the a-to-d conversion correction
BLEVCORR - subtract the bias level from the overscan region
BIASCORR - subtract the bias image
FLSHCORR - subtract the post-flash image
wf3ccd first subtracts the bias and trims the overscan regions from the image. If an associated set of UVIS CR-SPLIT or REPEAT-OBS images is being processed, all of the overscan-trimmed images are sent through wf3rej to be combined and receive cosmic-ray rejection. The resulting combined image then receives final calibration with wf32d, which includes dark subtraction and flat fielding. If there are multiple sets of CR-SPLIT or REPEAT-OBS images in an association, each set goes through the cycle of wf3ccd, wf3rej and wf32d processing.
Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.
Running wf3ccd from a python terminal
In Python without TEAL:
from wfc3tools import wf3ccd
wf3ccd(filename)
In Python with TEAL:
from stsci.tools import teal
from wfc3tools import wf3ccd
teal.teal('wf3ccd')
Displaying output from wf3ccd in a Jupyter Notebook
When calling wf3ccd from a Jupyter notebook, informational text output from the underlying wf3ccd.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3ccd. As output is read from the underlying program, the wf3ccd Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf3ccd to the Python logging system by passing the logging.debug function or similar.)
If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.
Input Parameters for the Python interface
a single filename (iaa012wdq_raw.fits)
Command Line Options for the wf3ccd executable
input may be a single filename
Where the options include:
3.4.3 wf32d
This routine performs the remaining series of tasks in the UVIS pipeline
The wf32d primary functions include:
DARKCORR: dark current subtraction
FLATCORR: flat-fielding
PHOTCORR: photometric keyword calculations
FLUXCORR: photometric normalization of the UVIS1 and UVIS2 chips
Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.
Examples
In Python without TEAL:
from wfc3tools import wf32d
wf32d(filename)
In Python with TEAL:
from stsci.tools import teal
from wfc3tools import wf32d
teal.teal('wf32d')
Displaying output from wf32d in a Jupyter Notebook
When calling wf32d from a Jupyter notebook, informational text output from the underlying wf32d.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf32d. As output is read from the underlying program, the wf32d Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf32d to the Python logging system by passing the logging.debug function or similar.)
Parameters
a single filename (iaa012wdq_raw.fits)
Command Line Options for the wf32d executable
input may be a single filename
Where the options include:
3.4.4 wf3ir
This routine contains all the instrumental calibration steps for WFC3 IR channel images. The steps are:
DQICORR - initialize the data quality array
ZSIGCORR - estimate the amount of signal in the zeroth-read
BLEVCORR - subtract the bias level from the reference pixels
ZOFFCORR - subtract the zeroth read image
NLINCORR - correct for detector non-linear response
DARKCORR - subtract the dark current image
PHOTCORR - compute the photometric keyword values
UNITCORR - convert to units of count rate
CRCORR - fit accumulating signal and identify the cr hits
FLATCORR - divide by the flat-field images and apply gain conversion
The output images include the calibrated image (flt file) and the intermediate multi-accumulated ramp image (ima file).
Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.
Running wf3ir from a python terminal
In Python without TEAL:
In Python with TEAL:
Displaying output from wf3ir in a Jupyter Notebook
When calling wf3ir from a Jupyter notebook, informational text output from the underlying wf3ir.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3ir. As output is read from the underlying program, the wf3ir Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf3ir to the Python logging system by passing the logging.debug function or similar.)
Input Parameters for the Python interface
Command Line Options for the wf3ir executable
input may be a single filename
Where the options include:
3.4.5 wf3rej
wf3rej, the cosmic-ray rejection and image combination task in calwf3, combines CR-SPLIT or REPEAT-OBS exposures into a single image, first detecting and then replacing flagged pixels. The task uses the same statistical detection algorithm developed for ACS (acsrej), STIS (ocrrej), and WFPC2 data (crrej), providing a well-tested and robust procedure.
First, wf3rej temporarily removes the sky background from each input image (if requested via the SKYSUB parameter in the CRREJTAB), usually computed using the mode of each image. Sky subtraction is performed before any statistical checks are made for cosmic rays. Next, wf3rej constructs an initial comparison image from each sky-subtracted exposure. This comparison image can either be a median- or minimum-value sky-subtracted image constructed from all the input images, and it represents the ‘initial guess’ of a cosmic-ray free image. The comparison image serves as the basis for determining the statistical deviation of each pixel within the input images.
A detection threshold is then calculated for each pixel based on the comparison image:
where:
noise is the readnoise in DN squared and gain is the e-/DN of the amplifier used to read the pixel,
scale is the scale factor for the noise model,
value is the pixel value (in DN) from the median or minimum combined comparison image.
The actual detection criterion for a cosmic ray is determined as:
where:
pixeln is the pixel value from input image n,
skyn is the sky background of image n, and
median is the median or minimum pixel value from the comparison image.
If the pixel is flagged as a cosmic ray in the input image’s DQ array and is ignored when images are summed together. Surrounding pixels within some expansion radius (CRRADIUS) are marked as ‘SPILL’ pixels and are given less stringent detection thresholds.
When all input images have been processed, the values of the non-rejected pixels are summed over all input images. Each pixel in the summed output array is then scaled by the total exposure time:
where:
T is the total exposure time (regardless of whether all input images were used for that particular pixel). This corresponds to the value recorded in the header keywords TEXPTIME and EXPTIME.
The following keywords are also derived from the variables in this equation:
TEXPTIME = EXPTIME = T
REJ_RATE = averaged over all pixels
The remaining keywords EXPSTART, EXPEND are updated based on the values corresponding to the first and last input images, respectively.
In summary, the cosmic ray rejection task sums all non-rejected pixel values, computes the true exposure time for that pixel, and scales the sum to correspond to the total exposure time. The final scaled, cleaned pixel is written to the comparison image to be used for the next iteration. This process is then repeated with increasingly stringent detection thresholds, as specified by CRSIGMAS.
Cosmic Ray Rejection Table
wf3rej uses the Cosmic Ray Rejection parameter table (CRREJTAB) to determine the number of iterations for cosmic-ray rejection, the sigma levels to use for each iteration, and the spill radius to use during detection. This allows the rejection process to be tuned to each detector and observation, with suitable defaults being applied during pipeline processing. Observers may fine-tune the cosmic-ray rejection parameters when manually reprocessing data with wf3rej by editing the CRREJTAB.
The CRREJTAB reference file contains the basic parameters necessary for performing cosmic-ray rejection. The column names and default values for the CRREJTAB are given in Table 3.6. The appropriate row is selected based on the chip being processed (CCDCHIP), the number of images into which the exposure was split (CR-SPLIT), and the exposure time of each CR-SPLIT image (MEANEXP). If an exact match is not found for the exposure time, the table row with the closest value is used. If the CR-SPLIT value of the input images exceeds the values in the table, the table row with the largest CR-SPLIT value will be used. The sky fitting algorithm is controlled by the parameter SKYSUB, which can have values of ‘mode’, ‘mean’ or ‘none’. The ‘initial guess’ image is created using the median or minimum value of the input exposures, as specified by the value of INITGUES.
Cosmic-ray detection requires the specification of a threshold above which a pixel value is considered a cosmic ray. This threshold was defined above and uses the sigma rejection thresholds. These sigmas correspond to the CRSIGMAS column values in the CRREJTAB file. SCALENSE is a multiplicative term (in percent) for the noise model and is given as scale in the threshold equation above. This term can be useful when the pointing of the telescope has changed by a small fraction of a pixel between images. Under such circumstances, the undersampling of the image by the detector will cause stars to be mistakenly rejected as cosmic rays if a scale noise term is not included. This is a crude but effective step taken to satisfy the maxim of ‘do no harm’. However, for cases in which there have been no image-to-image offsets or the image is locally well-sampled, this will unduly bias against rejecting cosmic rays.
Pixels within a given radius, CRRADIUS, of a cosmic ray will also be treated as cosmic rays. A less stringent rejection threshold, CRTHRESH, can be used for detecting pixels adjacent to a cosmic ray. As for CRSIGMAS, CRTHRESH is also given as a sigma value. If CRTHRESH is exceeded, pixels within the defined radius of the cosmic ray will also be flagged. All pixels determined to be affected by a cosmic ray will have their DQ values set to 8192, as described in Table 3.6.
Table 3.6: Columns in cosmic-ray rejection parameter table.
Running wf3rej from a python terminal
In Python without TEAL:
from wfc3tools import wf3rej
wf3rej(filename)
In Python with TEAL:
from stsci.tools import teal
from wfc3tools import wf3rej
teal.teal('wf3rej')
Displaying output from wf3rej in a Jupyter Notebook
When calling wf3rej from a Jupyter notebook, informational text output from the underlying wf3rej program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3rej. As output is read from the underlying program, the wf3rej Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf3rej to the Python logging system by passing the logging.debug function or similar.)
Input Parameters for the Python interface
a single filename (iaa012wdq_raw.fits)
Command Line Options for the wf3rej executable
Input can be a single file
Where the options include:

WFC3 Data Handbook > Chapter 3: WFC3 Data Calibration > 3.4 Pipeline Tasks

Table of Contents Previous Next Index PDF