WFC3 Data Handbook v.4
help@stsci.edu

Table of Contents Previous Next Index PDF


WFC3 Data Handbook > Chapter 3: WFC3 Data Calibration > 3.1 The calwf3 Data Processing Pipeline

3.1
The main executable which processes data taken with either the UVIS or IR detectors of the WFC3 instrument onboard the Hubble Space Telescope is called calwf3. The code is organized into subroutines that are called by the calwf3 executable. The subroutines may be called independently if users desire specialized processing for their dataset. The subroutines used for processing UVIS images are called w3cte, wf3ccd, and wf32d. The main subroutine used for processing IR images is wf3ir. The wf3rej subroutine is shared between the UVIS and IR pipelines and is used for combining CR-SPLIT or REPEAT-OBS image sets. Figure 3.1 shows the flow diagram for the UVIS pipeline as a whole, while Figure 3.3 contains the flow for the IR pipeline.
A detailed description of the improvements in calwf3 v3.3, which is more generally referred to as the UVIS2.0 update, can be found in the February 22, 2016 issue of the STAN and references therein. Each WFC3 image is calibrated with reference files particular to the observing mode used. Currently, raw images and telemetry files are processed and calibrated, using the most up-to-date reference files, parameters, and software (see Section 1.1.1 of the Introduction to the HST Data Handbooks). Then the data products, as well as the raw data are stored by MAST ( Mikulski Archive for Space Telescopes).
During automatic pipeline processing by the STScI archive, Astrodrizzle follows calwf3. All calibrated images are corrected for geometric distortion and associated sets of dithered images are combined into a single product.
3.1.1 Where to Find calwf3
calwf3 is part of the HSTCAL package, which can be downloaded separately from its GIT repository in the Space Telescope Area. Its binaries are also installed along with the STScI distributed package Astroconda.
3.1.2 Running calwf3
calwf3 can be run on a single input raw file or an asn table listing the members of an association. When processing an association, it retrieves calibration switch and reference file keyword settings from the first image listed in the asn table. calwf3 does not accept a user-defined list of input images on the command line (e.g. *raw.fits to process all raw files in the current directory).
The wf3ccd, wf32d, wf3cte, wf3ir tasks on the other hand, will accept such user-defined input file lists, but they will not accept an association table ( asn ) as input.
Below are some basic examples of how to run calwf3 in python and directly from a terminal. Pure python commands may still be used in a PyRAF session. More advanced usage examples are available in Sections 3.5
Running calwf3 from a python terminal using wfc3tools
In Python without TEAL:
from wfc3tools import calwf3
calwf3(filename)
In Python with TEAL:
from stsci.tools import teal
from wfc3tools import calwf3
teal.teal('calwf3')
Running many files at the same time with python
The recommended method for running calwf3 in batch mode is to use Python and the wfc3tools package in the Astroconda distribution.
For example:
from wfc3tools import calwf3
from glob import glob
for fits in glob('i*_raw.fits'):
calwf3(fits)
Displaying output from calwf3 in a Jupyter Notebook
When calling calwf3 from a Jupyter notebook using the calwf3() function from wfc3tools, informational text output from the underlying calwf3.e program will be passed through the print function as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to calwf3. As output is read from the underlying program, the calwf3 Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect calwf3 to the Python logging system by passing the logging.debug function or similar). If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.
Command line options for the calwf3 executable
calwf3 can also be called directly from the OS command line by supplying the executable calwf3.e with an input file and a list of options. This is the same executable that the wfc3tools package calls.
a single filename (``iaa012wdq_raw.fits``)
-v : Print verbose time stamps and information
Running many files at the same time from the command line
The command line executable only accepts one file at a time, but you can use operating system tools like awk or xargs to process everything in a directory:
ls *raw.fits | awk '{print "calwf3.e",$1}' | csh
or alternatively:
3.1.3 Keyword Usage
calwf3 processing is controlled by the values of keywords in the input image headers. Certain keywords, referred to as calibration switches, are used to control which calibration steps are performed. Reference file keywords indicate which reference files to use in the various calibration steps. Users who wish to perform custom reprocessing of their data may change the values of these keywords in the _raw FITS file primary headers and then rerun the modified file through calwf3.
Other keyword values record instrument and detector parameters that are used in the calibration and some record information that is computed or derived during calibration. Table 3.1 provides a summary of these keywords used by calwf3, specifying whether they function as input or output to the task(s) listed in column 2. For a definition of each keyword see Tables 2.7, 2.8, and 2.9.
3.1.4 Using CRDS To Update Reference Files
The HST Calibration Reference Data System (CRDS) is the reference file management software used by STSCI for organizing and assigning reference files to datasets. Users can query CRDS to get the best references files for their data at hand.
The WFC3 team recommends that the most up-to-date reference files be used for re-processing.
This is particularly important for data taken at multiple epochs. For example, if the data are retrieved from MAST at different times (e.g., after the execution of each visit), it may be possible to observe systematic differences, due to changes in the reference files, or more generally, to the whole data processing flow. Thus it is recommended that the whole dataset pertinent to the given scientific investigation is retrieved simultaneously. Alternatively, the data need to be reprocessed off-line (starting from the unprocessed raw files) using a self-consistent configuration of pipeline and reference files.Occasionally there may be an instrument mode for which there is no calibration data. In these cases there will be a placeholder reference file in CRDS that is filled with zeros or ones, as appropriate. Such reference files are identified by having their header keyword PEDIGREE set to DUMMY. When calwf3 encounters one of these reference files it will automatically skip the calibration step for which the file is used (e.g., DARKCORR will be set to the value SKIPPED if the DARKFILE is dummy).

WFC3 Data Handbook > Chapter 3: WFC3 Data Calibration > 3.1 The calwf3 Data Processing Pipeline

Table of Contents Previous Next Index PDF