The Institute

HST

JWST

Community Missions

Data Archives

News and Outreach

Resources

Part II: ACS Data Handbook

4.4 Reprocessing with PyDrizzle

The goal of the ACS pipeline is to provide data calibrated to a level suitable for initial evaluation and analysis for all users. Observers frequently require a detailed understanding of the calibrations applied to their data and the ability to repeat, often with improved products, the calibration process at their home institution. There are several occasions when off-line interactive processing with PyDrizzle is required:

Observations which have been manually reprocessed with CALACS will still require a correction for geometric distortion.
Users may wish to modify the default PyDrizzle parameters, specifying an alternate output image orientation, pixel "shrinking factor" pixfrac, or pixel scale.
When images have been taken across multiple visits, the observations are not automatically associated. Creating custom association tables will allow data from multiple visits to be combined.
For optimal image registration, the user may wish to fine-tune the WCS information by supplying additional shifts and/or rotations.
During pipeline processing, PyDrizzle makes assumptions about which DQ flags should be considered bad. These pixels will be excluded when creating the drizzled product. Users may wish to select which flagged pixels should actually be considered good and included in the drizzled product.

The PyDrizzle software uses the Python language and can only be executed in the PyRAF environment. PyRAF, developed at STScI, allows users to run tasks using the standard IRAF parameter-based interface or by using separate Python programs. The PyRAF interface and the most recent version of PyDrizzle can be downloaded from:

http://stsdas.stsci.edu/pydrizzle.

The calibrated products from CALACS and the corresponding input to PyDrizzle are described for various observing modes in Table 3.1. For example, a single image will produce an FLT file, a RPT-OBS exposure will produce an SFL file, a CR-SPLIT exposure will produce a CRJ file, and a DITH-PATTERN will produce a series of FLT, CRJ, or SFL files at each dither position. PyDrizzle may be executed using any of the calibrated data products or the association table itself. When executed on an association table, PyDrizzle looks only for the existing product file and geometrically corrects that image. PyDrizzle will create a single drizzled image with the DRZ suffix for each calibration product. In the case of dithered observations which are part of a pattern, the individual FLT files will be combined to create a single DRZ image.

PyDrizzle was not developed with the capability of detecting cosmic-rays, so unless cosmic-ray rejection is performed in the first stage by CALACS, the calibrated, drizzled product will still contain cosmic-rays. A new package, MultiDrizzle, addresses this deficiency and is discussed in detail in Section 4.5.

4.4.1 Specifying the 'Bits' Parameter

The data quality flags which were set during CALACS processing can be used as bit masks when drizzling, and the user may specify which bit values should actually be considered 'good' and included in image combination. This is done via the parameter 'bits'. Any pixel flagged otherwise will be considered 'bad' and will be given zero weight. If a pixel is flagged as bad but has non-zero weight in any other image, the pixel will be replaced with the value from that image during image combination. Otherwise, the pixel will be replaced with the 'fill value'. The default 'fill value' used by PyDrizzle during pipeline processing is zero. If the 'fill value' is set to INDEF during reprocessing with PyDrizzle or MultiDrizzle, and there are no pixels with non-zero weight, the pixel will retain its original value.

The flags for the DQ array were presented in Chapter 3 and are summarized in Table 3.4. The bits value used during pipeline processing is 8578 which is the sum of 8192 (corrected cosmic rays) + 256 (saturated data) + 128 (bad pixel in bias file) + 2 (pixel rejected in second iteration of cr-rejection). Note that these pixels were flagged during CALACS processing as "bad" or "suspect", but may have been corrected in later processing steps. For example, cosmic rays are flagged 8192, but these pixels are replaced by unaffected pixels from other images in the CR-SPLIT association. The bits parameter indicates which "suspect" pixels to keep. When the total counts in a given pixel have exceeded the full well, for example, the DQ flag is 256. However, testing shows that counts still accumulate in a highly linear manner and that photometry of saturated stars is quite practical if using a gain that samples the full well depth.

The default bits value in the PyDrizzle and MultiDrizzle software is zero, but should be reset by the user prior to reprocessing. The value chosen for this parameter is completely up to the user and should be selected based on the specific calibration needs. For HRC processing, the user may want to add 8 (masked by aperture) to the 8578 value so that pixels which lie behind the occulting finger mask are not given zero weight. (In Figure 4.8 of Section 4.4.4 "PyDrizzle Examples", bit 8 was not included in the bits parameter prior to drizzling. Pixels behind the occulting finger were therefore given zero weight and replaced with the default fill value=zero.) Of course, the photometry in this region will be inaccurate due to improper flat fielding, so this bit would likely only be set for aesthetic reasons.

4.4.2 Creating Custom Association Tables

Association tables for dithered, REPEAT-OBS, or CR-SPLIT observations are generated by the CALACS pipeline. These tables keep track of the input exposure filenames and the output product filenames. Some types of observations, however, will not have association tables. Others will have multiple tables from different visits which need to be combined into one. In the following discussion, we present the methodology for creating custom association tables, either by merging association tables or creating them from scratch. Users also have the capability to manually edit association tables to include any known image offsets. This can be done using the TTOOLS task tedit in IRAF, where the user adds the columns XOFFSET, YOFFSET and/or ROTATION to the association table. Alternately, an ascii file with the known image offsets may be specified and the association automatically updated.

Merging Association Tables

Observing programs which cover a large portion of the sky will generally be made up of multiple pointings. Each of these pointings may be dithered or split for cosmic-ray rejection and will possess their own association table. In order for PyDrizzle to produce a single mosaic product, a composite association table must be built from the separate association tables. Users can easily create this composite by merging the individual tables using the TTOOLS task tmerge with "option = append".

The default product rootname will be taken from the first listed DTH-PROD in the composite association table. This rootname can be modified by the user (with tedit) to suit the desired naming convention. Generally, this rootname should match the rootname of the composite association table. A detailed example of merging association tables is given in step 1 of Example 4 from Section 3.5.2.

Creating Association Tables from Scratch

Some images will not have association tables. These may be single exposures of a given target which were taken at different times and orientations. They may even be exposures taken with different instruments. Observations taken using POS-TARG offsets, which are specified manually in the proposal, will not have association tables either.

A Python tool called buildAsn has been developed to create association tables which may then be used by PyDrizzle. This task can be run under PyRAF using an IRAF parameter-based "epar" interface or directly using a Python command-line interface. Since it resides alongside the PyDrizzle code, no extra setup is required to use this task.
A built-in help facility provides details on additional parameters which can be used with buildAsn when using the Python syntax. (For those needing a step-by-step introduction to starting PyRAF, see the examples in Section 4.4.4. Building an association table will be part of the PyDrizzle Examples.) It can be accessed using:

pyraf> from pydrizzle import buildasn pyraf> buildasn.help()
The following illustrates how to use buildAsn to create an association table from all calibrated input files with the FLT suffix found in a single directory. Within PyRAF and using the IRAF syntax, we type:

pyraf> buildasn mymosaic suffix='flt'
Alternately, the same result may be obtained using the Python syntax:

pyraf> buildasn.buildAsnTable('mymosaic',suffix='flt')
In this example, the association table will have the name 'mymosaic_asn.fits', the product name will be 'mymosaic_drz.fits', and all files with the FLT suffix found in the working directory will be included in the association. To specify a subset of images within a given directory, the user may specify the 'suffix' parameter to be a filelist ('@filename'), a wild-card list ('*8cd*flt*'), or any portion of the filename ('crj' or 'f555w').
If user determined offsets are available, buildAsn has the capability of incorporating them into the association table. These offsets (XOFFSET, YOFFSET, and/or ROTATION) are specified by the file listed in the 'shiftfile' parameter. Using the IRAF syntax:

pyraf> buildasn mymosaic suffix='flt' shiftfile='shift.txt'
This option allows users to fine-tune the final image combination by providing corrections to the header WCS information. For more information on the format of user-defined shifts, see Section 4.4.3. For an example illustrating the steps required to apply a shiftfile to an existing association, see Example 3 in Section 4.4.4.

4.4.3 Applying User-defined Shifts

The ability of PyDrizzle to properly align dithered observations relies on the accuracy of the header WCS information. Unfortunately, there are times when those WCS values may be slightly inaccurate, resulting in a mis-alignment of the final drizzled product. For example, one might wish to combine observations taken in different visits. If the visits were separated by a substantial time interval, then the telescope probably used different guide stars. Due to limitations in the intrinsic accuracy of the Guide Star Catalog, this can lead to WCS coordinates that differ by up to 1-2" between visits. Also when all exposures that one wishes to combine were taken in a single visit, it might still be important to correct for relative shifts between images. Even in two-star fine-guiding mode, thermal effects on the telescope can cause slow drifts that can build up to ~50 mas over a few orbits (or in rarer occasions within a single orbit). This corresponds approximately to 1 WFC pixel or 2 HRC pixels. Such shifts between exposures can degrade image quality and corrupt cosmic-ray rejection when left uncorrected.

By deriving shifts based on the position of objects in the data, the user may refine the shifts computed from the WCS header information to create a precisely-aligned drizzled product. Many observers have developed their own methods of comparing images and computing offsets. The conventions described here should support the majority of users, making it a simple matter to incorporate shift computations into MultiDrizzle or PyDrizzle.

Delta Shifts

A delta shift is defined as the residual shift required to align sources, after applying the offsets implied by the header WCS. Delta shifts may be determined by separately drizzling images onto a common WCS frame with the same central RA/Dec. This is performed in the 'driz_separate' step of MultiDrizzle (see section 4.5 for more information) or by setting the PyDrizzle parameter 'single=yes'. Object lists derived for each drizzled image may then be matched and fit to derive a single delta shift for each image.

Delta shifts are defined to be in the 'input' frame of reference when the distortion-corrected, drizzled image has the same orientation and scale as the distorted image. Shifts will be in the 'output' frame of reference if any rotation or scaling was applied while drizzling. When the shifts are given in the 'output' frame of reference, PyDrizzle requires the specification of the reference image to properly account for any orientation and scale changes.

Absolute Shifts

An absolute shift, on the other hand, is the total shift between two images and includes offsets implied by the header WCS. Absolute shifts may be determined by separately drizzling images to their own unique WCS frame. PyDrizzle will use the header information to optimally place the drizzled image in the frame.

To derive absolute shifts in the 'input' frame of reference, each distorted image is drizzled separately, and PyDrizzle uses the unique WCS information from each frame to choose the central RA/Dec, the image orientation and final image size. No additional rotation or scaling is applied while drizzling. (Alternately, the user may catalogue sources in each distorted image and then apply the distortion model to the X/Y positions. However, this requires tasks to transform coordinates from distorted to undistorted space. These tasks are still being developed for release in the STSDAS dither package.)

Absolute shifts in the 'output' frame of reference may be computed by separately drizzling each distorted image, with the same rotation and scaling, onto a unique WCS frame. Target lists in each drizzled image may then be matched to derive the absolute shifts.
Figure 4.6: Delta Shifts in the 'input' and 'output' frame of reference.

Figure 4.7: Absolute Shifts in the 'input' and 'output' frame of reference. The direction of north is indicated by the bold arrows.

Shift Units

PyDrizzle is capable of interpreting shifts in units of pixels or arcseconds. A pixel shift is the difference in the X/Y pixel position of a source in a given image compared to the reference image. The shifts are interpreted with respect to each image's X/Y pixel reference position and require the specification of 'input' or 'output' frame of reference.

An arcsecond shift is the difference in the RA/Dec coordinates of a source in a given image compared to the reference image. Object coordinates must be derived after correcting for distortion. These shifts are not angular offsets on the sky, rather they are changes in the RA/Dec of the reference pixel and do not require any 'cos(Dec)' adjustment. Arcsecond shifts are a natural result of using the RA/Dec coordinates of image sources. They eliminate any ambiguity with the chosen reference frame since RA/Dec coordinates are fixed on the sky. Arcsecond shifts are interpreted as offsets in the direction of increasing RA and Dec.

Sign Conventions

The shifts are defined in terms of how the targets in the image need to move, rather than on how the telescope would need to move, to allow precise registration. PyDrizzle assumes that the shifts were computed as 'image - reference'. For example, when using geomap to cross-correlate two source lists, the input image coordinates will be in the first two columns and the reference image coordinates will be in the last two columns.

Use of Shifts in Association Tables
The shift file is an ASCII file containing the user-computed shifts for a set of images and is used for updating the offsets in the association table. Observers may use either the ASN table delivered by the pipeline or create/modify their own table as described in Section 4.4.2. Both PyDrizzle and MultiDrizzle use the buildasn task to incorporate information from the shift file into the association table. The shift file uses the following format:

# units: pixels Values: pixels (default), arcseconds # frame: input Values: input (default), output # form: absolute Values: absolute (default), delta # reference: <filename> Only required when 'frame = output' filename1 xshift1 yshift1 [rotation1] filename2 xshift2 yshift2 [rotation2]
The first three lines specify the units, the frame of reference, and the form of the shifts, and only those lines with non-default values are necessary. While the '#' symbol is required, the 'Values:' comments are not. When shifts are given in the 'output' frame of reference, it is necessary to specify the name of the reference image used.
The following is an example of a shift file in units of pixels, measured in the 'input' frame of reference, with the form specified as 'delta' shifts. An x and y shift and a rotation (in degrees) is specified for each dataset. If only a simple shift is required, then the rotation column need not be specified.

#units: pixels #frame: input #form: delta j8c0b1skq_flt -2.4 -1.2 -0.002 j8c0b1snq_flt -4.3 -2.3 0.001 j8c0b1sqq_flt -6.9 -3.5 0.003
Using this shiftfile as input, the buildasn task will update the ASN table as indicated in Table 4.3.
Table 4.3: Association table updated with user-defined shifts.

MEMNAME MEMTYPE XOFF- SET YOFF- SET XDELTA YDELTA ROTATION

j8c0b1skq

EXP-DTH

0

0

-2.4

-1.2

-0.002

j8c0b1smq

EXP-DTH

0

0

-4.3

-2.3

0.001

j8c0b1sqq

EXP-DTH

0

0

-6.9

-3.5

0.003

The shift file can contain shifts for files other than those contained in the ASN table. In that case, only those entries found in the ASN table will be used to update the table. Thus, a single shift file can be generated for a whole set of observations which are represented by multiple association tables.

Association (ASN) tables can support either delta shifts or absolute shifts in units of pixels or arcseconds. Absolute shifts are stored in columns 'XOFFSET' and 'YOFFSET', while delta shifts are stored in columns 'XDELTA' and 'YDELTA'. The 'ROTATION' column is also available if a simple shift does not produce ideal registration and indicates how much should be added to the image's ORIENTAT to get proper alignment.

If absolute offsets are provided in the shift file, PyDrizzle will populate the OFFSET columns in the association and fill the DELTA values with zeroes. Similarly, if delta shifts are provided, those values will be used to populate the DELTA columns and the OFFSET columns will be zero. If shifts are given in both the OFFSET and DELTA columns, the OFFSET column will be applied and the DELTA columns will be ignored. This convention used by buildasn eliminates any ambiguity as to which values are applied to the data. Editing the ASN table directly will then allow the user to further update the offsets using either delta or absolute shifts.

4.4.4 PyDrizzle Examples

These examples further describe the steps required for pipeline reprocessing and serve as a continuation of the CALACS examples described in Section 3.5.2. We advise the reader to work through the CALACS examples first to develop a solid understanding of the calibrated data products and their potential limitations.

Example 1: Drizzling a Single Exposure

The following is a continuation of Example 1 from Section 3.5.2 where manual recalibration of a single raw exposure with CALACS produced a single image with the FLT suffix.
In the Unix environment, the 'jref' environment must be established prior to loading PyRAF. This directory indicates the location of the geometric distortion reference file (IDCTAB) specified in the image header. For more information, see "Setting up jref" in Section 3.5.1. If running IRAF and PyRAF simultaneously, 'jref' will need to be defined in each xterm window before loading the software. Alternately, this path may be initialized in the user's .setenv file. PyRAF should then be started in its own xterm window and the stsdas, analysis, and dither packages loaded.

unix> setenv jref /mydisk/myjref/ unix> pyraf pyraf> stsdas pyraf> analysis pyraf> dither
Next, we run PyDrizzle on the calibrated FLT image (j8bt07oyq_flt.fits). By default, the drizzled output product will be named j8bt07oyq_drz.fits. To specify an output file which is different than the default, the user may set the 'outfile' parameter to 'myfile_drz.fits', for example. Section 4.4.1 defined the 'bits' parameter; here we adopt the recommended value of 8578. The reader may also wish to add 8 to this value to see the effect of not dropping pixels behind the occulting finger. Prior to running, we recommended clearing any preset values using the 'unlearn' command.

pyraf> unlearn pydrizzle pyraf> pydrizzle j8bt07oyq_flt.fits bits=8578
The distortion corrected HRC science image is shown in Figure 4.8. The image on the left is the flat fielded FLT product from CALACS and still contains distortion. The image on the right is the calibrated, distortion-corrected DRZ product from PyDrizzle using 'bits=8578'.
Figure 4.8: The effect of applying the distortion correction to a single HRC image. The image on the left is the flat-fielded FLT product from CALACS. The image on the right is the distortion-corrected DRZ product from PyDrizzle.

Example 2: Drizzling Images taken as part of a Dither Pattern

The following is a continuation of Example 5 from Section 3.5.2 where manual recalibration of a 2-point line dither pattern produced two FLT images (j8e654c0q_flt.fits, j8e654c4q_flt.fits).
Using tprint, we can view the contents of the image association, including the rootnames of the individual dithered images and the default rootname of the drizzled product.

pyraf> tprint j8e654010_asn.fits # MEMNAME MEMTYPE MEMPRSNT j8e654c0q EXP-DTH yes j8e654c4q EXP-DTH yes j8e654011 PROD-DTH yes
Next we run PyDrizzle on the image association. The two FLT images are combined to create a single DRZ image called 'j8e654011_drz.fits'.

pyraf> pydrizzle j8e654010_asn.fits bits=8578
Note that PyDrizzle is not designed to remove cosmic rays from these images. This step requires the new software MultiDrizzle, which is described in Section 4.5.

Example 3a: Improving the Image Registration - 'Delta Shifts' in the 'Input Frame'

For observations with large dithers or for images which were taken at nominally the same pointing but during different visits, users may wish to fine-tune the image registration, improving on the world coordinate system in the image header. In this case, the images must be separately drizzled and the position of stars cross-correlated. The derived shifts may then be used to update the image association prior to drizzling the final product. More generally, if the drizzle combined product of multiple exposures have PSFs that are not round or as sharp as expected, a likely culprit is the need for improved shifts.

The following example uses WFC observations from separate visits in program 9018: visit 61, exposure 02 (j8c061020_asn.fits) on 19April 2002 and visit C1, exposure 01 (j8c0c1010_asn.fits) on 09May2002. Each observation is CR-SPLIT with an exposure time of 45 seconds in F606W. The images are at the same pointing, use the same guide stars, and are at the same ORIENT. We describe the steps required to register the calibrated CRJ products from separate visits. However, the individual FLT files could be alternatively used if improved registration of each CRJ image is desired, although use of FLT images would result in cosmic-ray retention.
First, we use the CRJ images from visits 61 and c1 to create a new association table with the suffix 'v61c1'. We advise using the tprint task to verify the contents of the association.

pyraf> import buildasn pyraf> buildasn f606w_v61c1 suffix='crj.fits' pyraf> tprint f606w_v61c1_asn.fits # MEMNAME MEMTYPE MEMPRSNT XOFF YOFF XDELTA YDELTA ROTATION j8c061021 EXP-DTH yes 0. 0. 0. 0. 0. j8c0c1011 EXP-DTH yes 0. 0. 0. 0. 0. f606w_v61c1 PROD-DTH no 0. 0. 0. 0. 0.
Next, we separately drizzle the images onto a common output WCS. This is achieved by setting the PyDrizzle parameter 'single'=yes. Any differences in the aperture RA and Dec, any POS-TARG offsets, or any orientation differences will be automatically accounted for by PyDrizzle which uses the header WCS to transform the images. The drizzled CRJ products are appended with the suffix '_single_sci.fits'.

pyraf> pydrizzle f606w_v61c1_asn.fits bits=8578 single=yes
Using point sources in each image, we create and cross correlate coordinate lists from each visit. For example, we used daofind to create star lists for each DRZ product separately. Then we used xyxymatch to match the coordinate lists within a user specified tolerance. Finally, we use geomap with the matched coordinate list to compute the transformation required to map the reference coordinate system to the input coordinate system, allowing for a shift and/or rotation. We create the following "shiftfile", a simple ascii table called 'shift.txt', using the derived offsets. The shifts were derived in units of pixels and are in the 'input' frame of reference since no additional rotation or scale terms were introduced while drizzling. The derived shifts using the above methods are delta shifts because the images were drizzled onto a common WCS.

#units: pixels #frame: input #form: delta j8c061021_crj 0.00 0.00 0.000 j8c0c1011_crj 24.46 -56.54 0.001
Next, we delete and rebuild the previous association table, this time specifying the derived offsets via the shiftfile. To verify that the association has been correctly populated, we use the tprint command. Note that the delta shifts appear in the DELTA columns in the association table.

pyraf> delete f606w_v61c1_asn.fits pyraf> buildasn f606w_v61c1 suffix='crj.fits' shiftfile='shift.txt' pyraf> tprint f606w_v61c1_asn.fits # MEMNAME MEMTYPE MEMPRSNT XOFF YOFF XDELTA YDELTA ROTATION j8c061021 EXP-DTH yes 0. 0. 0. 0. 0. j8c0c1011 EXP-DTH yes 0. 0. 24.46 -56.54 0.001 f606w_v61c1 PROD-DTH no 0. 0. 0. 0. 0.
Finally we run PyDrizzle on the new image association. The two CRJ images are combined to create a single DRZ image called 'f606w_v61c1_drz.fits'.

pyraf> pydrizzle f606w_v61c1_asn.fits bits=8578 single=no
Example 3b: Improving the Image Registration - 'Absolute Shifts' in the 'Output Frame'

As a continuation of the previous example, we illustrate how to specify absolute shifts in the 'output' frame of reference.
First, the CRJ images from each visit are drizzled to separate output images. Absolute shifts are determined by separately drizzling images onto their own unique WCS frame. In this example, an additional rotation is applied to the drizzled images so that north is oriented up.

pyraf> pydrizzle j8c061021_crj.fits output=j8c06102X_drz.fits bits=8578 rotate+ orient=0 pyraf> pydrizzle j8c0c1011_crj.fits output=j8c0c101X_drz.fits bits=8578 rotate+ orient=0
As in the previous example, we create and cross correlate coordinates lists from each visit using point sources in each image. We create the following "shiftfile", a simple ascii table called 'shift1.txt', using the derived offsets. Note that because we are using absolute shifts, we must specify the name of the reference image.

#units: pixels #frame: output #form: absolute #reference: j8c06102X_drz.fits j8c061021_crj 0.00 0.00 0.000 j8c0c1011_crj -41.64 45.51 0.001
Next, we build a new image association using the two CRJ files and the derived shift file.

pyraf> import buildasn pyraf> buildasn f606w_v61c1rot suffix='crj.fits' shiftfile='shift1.txt'
To verify that the association has been correctly populated, we use the tprint command. Notice that the absolute shifts have been correctly added to the OFFSET columns, not the DELTA columns.

pyraf> tprint f606w_v61c1rot_asn.fits # MEMNAME MEMTYPE MEMPRSNT XOFF YOFF XDELTA YDELTA ROTATION j8c061021 EXP-DTH yes 0. 0. 0. 0. 0. j8c0c1011 EXP-DTH yes -41.64 45.51 0. 0. 0.001 f606w_v61c1rot PROD-DTH no 0. 0. 0. 0. 0.
Finally we run PyDrizzle on the new image association, specifying a rotation such that north is up. The two CRJ images are combined using the updated shift information to create a single DRZ image called 'f606w_v61c1rot_drz.fits'.

pyraf> pydrizzle f606w_v61c1rot_asn.fits bits=8578 rotate+ orient=0