Introduction
Updates made to the ACS calibration pipeline, CALACS, are documented here. Several times a year, updates to the software packages in the HST calibration pipelines (HSTCAL) are released in an HST data processing (HSTDP) build (e.g. HSTDP 2018.1). After each build is released, the HSTCAL package is available to the science community on conda-forge.
Latest Updates
Current Version Information
ACSTOOLS version: 3.5.0
HSTCAL version: 2.7.4
CALACS version: 10.3.5
Build Information
-
CALACS updated to Version 10.3.5.
-
Update to the cosmic ray rejection algorithm of CALACS as to the way the output ERR extension is computed for the CRJ/CRC file. The output ERR is now propagated from the usable input ERR extensions versus being computed by a model.
-
Modified CALACS to ensure the ERR extension of the CRC/CRJ file is always a two-dimensional image, even if the entire image could be represented by a single value. This change was triggered by ACS Instrument Team testing and to avoid possible confusion by users.
-
HSTCAL v2.7.3 standalone is distributed by conda-forge.
-
CALACS updated to Version 10.3.4.
-
Modifications to resolve an issue seen only when the executables were compiled under the new conda-forge system. These changes included some modest clean-up.
-
CALACS updated to Version 10.3.3.
-
Fixed a bug associated with the FLASHDUR computation. The FLASHDUR keyword has been updated in the CRJ/CRC headers to be the sum of the individual input images as this keyword is used in ACS2D to scale the flash reference file for flash correction.
-
Fixed a bug associated with the DARKTIME computation. Code written for CALACS 10.2.2 has been moved from ACS2D to ACSCCD so every BLV_TMP file has the correct DARKTIME keyword value. Also, the DARKTIME keyword has been updated in the CRJ/CRC headers to be the sum of the individual input images. Only the DARKTIME value in the input file header is used to scale the dark reference file for the dark correction.
-
CALACS updated to Version 10.3.2.
-
Reverted the name of the ACS forward model executable back to "acscteforwardmodel.e" from "acsforwardmodel.e" to mitigate documentation and script issues. The name was inadvertently changed when code was reorganized to resolve circular references. This update does NOT affect the calibration pipeline processing of data nor any data in the archive.
-
CALACS updated to Version 10.3.1.
- Fixed a bug which specifically was only an issue when trying to run "acscte.e" or "acsforwardmodel.e" as
standalone components. These two executables would generate a SegFault. This problem did NOT affect
the calibration pipeline processing (i.e., CALACS).
-
CALACS updated to Version 10.3.0.
- Implementation to use a full two-dimensional image to detect and flag full-well saturation values in the science data versus a simple single scalar as used previously. The detection/flagging occurs after bias correction, conversion to electrons, and blev correction.
-
CALACS updated to Version 10.2.4.
-
Removed residue sort routine.
-
Addresses issues associated with routines under copyright.
-
CALACS updated to Version 10.2.3
-
The A-to-D saturation threshold is no longer a hard-coded constant value. A threshold value appropriate for the input raw data (matching in chip number, amp, gain, and offset) is now read from the CCDTAB reference file.
-
CALACS updated to Version 10.2.2.
-
In the DARKCORR step of CALACS, the value of the DARKTIME keyword is used to account for time that dark current accumulates. It has been determined that depending upon the type of WFC observation (i.e., full frame or subarray, post-flash or no post-flash) there is an additional correction needed. The additional correction values need to be read from updated CCDTAB reference files, and correspondingly, the DARKCORR algorithm has been updated to accommodate new logic for applying the correction.
-
CALACS updated to Version 10.2.1.
-
CALACS has been modified to allow the FITS PHOTMODE keyword to be updated with the MJD for the SBC, in addition to the HRC and WFC. This will allow time-dependent sensitivity to be applied properly to SBC.
-
CALACS updated to Version 10.2.0.
-
For bias or dark images, ACSREJ ignores the bad pixels (BPIXTAB flag of 4) during the combination process for the SCI and ERR arrays (i.e. treat bad pixels as normal pixels).
-
CALACS updates to Version 10.1.0 due to updates in the ACSCCD and ACS2D algorithms.
-
The dark correction, DARKCORR step of the ACS2D algorithm, has been modified to work directly on the full 2D data rather than on a line-by-line basis.
-
The bias shift correction, BLEVCORR step of the ACSCCD algorithm, has been upgraded to accommodate the new supported subarrays which have both physical and virtual overscan. This is the same algorithm which applies to full frame data.
-
Forward model functionality was implemented which exploits the existing CTE code and usage (Generation 2 only). Instead of correcting the CTE trails, the forward model functionality simulates (adds) the trails. This functionality should be accessed via the new ACSTOOLS function, acscteforwardmodel.py
-
Updated ACS library routine, getacskeys, updated to call the new HSTIO function which robustly determines the number of IMSETS in a FITS file.
-
CALACS updates to Version 10.0.0 due to the modifications imposed by the upgraded ACSREJ algorithm.
-
An updated ACSREJ algorithm was implemented where all calculations are now done in electrons versus DN. The threshold computation uses the ERR arrays to account for post-flash noise. The C interface to the ACSREJ standalone component of CALACS was modified for clarity. Specifically, the "newbias" option was renamed "readnoise_only". The functionality associated with this option remains the same. For more information see ACS ISR 2018-05.
-
CALACS now uses the amp-dependent read noise as the threshold to determine if a pixel has enough signal-to-noise ratio such that a reasonable CTE correction can be applied.
-
A bug fix has been implemented which is only applicable for RAWs with PCTECORR=PERFORM for ASNs used for cosmic ray rejection. In this case the CR rejected output (CRJ) previously used the wrong dark; it used the CTE-corrected dark. This has been addressed, and the CRJ product is now correct utilizing the non-CTE corrected dark.
-
All BIAS files are now processed by CALACS to use only read noise (vs Poisson) for the computation of the ERR array as the date discriminant (pre- and post-SM4) has been removed.
-
CALACS updates to version 9.2.0 with this release.
-
New BLEVCORR algorithm that uses new OSCNTAB, which now correctly calculates bias level for AMP B and D in subarrays.
-
New SINKCORR step to apply new SNKCFILE in order to flag sink pixels. For information about the SINKCORR step see ACS ISR 2017-01.
-
New pixel based PCTECORR algorithm implemented (
CTE_NAME= 'PixelCTE 2017').- The code defaults to this new algorithm, however, the old one can still be executed using the command line option
--ctegen <1|2>. Note: The correct accompanying PCTETAB file must also be used. - The user can now override the PCTETAB file used (normally specified in the image header) with the command line option
--pctetab <file path>. - The number of threads used for computing the PCTECORR can now be specified with the command line option
--nthreads <N>. Note: OPENMP is required for N > 1.
- The code defaults to this new algorithm, however, the old one can still be executed using the command line option
-
Fixed uninitialized variable in
prinfo.c. -
Corrected
--versionreporting and added-rto report version string that was previously reported by--version.
Overview of CALACS
The ACS calibration pipeline, CALACS, is divided up into five main tasks that are run sequentially. In order, these five tasks are: acsccd, acscte, acsrej, acs2d, acssum. The exact path an image takes through the pipeline is controlled by various switches stored in the image's primary header. In the boxes below, we describe each of the processing steps applied in the five main tasks and the relevant header switches that control them. For a complete description of each of these tasks, we refer users to the ACS Data Handbook.
This task contains the initial processing functions for all ACS CCD data.
| Function Name | Header Switch | Action Applied |
|---|---|---|
| doDQI |
DQICORR |
Initializes the data quality (DQ) array by combining it with a table of known permenant bad pixels marked in the bad pixel table reference file. |
|
doBias |
BIASCORR |
Subtracts the superbias reference file to remove the low order, quasi-static structure of the bias, including the bias gradient in post-SM4 images |
| toElectrons | NONE, applied to all CCD data | Mulitplies the image data by the gain to convert from DN to electrons. |
| doBlev |
BLEVCORR |
For observations taken before Servicing Mission 4 (SM4), computes and subtracts the bias level from the image data. For observations taken after SM4, corrects for the signal-dependent bias shift, cross-talk, and bias-striping. |
| doNoise | NONE, applied to all CCD data | Initializes the error array. |
| doSink |
SINKCORR |
Flags sink pixels and the adjacent affected pixels with the value 1024 in the DQ array of ACS/WFC images. |
This task applies to pixel-based CTE correction to the ACS/WFC data.
| Function Name | Header Switch | Action Applied |
|---|---|---|
| N/A |
PCTECORR |
Applies the pixel-based CTE correction software to generate CTE-corrected products (e.g. FLC, CRC, or DRC) |
For CR-SPLIT observations, this task will combine the individual observations and perform cosmic ray rejection.
| Function Name | Header Switch | Action Applied |
|---|---|---|
| N/A |
CRCORR |
Runs the cosmic ray rejection routine to generate a single CRJ or CRC from a list of FLTs or FLCs. |
This task is run on all ACS observations. A number of the header switches are detector-dependent.
| Function Name | Header Switch | Action Applied |
|---|---|---|
| doDQI |
DQICORR |
If the DQ array has not already been initialized by acsccd, initializes the DQ array using information from the bad pixel table reference file. |
| doNonLin |
LFLGCORR, GLINCORR |
Corrects and flags non-linear data in observations taken with the SBC MAMA |
| toElectrons | None | If the units of CCD data are not already electrons, multiplies by the gain to convert them. |
| doNoise | None | Checks the error extension of the image to determine if there are non-zero values generated in previous steps. If the values are all zero, it computes the error using the applicable noise model. |
| doDark |
DARKCORR |
Performs the dark subtraction. |
| doFlash |
FLSHCORR |
Performs the flash subtraction. |
| doFlat |
FLATCORR |
Performs the flat-field division. |
| doShad |
SHADCORR |
Performs the CCD shutter shading correction. Currently skipped as calibration data show that a shading correction is not needed for ACS. |
| doPhot |
PHOTCORR |
Computes the photometric keywords values and populates them in the header. |
| doStat | None | Computes basic image statistics. |
This task will combine repeated sub-exposures taken with the SBC MAMA
| Function Name | Header Switch | Action Applied |
|---|---|---|
| N/A | RPTCORR | Combines repeated sub-exposures using a straight pixel-to-pixel addition of the science values. Propagates uncertainties by summing in quadrature. |
ACSTOOLS
To facilitate scripting of calibration workflows, the ACS Team developed the python package, ACSTOOLS. This package is a wrapper around the compiled C executables used by the CALACS pipeline. It provides users with an easy-to-use python interface, as well as, access to other utility functions developed by the ACS Team (e.g. the CTE forward modeler). The interface simplifies the process of building up calibration workflows, making it much easier for users to reprocess their data if they so choose. For more information on ACSTOOLS, please see the website.
Example Usage
Suppose you wanted to reprocess CR-SPLIT observations while experimenting with the parameters in CRREJTAB. Accomplishing this using the executables is a drawn out process that involves iteratively modifying the parameters stored in the CRREJTAB reference file. A simpler way to do this is by leveraging the python interface provided with ACSTOOLS. In the section above on CALACS, we see that only two tasks are executed before acsrej. We write a simple python script to process our observations through acsccd and acscte to generate intermediate data products with this minimum level of required calibration completed.
from acstools import acsccd, acscte
# Run the acsccd task (*raw.fits -> *blv_tmp.fits)
acsccd.acsccd('*raw.fits', time_stamps=True, verbose=True)
# Run CTE correction (*blv_tmp.fits -> *blc_tmp.fits)
acscte.acscte('*blv_tmp.fits')
Now that we have reached the level of calibration expected before entering the acsrej task, we can experiment with the parameters of CRREJTAB much more easily. The values stored in the CRREJTAB reference file (see Section 2 in ACS ISR 2018-05) can be accessed through keyword arguments in the ACSTOOLS implementation. Instead of updating the reference file, change the values passed in the function call:
from acstools import acsrej
acsrej.acsrej('*blc_tmp.fits',
output='combined_product.fits', # output filename
crsigmas='6,5,4', # specify the rejection threshold
crradius=0.75, # specify the propagation radius
initgues='med') # specify the method for computing the initial guess