3.6	Customizing COS Data Calibration

3.6.1 Mechanics of Tailored Recalibration

3.6.2 Using GO Wavecals

Sometimes the pipeline calibration performed shortly after the data were obtained from the telescope is not the best possible calibration for your science program. There are a number of reasons why it may be desirable to recalibrate your data. The most likely reasons include:

•	More appropriate reference files have become available since the data were originally obtained.

•	Some steps need to be repeated with different input parameters. For example, you may wish to re-perform the 1-D spectral extraction with a smaller height in order to minimize the background, or you may wish to cut a TIME-TAG exposure into sub-exposures, in order to study time variability.

The simplest way to recalibrate your data with the most appropriate reference files is to request the data from the archive again. However, to tailor the calibration to your individual preferences, it may be beneficial to run calcos yourself on your local machine, or to use tasks that improve the reference files or allow customized treatment of the data. Calcos can be imported and executed when running PyRAF or Python.

 

Be sure you are using the latest versions of the calcos and STSDAS software, COS calibration files, and raw data files (which list the latest reference files in their headers). STSDAS release information can be found at www.stsci.edu/resources/software_hardware/stsdas.

Calcos contains provisions for recalibrating raw data. Users can specify the pipeline processing steps to be performed and select the associated reference files. However, calcos was not designed to run its various modules independently, i.e. it is not re-entrant. The pipeline flow is modified by setting calibration switches or reference file names and then rerunning the entire pipeline. The calibration switches in the headers of the calibrated data files will reflect the operations performed on the calibrated data and the reference files used.

3.6.1 Mechanics of Tailored Recalibration

Set up the Directory Structure for Running calcos

Retrieve Reference Files

Edit the Calibration Header Keywords

Edit the Input Association File

Run Calcos

If you chose to recalibrate your COS data on your local machine, there is a certain amount of set up required for calcos to run properly. The operations mentioned in the checklist below will be described in detail in the following subsections:

•	Set up a directory with the required reference files.

•	Determine which reference files are needed and retrieve them from the Archive.

•	Set the environment variable lref to point to your reference file directory. Note: you must do this before starting a PyRAF session!

•	Update the input data file headers (including reference file names). In IRAF, this would be done using thedit.

•	Update the input association files if changing files to be included.

•	Set the calibration switches in the headers of the raw data files to perform the needed steps.The default calibration switches are listed in Table 2.16.

•	Run calcos.

Set up the Directory Structure for Running calcos

Before running calcos, you will need to define an environment variable to indicate the location of the directory containing the needed calibration reference files. The names of the calibration files are preceded with the logical path name “lref$” in the COS science headers. Ordinarily you would define this directory in a PyRAF session to be, for example, “/data/vega3/cos/lref/” using the set command:

cl> set lref "/data/vega3/cos/lref/" # Won’t work!

Note the trailing slash (/). However, calcos and all of its modules are actually foreign tasks and as such do not access PyRAF environment variables. Therefore, before invoking the cl, you will need to define an environment variable from the host command line (see below) that is appropriate to your host machine. For Unix/Linux/Mac systems, the appropriate command for the example above is:

% setenv lref /data/vega3/cos/cal_ref/

Note that an alternative to using the lref$ variable is specifying the full pathnames to the reference files in the science headers.

When running calcos or any of its modules, you must define environment variables (such as lref$) before starting the cl. It is not possible to define them within IRAF using the set command, nor is it possible to define them with an escape to the host level, such as: !setenv lref /data/vega3/cos/lref/

Retrieve Reference Files

To recalibrate your data, you will need to retrieve the reference files used by the different calibration steps to be performed. The names of the reference files to be used during calibration must be specified in the primary header of the input files, under the section “CALIBRATION REFERENCE FILES.” Note that the data headers will be populated already with the names of the reference files used during pipeline calibration at STScI.

Chapter 1 of the Introduction to HST Data Handbooks describes how to retrieve data and reference files via the World Wide Web. To retrieve the best reference files via MAST, check “Best Reference Files” in the “Reference Files” section of the Retrieval Options form.

The COS reference files are all in FITS format, and can be in either IMAGE or BINTABLE extensions. The names of these files along with descriptions of their contents are given in Section 3.7. The rootname of a reference file is based on the time that the file was delivered to the Calibration Data Base System (CDBS).

Edit the Calibration Header Keywords

To edit file headers in preparation for recalibration, use the STSDAS task thedit. The thedit task takes several input parameters: the name(s) of the raw data files to be edited, the header field to be edited, and the new value of the header field. It can be used to change the values of any calibration switches, reference files or tables to the values you wish to use for recalibrating your data. To edit the calibration keyword values:

1.

Run the thedit task, specifying a list of files in which you want to change calibration keyword values. You may specify more than one file (using wildcards, for example) to be updated. For example, you could change the flat reference file to be used for all COS raw science files in the current directory using the command:

ct> thedit *raw*.fits[0] flatfile ’lref$n9n201821_flat.fits’ up+

Similarly, to turn off the FUV burst calibration switch use the command:

ct> thedit *raw*.fits[0] brstcorr ’OMIT’ up+

If you are changing keywords that reside in the FITS primary header unit with hedit or thedit, be sure to explicitly specify the primary header by appending “[0]” to the FITS file name.

Edit the Input Association File

Users may find it necessary to edit the input association file for calcos. Reasons for editing an association file might include the use of a different wavecal or to remove a compromised exposure from an association. One way to update an association file is to use the STSDAS task, tedit. For example, use the PyRAF task tprint to first look at the contents of association table, l9v221010_asn.fits.

-->tprint l9v221010_asn.fits prparam=no prdata=yes # MEMNAME MEMTYPE MEMPRSNT L9V221EUQ EXP-FP yes L9V221EWQ EXP-AWAVE yes L9V221EYQ EXP-FP yes L9V221F0Q EXP-AWAVE yes L9V221F2Q EXP-FP yes L9V221F4Q EXP-AWAVE yes L9V221F6Q EXP-FP yes L9V221F8Q EXP-AWAVE yes L9V221010 PROD-FP yes

To quickly see basic exposure information for each exposure listed in the association use the thselect command:

--> thselect l9v221*raw*fits[0] \ filename,detector,aperture,opt_elem,cenwave,exptype,obsmode,fppos ’yes’   l9v221euq_rawtag_a.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 1 l9v221euq_rawtag_b.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 1 l9v221ewq_rawtag_a.fits FUV WCA G130M 1309 WAVECAL TIME-TAG 1 l9v221ewq_rawtag_b.fits FUV WCA G130M 1309 WAVECAL TIME-TAG 1 l9v221eyq_rawtag_a.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 2 l9v221eyq_rawtag_b.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 2 l9v221f0q_rawtag_a.fits FUV WCA G130M 1309 WAVECAL TIME-TAG 2 l9v221f0q_rawtag_b.fits FUV WCA G130M 1309 WAVECAL TIME-TAG 2 l9v221f2q_rawtag_a.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 3 l9v221f2q_rawtag_b.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 3 l9v221f4q_rawtag_a.fits FUV WCA G130M 1309 WAVECAL TIME-TAG 3 l9v221f4q_rawtag_b.fits FUV WCA G130M 1309 WAVECAL TIME-TAG 3 l9v221f6q_rawtag_a.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 4 l9v221f6q_rawtag_b.fits FUV PSA G130M 1309 EXTERNAL/SCI TIME-TAG 4

To remove a member from association, l9v221010_asn.fits, and to use a different wavecal file, use the following commands in PyRAF.

cl> tedit l9v221010_asn.fits # row MEMNAME MEMTYPE MEMPRSNT 1 L9V221EUQ EXP-FP yes 2 L9V221EWQ EXP-AWAVE yes 3 L9V221EYQ EXP-FP yes 4 L9V221F0Q EXP-AWAVE yes 5 L9V221F2Q EXP-FP yes 6 L9V221F4Q EXP-AWAVE yes 7 L9V221F6Q EXP-FP yes 8 L9V221F8Q EXP-AWAVE yes 9 L9V221010 PROD-FP yes   Change the MEMNAME of row 8 to L9V221FSQ Change the MEMPRSNT of rows 1 and 2 to no   cl> tprint l9v221010_asn.fits # MEMNAME MEMTYPE MEMPRSNT L9V221EUQ EXP-FP no L9V221EWQ EXP-AWAVE no L9V221EYQ EXP-FP yes L9V221F0Q EXP-AWAVE yes L9V221F2Q EXP-FP yes L9V221F4Q EXP-AWAVE yes L9V221F6Q EXP-FP yes L9V221FSQ EXP-AWAVE yes L9V221010 PROD-FP yes

Finally, reprocess the data by running calcos on the updated association file. See the following section for syntaxes of running calcos.

Run Calcos

Users may choose any of three ways to run calcos using Python, PyRAF, or from the Unix/Linux/Mac command line. The input arguments and examples for each case are as follows:

1.	To run calcos in Pyraf from the hstcos package:

--> stsdas.hst_calib.hstcos --> calcos filename_asn.fits outdir=new

Table 3.2: Arguments for Running calcos in PyRAF

Argument	Values	Default	Description
input	filename		Association table (asn) or individual raw file (rawtag, rawaccum) or corrtag file to be processed
verbosity	0-2	1	Verbose output from calcos
savetmp	yes or no	no	Save temporary files: x1d_a, x1d_b, lampflash_a, and lampflash_b
outdir	directory name		The name of the ouptut directory. If blank, current directory.
find_target	yes or no	no	Have calcos find the spectrum location and centre the extraction box on that location
cutoff	-1, 0, positive floating point	-1	If specified, this value will be used as cutoff of standard deviation for find_target
shift_file	filename		File containing wavecal shifts (will override shifts calculated by calcos)
stimfile	filename		If specified, the stim positions will be written to (or appended to) this text file
livefile	filename		If specified, the livetime factors will be written to (or appended to) this text file
burstfile	filename		If specified, burst information will be written to (or appended to) this text file

2.	To run calcos in Python:

>>> import calcos >>> calcos.calcos(’filename_asn.fits’, verbosity=2, \ outdir="new")

Table 3.3: Arguments for Running calcos in Python:

Argument	Values	Default	Description
asntable	’filename’	’ ’	Association table (asn) or individual raw file (rawtag, rawaccum) to be processed
outdir	directory name	None	The name of the ouptut directory
verbosity	0, 1, 2	1	0=quiet, 1=verbose, 2=very verbose
find_target	True or False	False	Have calcos find the spectrum location and centre the extraction box on that location
shift_file	‘filename’		File containing wavecal shifts (will override shifts calculated by calcos)
save_temp_files	True or False	False	Save temporary files: x1d_a, x1d_b, lampflash_a, and lampflash_b
stimfile	"filename"		If specified, the stim positions will be written to (or appended to) this text file
livetimefile	"filename"		If specified, the livetime factors wil be written to (or appended to) this text file
burstfile	"filename"		If specified, burst information will be written to (or appended to) this text file

3.	To run calcos from the Unix/Linux/Mac command line:

% calcos -o new --stim stim.txt fileaname_asn.fits

Table 3.4: Command-line Options for Running calcos in Unix/Linux/Mac:

Option	Description
--version	print the version number and exit
-r	print the full version string and exit
-q	Quiet
-v	Very verbose
-s	save temporary file
-o outdir	Output directory
--find yes	Have calcos find Y location of spectrum
--find no	Extract spectrum at default location
--find cutoff	Find Y location if sigma <= cutoff
--shift filename	File to specify shift values
--stim filename	Append stim locations to filename
--live filename	Append livetime factors to filename
--burst filename	Append burst information to filename

To redirect the calcos STDOUT to a file use the following command:

% calcos -v -o new filename_asn.fits > log.txt

While users are recommended to run calcos on association files, it is possible to run calcos with a single raw or corrtag file as the input. In this mode, calcos will always automatically process both segment files for FUV data if they both exist. For example if rootname_rawtag_a.fits is the input for calcos, then rootname_rawtag_b.fits will automatically be processed. The data from both segments will be calibrated and combined to create the final product, rootname_x1d.fits

3.6.2 Using GO Wavecals

Through the use of associations, calcos also contains a provision to select wavecals other than the default for calibration of the science exposures. To use an exposure other than or in addition to the default wavecal, the user can add a row to the association table. The rootname (case insensitive) should be given in the MEMNAME column, the string EXP-GWAVE in the MEMTYPE column, and the value in the boolean MEMPRSNT column set to true (e.g. yes if you use the IRAF tedit task). Make sure that the WAVECORR keyword in the primary header of the raw science file is set to PERFORM, and then run calcos as normal. Note GO Wavecals can only be used with non tagflash data.