|Space Telescope Science Institute|
|COS Data Handbook v.3|
Sometimes the pipeline calibration performed shortly after the data were obtained from the telescope is not the best possible calibration for your science program. There are a number of reasons why it may be desirable to recalibrate your data. The most likely reasons include:
• Some steps need to be repeated with different input parameters. For example, you may wish to re-perform the 1-D spectral extraction with a smaller BOXCAR height in order to minimize the background, or you may wish to cut a TIME-TAG exposure into sub-exposures, in order to study time variability.In the first case, we recommend simply re-requesting the data from the archive, giving a reduction produced with the latest reference files. In the second case, to tailor the calibration to your individual preferences, it may be beneficial to run calcos yourself on your local machine, or to use tasks that improve the reference files or allow customized treatment of the data. Calcos can be imported and executed when running PyRAF or Python.
Be sure you are using the latest versions of the calcos and STSDAS software, COS calibration files, and raw data files (which list the latest reference files in their headers). STSDAS release information can be found at www.stsci.edu/resources/software_hardware/stsdas.Calcos contains provisions for recalibrating raw data. Users can specify the pipeline processing steps to be performed and select the associated reference files. However, calcos was not designed to run its various modules independently, i.e. it is not re-entrant. The pipeline flow is modified by setting calibration switches or reference file names and then rerunning the entire pipeline. The calibration switches in the headers of the calibrated data files will reflect the operations performed on the calibrated data and the reference files used.If you chose to recalibrate your COS data on your local machine, there is a certain amount of set up required for calcos to run properly. The operations mentioned in the checklist below will be described in detail in the following subsections:
• Set the environment variable lref to point to your reference file directory. Note: you must do this before starting a PyRAF session!
• Update the input data file headers (including reference file names). In IRAF, this would be done using thedit.
• Run calcos.Before running calcos, you will need to define an environment variable to indicate the location of the directory containing the needed calibration reference files. The names of the calibration files are preceded with the logical path name “lref$” in the COS science headers. Ordinarily you would define this directory in a PyRAF session to be, for example, “/data/vega3/cos/lref/” using the set command:
Note the trailing slash (/). However, calcos and all of its modules are actually foreign tasks and as such do not access PyRAF environment variables. Therefore, before invoking the cl, you will need to define an environment variable from the host command line (see below) that is appropriate to your host machine. For Unix/Linux/Mac systems, the appropriate command for the example above is:
Note that an alternative to using the lref$ variable is specifying the full pathnames to the reference files in the science headers.
When running calcos or any of its modules, you must define environment variables (such as lref$) before starting the cl. It is not possible to define them within IRAF using the set command, nor is it possible to define them with an escape to the host level, such as:
!setenv lref /data/vega3/cos/lref/To recalibrate your data, you will need to retrieve the reference files used by the different calibration steps to be performed. The names of the reference files to be used during calibration must be specified in the primary header of the input files, under the section “CALIBRATION REFERENCE FILES.” Note that the data headers will be populated already with the names of the reference files used during pipeline calibration at STScI.Chapter 1 of the Introduction to HST Data Handbooks describes how to retrieve data and reference files via the World Wide Web. To retrieve the best reference files via MAST (generally meaning the most recent reference files), check “Best Reference Files” in the “Reference Files” section of the Retrieval Options form.The COS reference files are all in FITS format, and can be in either IMAGE or BINTABLE extensions. The names of these files along with descriptions of their contents are given in Section 3.7. The rootname of a reference file is based on the time that the file was delivered to the Calibration Reference Data System (CRDS).To edit file headers in preparation for recalibration, use the STSDAS task thedit. The thedit task takes several input parameters: the name(s) of the raw data files to be edited, the header field to be edited, and the new value of the header field. It can be used to change the values of any calibration switches, reference files or tables to the values you wish to use for recalibrating your data. To edit the calibration keyword values:
1. Run the thedit task, specifying a list of files in which you want to change calibration keyword values. You may specify more than one file (using wildcards) to be updated. For example, you could change the flat reference file to be used for all COS raw science files in the current directory using the command:
There are straightforward ways to do this in python and IDL too.
If you are changing keywords that reside in the FITS primary header unit with hedit or thedit, be sure to explicitly specify the primary header by appending “” to the FITS file name.Users may find it necessary to edit the input association file for calcos. Reasons for editing an association file might include the use of a different wavecal or to remove a compromised exposure from an association. For this option, the full file name (but not the directory) must be given, and the case must be correct. One way to update an association file is to use the STSDAS task, tedit. For example, use the PyRAF task tprint to first look at the contents of association table, l9v221010_asn.fits.
To quickly see basic exposure information for each exposure listed in the association use the thselect command:
--> thselect l9v221*raw*fits \ filename,detector,aperture,opt_elem,cenwave,exptype,obsmode,fppos ’yes’To remove a member from association, l9v221010_asn.fits, and to use a different wavecal file, use the following commands in PyRAF.
Finally, reprocess the data by running calcos on the updated association file. See the following section for syntaxes of running calcos.Run calcosUsers may choose any of three ways to run calcos using PyRAF, Python, or from the Unix/Linux/Mac command line. The input arguments and examples for each case are as follows:
1. To run calcos using TEAL in PyRAF using the Python package costools (the splittag, timefilter, and x1dcorr modules are in costools):
Table 3.2: Arguments for Running calcos in PyRAF
Association table (asn) or individual raw file (rawtag, rawaccum) or corrtag file to be processed Verbose output from calcos Save temporary files: x1d_a, x1d_b, lampflash_a, and lampflash_b Have calcos find the spectrum location and centre the extraction box on that location
2. To run calcos in Python:
Table 3.3: Arguments for Running calcos in Python:
Association table (asn) or individual raw file (rawtag, rawaccum) to be processed Have calcos find the spectrum location and center the extraction box on that location Save temporary files: x1d_a, x1d_b, lampflash_a, and lampflash_b
3. To run calcos from the Unix/Linux/Mac command line:
Table 3.4: Command-line Options for Running calcos in Unix/Linux/Mac:
Have calcos find Y location of spectrum To redirect the calcos STDOUT to a file use the following command:
While we recommend that users run calcos on association files, it is possible to run calcos with a single raw or corrtag file as the input. In this mode, calcos will always automatically process both segment files for FUV data if they both exist. For example if rootname_rawtag_a.fits is the input for calcos, then rootname_rawtag_b.fits will automatically be processed. The data from both segments will be calibrated and combined to create the final product, rootname_x1d.fits.Running calcos on rawtag or corrtag files instead of the asn file will cause the FUVB-only blue modes (G130M cenwaves 1055 and 1096) to be calibrated without the associated segment A EXP-IWAVE file contained in the asn.3.6.2 Using GO WavecalsThrough the use of associations, calcos also contains a provision to select wavecals other than the default for calibration of the science exposures. To use an exposure other than or in addition to the default wavecal, the user can add a row to the association table. The rootname (case insensitive) should be given in the MEMNAME column, the string EXP-GWAVE in the MEMTYPE column, and the value in the boolean MEMPRSNT column set to true (e.g. yes if you use the IRAF tedit task). Make sure that the WAVECORR keyword in the primary header of the raw science file is set to PERFORM, and then run calcos as normal. Note GO Wavecals can only be used with non tagflash data.The new TWOZONE extraction algorithm and the associated pipeline reference files are optimized for the spectral extraction of bright point source spectra. There are, however, a number of circumstances under which a customized extraction might yield better results.
• If the user simply wishes to use the BOXCAR extraction in place of the TWOZONE algorithm, EXTALG should be set to “BOXCAR”, and TRCECORR and ALGNCORR should be set to “OMIT”. This will use the larger extraction regions defined for that algorithm. However, for observations at LP3 there may be significant overlap with the gain-sagged regions near LP1, and this may affect the accuracy of the calibration or even create artificial spectral features.Both the XTRACTAB, which is used with the BOXCAR algorithm, and the TWOZXTAB, which is used with the TWOZONE algorithm, contain columns named HEIGHT and B_SPEC. For the BOXCAR algorithm, these parameters together with the SLOPE column directly control the size and location of the extraction region. For the TWOZONE algorithm, the HEIGHT and BSPEC numbers instead control the size and initial location of the region used for the ALGNCORR step. The HEIGHT column in the TWOZONE algorithm is also used to define the cross-dispersion width of the reference profile that is assumed to include 100% of the enclosed energy. The actual extraction region at each wavelength is adjusted so that the enclosed energy fraction of the reference profile matches the values given in the LOWER_OUTER and UPPER_OUTER columns of the TWOZXTAB. For example, if LOWER_OUTER=0.005 and UPPER_OUTER=0.995, at each wavelength the extraction region will be adjusted so that the central 99% of the encircled energy as measured from the reference profile is included. Fractional pixel locations are rounded outwards, and the final extracted flux will be scaled for the exact encircled energy fraction in each column.To force a spectral extraction using the TWOZONE algorithm to sum over a region that contains only the central 80% of the reference profile’s encircled energy, the user would just need to change LOWER_OUTER to 0.1 and UPPER_OUTER to 0.9 in the appropriate row of the TWOZXTAB prior to recalibration of the data.Values of 0 or 1 for the enclosed energy boundaries have a special meaning. Setting the lower boundary to a value of 0 forces the extraction to start at the bottom of the region defined by a rectangular box of size HEIGHT, while setting the upper boundary to 1, forces it to end at the upper boundary. This can be used to give a rectangular extraction box rather than the wavelength dependent extraction region normally used for the TWOZONE algorithm. For a very extended target, it might be useful to force the use of the full height box, and also increase the HEIGHT allowing a further expansion of the extraction region.The LOWER_INNER and UPPER_INNER columns in the TWOZXTAB behave very similarly to the “OUTER” boundaries, except that they are used to control the region over which data quality flags are combined rather than the region over which counts are summed. The user can also adjust these values.The background regions in the TWOZONE algorithm are handled in a simpler fashion. To change where the background regions are located or the height of the background regions, edit the background centers (B_BKG1 and B_BKG2) and the background height (BHEIGHT). The background regions should not be placed directly above the spectrum at LP3, as that is where LP1 is located, and the detector is therefore very gain-sagged in that location. Also ensure that the background regions do not overlap the WCA (location found in XTRACTAB).The user can also override the shifts calculated by ALGNCORR. This can be useful if the automatic algorithm failed to properly center the target. To do this, the user should set the keyword SP_SET_A, (for detector segment FUVA), or SP_SET_B, (for FUVB), to the desired offset value which will be used in place of the SP_OFF_A or SP_OFF_B value calculated by the ALGNCORR algorithm. These keywords should be set in the extension header of the rawtag or corrtag file used as input for calcos.