Space Telescope Science Institute
COS Data Handbook v.3
help@stsci.edu
Table of Contents Previous Next PDF


COS Data Handbook > Chapter 5: COS Data Analysis > 5.3 Working with Extracted Spectra

5.3
Here we discuss ways of customizing the extraction of spectra and modifying reference files.
5.3.1 Working With x1d Files in IDL
While STScI does not support IDL tasks, the FITS files generated by the DP and calcos pipelines can be read directly into IDL by using the mrdfits.pro task. This task and many other are available as part of a library of astronomical IDL routines available from:
http://idlastro.gsfc.nasa.gov/
5.3.2 Working With x1d Files in Python
STScI supports the FITS module located in the (python) astropy.io package. This module provides the same core functionality as the earlier module PyFITS. Any script that is still using PyFITS can be modified by importing FITS into python as follows:
from astropy.io import fits as pyfits
Information on the FITS module can be found on the astropy webpage.
5.3.3 Working With x1d Files in IRAF/PyRAF
When calibrating a single spectroscopic exposure, the calcos pipeline creates a one-dimensional extracted spectra file, with suffix x1d and a filename such as “l9v220eqq_x1d.fits”.
COS x1d files are MEF format files and their data contents and extension formats are discussed in Section 2.4.3. As with other COS data files, the primary [0] header will contain only header information and no data. The extracted spectra are stored in a single [SCI] extension as multi-dimensional binary table. A standard FITS table consists of columns and rows forming a two-dimensional grid of cells; however, each of these cells can contain a data array, effectively creating a table of higher dimensionality. Tables containing extracted COS spectra take advantage of this feature and are three-dimensional.
Using the “Selectors Syntax” to work with 3-D tables
In order to analyze COS tabular spectral data with most STSDAS tasks, you will need to use the selectors syntax to specify the desired row and column if the column contains arrays (e.g. the wavelength or flux). The general syntax for selecting a particular cell is:
intable.fits[extension number][c:column_selector][r:row_selector]
intable.fits[keyword options][c:column_selector][r:row_selector]
A column selector is a list of column patterns separated by commas. The column pattern is either a column name, a file name containing a list of column names, or a pattern using the IRAF pattern matching syntax (type "help system.match" for a description of the IRAF pattern matching syntax). To obtain a list of the column names, you can run the tlcol task (type "tlcol infile.fits").
A row selector can be used to specify a certain row in the table. For example, if you specify:
IRAF will extract data from the table stored in the first extension of infile.fits, specifically from the columns labeled WAVELENGTH and FLUX, and will restrict the extraction to the row containing segment A data. Section 2.3.2 of the Introduction to HST Data Handbooks describes the selectors syntax and provides a few examples using the selector syntax to plot COS spectra.
Dumping x1d data to an ASCII File
In PyRAF, it is possible to dump the arrays of an x1d file to an ASCII file by using the tasks txtable and tdump. For example, to extract the WAVELENGTH, FLUX and ERROR columns of FUV x1d file l9v221fkq_x1d.fits, first use the txtable task to convert the 3-D x1d table to a 2-D table:
This will create two new 2-D tables, l9v221fkq_r0001.tab and l9v211fkq_r002.tab, containing the FUVA and FUVB data respectively.
Then use the tdump task to dump the WAVELENGTH, FLUX and ERROR columns of the 2-D tables into one ASCII file using the following commands:
Plotting COS x1d Data
Each row of each of the science extensions in an x1d file will contain the columns listed in Table 2.5; a similar table, including array dimensions, can be displayed by using the task tlcol (see Section 2.3.2 of the Introduction to HST Data Handbooks).
When using many IRAF and PyRAF routines with x1d files as input, it will be necessary to specify the extension number of the file. For example, to plot flux vs. wavelength in an x1d file, using the sgraph task, the command needed would be:
cl> sgraph "l9v220eqq_x1d.fits[1][r:row=1] wavelength flux"
For FUV data, the x1d files contain both1 segments A and B. For example, to plot flux vs. wavelength in an FUV x1d file for segment A, using the sgraph task, the command needed would be:
cl> sgraph "l9v220eqq_x1d.fits[1][r:row=1] wavelength flux"
cl> sgraph "l9v220eqq_x1d.fits[1][r:SEGMENT=FUVA] wavelength flux"
To plot flux vs. wavelength in an FUV x1d file for segment B, using the sgraph task, the command needed would be:
cl> sgraph "l9v220eqq_x1d.fits[1][r:row=2] wavelength flux"
cl> sgraph "l9v220eqq_x1d.fits[1][r:SEGMENT=FUVB] wavelength flux"
For NUV data, the x1d files contain the three stripes A, B, and C. For example, to plot flux vs. wavelength in an NUV x1d file for stripe A, using the sgraph task, the command needed would be:
cl> sgraph "l9v220efq_x1d.fits[1][r:row=1] wavelength flux"
cl> sgraph "l9v220efq_x1d.fits[1][r:SEGMENT=NUVA] wavelength flux"
To plot flux vs. wavelength in an NUV x1d file for stripe B, using the sgraph task, the command needed would be:
cl> sgraph "l9v220efq_x1d.fits[1][r:row=2] wavelength flux"
cl> sgraph "l9v220efq_x1d.fits[1][r:SEGMENT=NUVB] wavelength flux"
To plot flux vs. wavelength in an NUV x1d file for stripe C, using the sgraph task, the command needed would be:
cl> sgraph "l9v220efq_x1d.fits[1][r:row=3] wavelength flux"
cl> sgraph "l9v220efq_x1d.fits[1][r:SEGMENT=NUVC] wavelength flux"
x1d files can also be displayed and analyzed using the splot routine in the noao.onedspec package of IRAF or PyRAF. This requires converting the x1d binary table first into a standard IRAF format (or OIF) which can be read by splot. The standard format is usually.imh format, containing the flux and wavelength information. The most convenient way of extracting the wavelength and flux information from the x1d table is to use tomultispec in the stsdas.hst_calib.ctools package. tomultispec takes rows from the x1d table and converts them into standard IRAF form. It fits a dispersion solution to each wavelength in the array and stores the solution in the header of the output IRAF spectrum.
If a COS x1d table contains spectra from segments A and B of the FUV channel (as would be the case if calcos were run with both rawtag_a and rawtag_b files present), then the spectrum output by tomultispec will consist of two spectra, one from each channel. Running tomultispec on an NUV x1d file will always produce a multispec file with three separate spectra, one for each stripe.
The conversion can be done using the following command:
cl> tomultispec l9v220eqq_x1d.fits l9v220eqq_x1d.imh
This command will write the spectra from both channels to a single.imh file. However, a single channel can also be written if desired by specifying the row to be written. For example, to write spectra from FUV segment A use the following syntax:
Segment B would be denoted by row=2. For NUV data the three stripes are denoted as rows 1, 2, and 3 for Stripes A, B, and C respectively. The FLUX column is extracted by default, though other columns such as GROSS, NET, or ERROR could also be extracted.
If running tomultispec produces a segmentation fault, bus error, or other equivalent issue, the problem may be solved by setting a value for IMDIR in login.cl (in particular, setting imdir=HDR$ will generally ensure that tomultispec functions as intended).
The IRAF spectrum can now be displayed using splot:
cl> splot l9v220eqq_x1d.imh
Within splot, individual emission lines can be fit with commands like ‘k’ and composite lines can be deblended using ‘d’.
5.3.4 Redoing Spectral Extraction
The x1dcorr module in calcos is designed to extract flux calibrated 1-D spectra from corrected COS event lists (corrtag files). This module is called by calcos as part of standard pipeline processing; its functioning in that role is described in Section 3.4.17.
Correcting for Shifts Along the Dispersion Direction
Properly aligning the spectrum along the dispersion direction is important not only for obtaining the correct wavelength solution, but also for properly applying the flux calibration. Incorrect registration of the spectrum will result in the wrong sensitivity being applied at each wavelength. This is especially important for low resolution spectra, since at some wavelengths the sensitivity changes rapidly with wavelength. The throughput also changes rapidly for the blue modes.
For rawtag exposures the wavecal lamp exposures are taken either concurrently with the science rawtag spectra (TAGFLASH) or they are acquired as separate rawtag spectra (Auto or GO wavecals). For all science rawaccum exposures the wavecals are acquired as separate rawtag exposures.
The wavecal exposures are used by calcos to determine the location of both the wavecal image and the corresponding science image on the detector. The locations may vary in a non-repeatable manner due to non-repeatability of the COS grating positions, but are always at a fixed position from one another. When auto-wavecals are acquired as separate exposures they are taken close in time to the science exposures, with the grating in the same position as during the science exposure.
After processing data through calcos, you may decide that you need to shift the spectrum along the dispersion direction to correct offsets in the wavelength calibration. For example, wavelength calibration offsets may occur due to offsets of the target from the center of the PSA aperture (which can occur if the target acquisition was imperfect), or from drift of the grating due to thermal flexures. Assuming that calcos has been run on the data and a residual wavelength offset has been found in the calibrated spectrum, the offset can be corrected by first calculating the number of pixels corresponding to the offset, then subtracting it from the raw position of coordinates along the dispersion direction. The shift is applied to the RAWX column of the rawtag or rawtag_(a,b) file. In IRAF and PyRAF the tcalc task from the ttools package can be used to apply the shift:
In the example above, the RAWX positions of the science spectrum in an FUV A rawtag file have been moved by SHIFT pixels. The shift is only applied to the parts of the science spectrum located between the pixels RAWY=Y1 and RAWY=Y2 along the cross-dispersion direction.
There is also a terminal command line option to apply a SHIFT1 value that is different than the one calcos finds internally. The values need to be in a separate file with the following columns included in the file: Dataset rootname, fpoffset, flash number, stripe or segment, desired SHIFT1 value, and desired SHIFT2 value. calcos will recognize “any” as a value for fpoffset if the rootname is unique (not the association rootname). It will also recognize “any” for the flash number and stripe and segment. While SHIFT1 values (dispersion direction shift) must be specified in the file, specifying SHIFT2 values (cross-dispersion direction shift) is optional. This option can be called in the terminal as:
calcos –shift shift_filename.txt rawtag.fits
or in python after importing calcos as:
calcos.calcos(rawtag.fits,shift_file=“shift_filename.txt”).
Adjusting the Background Subtraction
For spectra, background regions offset from the extraction region are used to determine the background. You can adjust the default parameters for this background region by first copying the _1dx reference file listed under the XTRACTAB keyword in the primary header (or _2dx reference file if performing a two-zone extraction) to a local directory, then adjusting the background parameters within the local version of the _1dx or _2dx reference file. Once you have adjusted the parameters to your satisfaction, edit the primary header of the rawtag file (with an IRAF task such as hedit) to indicate the path to the local version of the _1dx or _2dx file. You can then run calcos with the updated background subtraction parameters.
The background parameters available for editing in the _1dx and _2dx files are described in Section 3.4.18. The values of these parameters in the local _1dx and _2dx files can be edited with the tedit task in IRAF/PyRAF.
5.3.5 Splicing Extracted Spectra
The task splice can be applied to combine overlapping extracted COS spectra (e.g. spectra taken with different central wavelengths). Users should be aware of differences in spectral resolution at a given wavelength between different cenwaves. splice takes into account the error (ERR) array as well as the data quality (DQ) array. Handling of the DQ array is important as it helps splice perform the combination properly and avoid bad or noisy data in the output file arising from the large changes in throughput at the edges of the detector.
Please refer to the splice task help file for more useful information. If a multispec format spectrum is preferred for further analysis, the task tomultispec can be run on the output file of the splice task.
Running splice as mentioned above (rather than transforming individual x1d fits tables into multispec format before combining them) has important advantages: it keeps the science data, error, and DQ arrays intact allowing for easier error analysis, and it does not have a limitation on the number of segments or wavelengths to include, which is the case with the multispec format. This limitation is caused by the size limit of the FITS header, which required the wavelength scale to be fit with a function.
1
For FUV x1d files, both segments A and B will be present as long as the individual raw data from both segments were available at the time of processing. If only one segment was present during processing, then a row selector of row=1 will point to the data from that segment. Similarly, a row selector of row=2 will result in an error.


COS Data Handbook > Chapter 5: COS Data Analysis > 5.3 Working with Extracted Spectra

Table of Contents Previous Next PDF