Space Telescope Science Institute
COS Data Handbook 2.00
Table of Contents Previous Next Print

COS Data Handbook > Chapter 5: COS Data Analysis > 5.3 Working with Extracted Spectra

Here we discuss ways of customizing the extraction of spectra and modifying reference files.
5.3.1 Working With x1d Files in IDL
While STScI does not support IDL tasks, the FITS files generated by the OPUS and calcos pipelines can be read directly into IDL by using the task. This task and many other are available as part of a library of astronomical IDL routines available from:
5.3.2 Working With x1d Files in Python
STScI provides support for PyFITS, a module that provides an interface to FITS formatted files in the Python scripting language and PyRAF, the Python-based interface to IRAF. It is useful both for interactive data analysis and for writing analysis scripts in Python using FITS files as either input or output. More information, including the PyFITS handbook, can be found at
5.3.3 Working With x1d Files in IRAF/PyRAF
When calibrating a single spectroscopic exposure, the calcos pipeline creates a one-dimensional extracted spectra file, with suffix x1d and a filename such as “l9v220eqq_x1d.fits”.
COS x1d files are MEF format files and their data contents and extension formats are discussed in Section 2.4.3. As with other COS data files, the primary [0] extension will contain only header information, but no data. The extracted spectra are stored in a single [SCI] extension as multi-dimensional binary table. A standard FITS table consists of columns and rows forming a two-dimensional grid of cells; however, each of these cells can contain a data array, effectively creating a table of higher dimensionality. Tables containing extracted COS spectra take advantage of this feature and are three-dimensional.
Using the “Selectors Syntax” to work with 3-D tables
In order to analyze COS tabular spectral data with STSDAS tasks other than sgraph and igi (which have been appropriately modified to handle three-dimensional data tables), you will need to extract the desired arrays from the three-dimensional table. Two IRAF tasks, named tximage and txtable, can be used to extract the table-cell arrays. Complementary tasks, named tiimage and titable, will insert arrays back into table cells. The task tscopy will copy rows, columns, and subsets of tables. To specify the arrays which should be extracted from or inserted into the table cells, you will need to use the selectors syntax to specify the desired row and column. The general syntax for selecting a particular cell is:
intable.fits[extension number][c:column_selector][r:row_selector]
intable.fits[keyword options][c:column_selector][r:row_selector]
A column selector is a list of column patterns separated by commas. The column pattern is either a column name, a file name containing a list of column names, or a pattern using the IRAF pattern matching syntax (type "help system.match" for a description of the IRAF pattern matching syntax). To obtain a list of the column names, you can run the tlcol task (type "tlcol infile.fits").
A row selector can be used to specify a certain row in the table. For example, if you specify:
IRAF will extract data from the table stored in the first extension of infile.fits, specifically from the columns labeled WAVELENGTH and FLUX, and will restrict the extraction to the row containing segment A data. Section 2.3.2 of the Introduction to HST Data Handbooks describes the selectors syntax and provides a few examples using the selector syntax to plot COS spectra.
Dumping x1d data to an ASCII File
In PyRAF, it possible to dump the arrays of an x1d file to an ASCII file by using the tasks txtable and tdump. For example, to extract the WAVELENGTH, FLUX and ERROR columns of FUV x1d file l9v221fkq_x1d.fits, first use the txtable task to convert the 3-D x1d table to a 2-D table:
This will create two new 2-D tables, and, containing the FUVA and FUVB data respectively.
Then use the tdump task to dump the WAVELENGTH, FLUX and ERROR columns of the 2-D tables into one ASCII file using the following commands:
Plotting COS x1d Data
Each row of each of the science extensions in an x1d file will contain the columns listed in Table 2.5; a similar table, including array dimensions, can be displayed by using the task tlcol (see Section 2.3.2 of the Introduction to HST Data Handbooks).
When using many IRAF and PyRAF routines with x1d files as input, it will be necessary to specify the extension number of the file. For example, to plot flux vs. wavelength in an x1d file, using the sgraph task, the command needed would be:
cl> sgraph "l9v220eqq_x1d.fits[1][r:row=1] wavelength flux"
For FUV data, the x1d files contain both1 segments A and B. For example, to plot flux vs. wavelength in an FUV x1d file for segment A, using the sgraph task, the command needed would be:
cl> sgraph "l9v220eqq_x1d.fits[1][r:row=1] wavelength flux"
cl> sgraph "l9v220eqq_x1d.fits[1][r:SEGMENT=FUVA] wavelength flux"
To plot flux vs. wavelength in an FUV x1d file for segment B, using the sgraph task, the command needed would be:
cl> sgraph "l9v220eqq_x1d.fits[1][r:row=2] wavelength flux"
cl> sgraph "l9v220eqq_x1d.fits[1][r:SEGMENT=FUVB] wavelength flux"
For NUV data, the x1d files contain the three stripes A, B, and C. For example, to plot flux vs. wavelength in an NUV x1d file for stripe A, using the sgraph task, the command needed would be:
cl> sgraph "l9v220efq_x1d.fits[1][r:row=1] wavelength flux"
cl> sgraph "l9v220efq_x1d.fits[1][r:SEGMENT=NUVA] wavelength flux"
To plot flux vs. wavelength in an NUV x1d file for stripe B, using the sgraph task, the command needed would be:
cl> sgraph "l9v220efq_x1d.fits[1][r:row=2] wavelength flux"
cl> sgraph "l9v220efq_x1d.fits[1][r:SEGMENT=NUVB] wavelength flux"
To plot flux vs. wavelength in an NUV x1d file for stripe C, using the sgraph task, the command needed would be:
cl> sgraph "l9v220efq_x1d.fits[1][r:row=3] wavelength flux"
cl> sgraph "l9v220efq_x1d.fits[1][r:SEGMENT=NUVC] wavelength flux"
x1d files can also be displayed and analyzed using the splot routine in the noao.onedspec package of IRAF or PyRAF. This requires converting the x1d binary table first into a standard IRAF format (or OIF) which can be read by splot. The standard format is usually.imh format, containing the flux and wavelength information. The most convenient way of extracting the wavelength and flux information from the x1d table is to use tomultispec in the stsdas.hst_calib.ctools package. tomultispec takes rows from the x1d table and converts them into standard IRAF form. It fits a dispersion solution to each wavelength in the array and stores the solution in the header of the output IRAF spectrum.
If a COS x1d table contains spectra from segments A and B of the FUV channel (as would be the case if calcos were ran with both _rawtag_a and _rawtag_b files present), then the spectrum output by tomultispec will consist of two spectra, one from each channel. Running tomultispec on an NUV x1d file will always produce a multispec file with three separate spectra, one for each stripe.
The conversion can be done using the following command:
cl> tomultispec l9v220eqq_x1d.fits l9v220eqq_x1d.imh
This command will write the spectra from both channels to a single.imh file. However, a single channel can also be written if desired by specifying the row to be written. For example, to write spectra from FUV segment A use the following syntax:
Segment B would be denoted by row=2. For NUV data the three stripes are denoted as rows 1, 2, and 3 for Stripes A, B, and C respectively. The FLUX column is extracted by default, though other columns such as GROSS, NET, or ERROR could also be extracted.
If running tomultispec produces a segmentation fault, bus error, or other equivalent issue, the problem may be solved by setting a value for IMDIR in (in particular, setting IMDIR=HDR$ will generally ensure that tomultispec functions as intended).
The IRAF spectrum can now be displayed using splot:
cl> splot l9v220eqq_x1d.imh
Within splot, individual emission lines can be fit with commands like ‘k’ and composite lines can be deblended using ‘d’.
5.3.4 Redoing Spectral Extraction
The x1dcorr module in calcos is designed to extract flux calibrated 1-D spectra from corrected COS event lists (corrtag files). This module is called by calcos as part of standard pipeline processing; its functioning in that role is described in Section 3.4.15.
Correcting for Shifts Along the Dispersion Direction
Properly aligning the spectrum along the dispersion direction is important not only for obtaining the correct wavelength solution, but also for properly applying the flux calibration. Incorrect registration of the spectrum will result in the wrong sensitivity being applied at each wavelength. This is especially important for low resolution spectra, since at some wavelengths the sensitivity changes rapidly with wavelength.
For rawtag exposures the wavecal lamp exposures are taken either concurrently with the science rawtag spectra (TAGFLASH) or they are acquired as separate rawtag spectra (Auto or GO wavecals). For all science rawaccum exposures the wavecals are acquired as separate rawtag exposures.
The wavecal exposures are used by calcos to determine the location of both the wavecal image and the corresponding science image on the detector. The location may vary in a non-repeatable manner due to non-repeatability of the COS grating positions. When auto-wavecals are acquired as separate exposures they are taken close in time to the science exposures, with the grating in the same position as during the science exposure.
After processing data through calcos, you may decide that you need to shift the spectrum along the dispersion direction to correct offsets in the wavelength calibration. For example, wavelength calibration offsets may occur due to offsets of the target from the center of the PSA aperture (which can occur if the target acquisition was imperfect), or from repositioning of the grating due to thermal flexures. Assuming that calcos has been run on the data and a residual wavelength offset has been found in the calibrated spectrum, the offset can be corrected by first calculating the number of pixels corresponding to the offset, then subtracting it from the raw position of coordinates along the dispersion direction. The shift is applied to the RAWX column of the rawtag or rawtag_(a,b) file. In IRAF and PyRAF the tcalc task from the ttools package can be used to apply the shift:
In the example above, the RAWX positions of the science spectrum in an FUV A rawtag file have been moved by SHIFT pixels. The shift is only applied to the parts of the science spectrum located between the pixels RAWY=Y1 and RAWY=Y2 along the cross-dispersion direction.
Adjusting the Background Subtraction
For spectra, a background region offset from the extraction region is used to determine the background. You can adjust the default parameters for this background region by first copying the _1dx reference file listed under the XTRACTAB keyword in the primary header to a local directory, then adjusting the background parameters within the local version of the _1dx reference file. Once you have adjusted the parameters to your satisfaction, edit the primary header of the _rawtag file (with an IRAF task such as hedit) to indicate the path to the local version of the _1dx file. You can then run calcos with the updated background subtraction parameters.
The background parameters available for editing in the _1dx file are:
The values of these parameters in the local _1dx file can be edited with the tedit task in IRAF/PyRAF.
5.3.5 Splicing Extracted Spectra
The task splice can be applied to combine overlapping extracted COS spectra (e.g. spectra taken with different central wavelengths). It takes into account the error (ERR) array as well as the data quality (DQ) array. Handling of the DQ array is important as it helps splice perform the combination properly and avoid bad or noisy data in the output file arising from the large changes in throughput at the edges of the detector.
Please refer to the splice task help file for more useful information. If a multispec format spectrum is preferred for further analysis, the task tomultispec can be run on the output file of the splice task.
Running splice as mentioned above (rather than transforming individual x1d fits tables into multispec format before combining them) has important advantages: it keeps the science data, error, and DQ arrays intact allowing for easier error analysis, and it does not have a limitation on the number of segments or wavelengths to include, a problem with the multispec format due to the limit on the size of the FITS header which requires fitting the wavelength scale with a function.

For FUV x1d files, both segments A and B will be present as along as the individual raw data from both segments were available at the time of processing. If only one segment was present during processing, then a row selector of row=1 will point to the data from that segment. Similarly, a row selector of row=2 will result in an error.

Table of Contents Previous Next Print