Space Telescope Science Institute
COS Data Handbook 2.00
help@stsci.edu
Table of Contents Previous Next Print


COS Data Handbook > Chapter 2: COS Data Files > 2.4 COS Data Products

2.4
The following sections discuss the COS raw science data files, intermediate calibration products, final calibration products, and auxiliary data files. Uncalibrated science data include all raw science data generated during Generic Conversion that have not been processed through the calibration pipeline. These raw files are the input files to the calcos pipeline, usually as part of an association (see “Association Tables (ASN)”). The result of the pipeline is both individual calibrated exposure files and, when appropriate, a final combined product file.
2.4.1 Uncalibrated Science Data Files
Raw ACCUM Images (rawaccum)
For ACCUM data, the raw files contain a set of images, as shown in Figure 2.1, and have filenames with the suffix rawaccum for NUV data, or rawaccum_a and rawaccum_b for the two segments of the FUV detector. The SCI extension contains an image of the total accumulated counts during an exposure. For NUV data the ERR and DQ extensions have only a header with no data. For FUV data the ERR extension has only a header with no data, and the DQ extension is populated with data quality information only for pixels that are outside the subarray boundaries (defined below). The DQ extensions will be populated in the flt files, after calibration pipeline processing. Even though FUV rawaccum_a[b] data are 16384 x 1024 images, only portions of them contain actual data. These portions are called subarrays. Typically, three subarrays are used for each segment of an FUV ACCUM image. Two are centered on the STIM positions and the third is a stripe 128 pixels wide which is centered on the wavecal spectrum of the object. Figure 2.2 shows these spectral region subarrays superimposed on two FUV rawtag images. As Figure 2.2 shows, the spectrum falls outside of the subarray. Consequently, wavecals must be taken separately for ACCUM data.
Figure 2.2: Overlay of FUV ACCUM Subarrays on FUV TIME-TAG Data
The above figures shows FUV TAGFLASH data for both segments with the corresponding ACCUM subarrays noted by the dark lines. The data plotted here are the raw event locations prior to calibration processing. The distortion in the data, particularly for segment A, is very noticeable and discussed further in Section 3.4.8.
Raw TIME-TAG Events Lists (rawtag)
Raw events tables contain the locations and arrival times of individual photon events collected in TIME-TAG mode. These files have the suffix rawtag for NUV or rawtag_a[b] for the two FUV segments. Figure 2.3 shows the format of a rawtag table. The first extension contains the events list, in which each row of the table corresponds to a single event in the data stream and the columns of the table contain scalar quantities that describe the event. The second extension contains the good time intervals (GTI) table, where an uninterrupted period of time is considered as one good time interval. Interruptions in the data taking due to memory overflow could result in more than one GTI. Table 2.2 shows the columns of a rawtag table.
Figure 2.3: FITS File Format for Raw and corrected TIME-TAG Tables
Table 2.2: Columns of a Raw TIME-TAG Data Table
PHA1

1
The PHA column is present in the NUV data only for symmetry with the FUV data columns. For NUV data the values in this column are set to 0, since no pulse height amplitudes are available.

For more information on working with TIME-TAG data see Section 5.4
Pulse Height Amplitude Files (pha)
For FUV ACCUM data only, a 7 bit pulse height amplitude histogram is accumulated in the detector electronics onboard. This information is placed in a file with the suffix pha. The pulse-height histogram files contain a primary header with no data and a single FITS image SCI extension containing a histogram of the pulse-height distribution during the exposure. The pulse height amplitude files do not contain an ERR or DQ extension, as shown in Figure 2.4. The pulse height distribution is an image array of length 128, corresponding to the number of photons with values from 0 to 127, corresponding to the pulse heights of 0-31 available in TIME-TAG data.
Figure 2.4: FITS Array Extension File for COS
2.4.2 Intermediate Science Data Files
Corrected Events Lists (corrtag)
The COS pipeline produces corrected TIME-TAG events lists and stores them in binary tables with suffix corrtag. These files have a main header and three extensions: a corrected events list extension, a good time interval extension, and a time line table extension, with a format similar to the one shown in Figure 2.3. The first extension of the corrtag file is the events table (see Table 2.3) which includes X and Y event locations that have been corrected for distortion, doppler shift, and offsets due to OSM motions in both the dispersion and cross-dispersion directions. It also includes wavelengths associated with events that occur within the active area of the detectors and a data quality (DQ) flag for each event (see, Table 2.19). The second extension gives the start and stop times of the good time intervals (as in the rawtag file), and the third extension is the time line table. The time line table includes second by second values for spacecraft position, solar and target altitude above the horizon, and count rates for the most prominent airglow lines and the background. These observed rates might include counts from other external sources in addition to the ones from the airglow line itself. The data in this extension can be useful for reprocessing TIME-TAG data to exclude, for example, daytime data using the Python tool timefilter, described in Chapter 5, which is also available as an IRAF task.
For ACCUM data, the corrtag files are somewhat different. All of the time stamps in the first extension are set to the median value of the observation. Each count in the rawaccum file becomes an event so, for example, a pixel in the rawccum that that had 100 counts, would have 100 entries in the corrtag file. The RAWX, XCORR and XDOPP entries are all the same for NUV data, but can be different for FUV. In addition, RAWY and YCORR entries will have the same values. However, XFULL and YFULL can be different. In the timeline extension, the SHIFT1, airglow and DARKRATE entries are fixed, but all others are time dependent.
Table 2.3: Columns of a COS corrtag Table
XCORR1

1
The XCORR and YCORR columns are present in the NUV data only for symmetry with FUV data. Currently no distortion correction is applied to NUV data, so for NUV data the XCORR and YCORR columns are identical to the RAWX and RAWY columns.

2
The PHA column is present in the NUV data only for symmetry with the FUV data columns. For NUV data this column is set to a default value of 0, since no pulse height amplitudes are available for NUV.

Lampflash Files (lampflash)
For TAGFLASH data, calcos produces an events list with suffix lampflash, that contains the extracted wavecal lamp flashes. Each row in the events list corresponds to a different segment or stripe and flash number (the first flash is number 1, the second is number 2, etc.). The lampflash files have the format shown in Figure 2.5. The contents of the columns in a lampflash events list are listed in Table 2.4. Columns TIME, LAMP_ON, and LAMP_OFF have the same temporal zero point as the TIME column of the rawtag and corrtag tables and the same unit (seconds). The shifts contained in the SHIFT_DISP and SHIFT_XDISP columns of the lampflash table are applied to the XDOPP and YCORR columns of the corrtag file to produce the X[Y]FULL entries. When multiple TAGFLASHES are present, the shifts are interpolated in time for events occurring between the first and last flashes. Events occurring before the first flash are shifted by a value extrapolated using the slope defined by the first two flashes; events beyond the last flash are given the shift determined by the last flash. As a result, the difference between the X[Y]FULL and X[Y]CORR entries in the corrtag file can be a function of time.
Figure 2.5: FITS File Format for Lampflash Table
Table 2.4: Columns of a COS Lampflash Table
Length of the WAVELENGTH, GROSS, NET, BACKGROUND, DQ, DQ_WGT, and ERROR arrays
Dispersion direction shift(s) determined by comparing each tagflash wavecal with a wavecal template
Cross-dispersion direction shift(s) determined by comparing each tagflash wavecal with a wavecal template
Counts Files (counts)
The counts images are an intermediate calibrated output product for both imaging and spectroscopic data with suffix counts. These files contain three extensions (SCI, ERR, and DQ) as shown in Figure 2.1. These files are constructed by summing up the events from each pixel using the XFULL and YFULL coordinates. The data are in units of counts per pixel. For FUV data the images are 16384 columns in the x (dispersion) direction by 1024 rows in the y (cross-dispersion) direction. The NUV images are 1274 columns in the x direction by 1024 rows in the cross-dispersion direction for spectroscopic data, and 1024 x 1024 for data obtained in imaging mode. The NUV spectroscopic files have more pixels in the dispersion direction than the actual NUV detector. This is necessary because Doppler shifts and shifts due to OSM motions can cause wavelengths at one time during an exposure to fall outside of those obtained at another time during the exposure. As a result, the format has to be expanded to accommodate the shifts that occur during an exposure. The FUV images are not extended since the active area is less than the size of the detector, so these effects can be incorporated into the images without the need to extend them. The FUV data are also corrected for Y-walk and geometric distortions.
Flat-Fielded Image Files (flt)
For spectroscopic data a flat-fielded image is an intermediate calibrated data file. These files have a suffix flt, and contain three extensions (SCI, ERR, and DQ) as shown in Figure 2.1. These files are constructed by summing up the values in the EPSILON column for each pixel using the XFULL and YFULL coordinates. The data are in units of the count rate. For FUV data the images are 16384 x 1024, and, like the counts images, the NUV images are 1274 x 1024 for spectroscopic data and 1024 x 1024 for data obtained in imaging mode. The flt images are corrected for deadtime effects. The NUV images are corrected for all flat-field effects and the FUV data are currently only corrected for the XDL grid wire shadows.
2.4.3 Final Science Data Files (and Product Files)
The initial input files to calcos are the association tables with suffix asn. These files provide the calibration pipeline with information about how the data files are associated. In general, only exposures taken in sequence with the same spectral element, central wavelength (if applicable), and aperture at any FPPOS will be associated. However, the user can create associations that include data taken with different central wavelengths, and calcos version 2.18.5 and later will calibrate such associations. For more information on COS association files see the “Association Tables (ASN)” portion of Section 2.4.4.
Processing of each individual exposure in the association produces a final calibrated result named with exposure rootname and suffix x1d (spectroscopy) or flt (imaging).
Next, for each FPPOS position <n> (where <n>=1,2,3, or 4), if there are multiple spectroscopic exposures in the association that use the same FPPOS position, calcos will combine them into a file named with the association rootname and suffix x1dsum<n>, where <n> is the integer FPPOS value.
Lastly, a final association product file is produced with association rootname and suffix x1dsum (spectroscopy) or fltsum (imaging) by combining all science exposures in the association.
One-Dimensional Extracted Spectra (x1d, x1dsum)
The COS pipeline produces extracted one-dimensional spectra and stores them in binary tables with suffix x1d, x1dsum<n>, or x1dsum. Figure 2.6 shows the format of the 1-D extracted spectra table.
Figure 2.6: FITS File Format for 1-D Extracted Spectrum Table
These COS extracted spectra tables can be 1 to 3-Dimensional, with one row for each unique segment or stripe. For FUV data there are two rows which correspond to segments A and B distinguished by “FUVA” and “FUVB” in the SEGMENT column respectively. For NUV data there are three rows, “NUVA”, “NUVB” and “NUVC” corresponding to stripes A, B, and C respectively. Each table column can contain either a scalar value or an array of values, such as WAVELENGTH or FLUX. For example, NELEM will contain a scalar number, while the WAVELENGTH column will contain an array of wavelengths. Table 2.5 shows the contents of the different columns in an extracted spectrum table. A discussion of the data in COS extracted spectra is provided in Section 3.4.15.
Table 2.5: Columns of a COS Extracted Spectrum Table
erg s-1 cm-2 -1
erg s-1 cm-2 -1
Flat-Fielded Image Files (flt, fltsum)
For NUV imaging observations, the flt and fltsum images are the final data products, with the latter being a simple sum of the individuals when several exposures are processed together. They are fully linearized and flat-field corrected images. Unlike the flt files produced for the spectroscopic data (which are intermediate data products with a format of 1274 x 1024, see Section 2.4.2), the formats of the flt and fltsum files for imaging data are 1024 x 1024, since Doppler and OSM motions are not applied.
2.4.4 Auxiliary Data Files
Association Tables (ASN)
An association file is created for all COS observation sets, and has the suffix asn (e.g., l9v221010_asn.fits). This file holds a single binary table extension, which can be displayed with the IRAF tasks tprint or tread.
Calcos calibrates raw data from multiple science exposures and any contemporaneously obtained line lamp calibration exposures through the pipeline as an associated unit. Each individual science exposure in an association is fully calibrated in the process. See Chapter 5 of Introduction to HST Data Handbooks for a general explanation of HST data associations. The information within an association table shows how a set of exposures are related, and informs the COS calibration pipeline how to process the data.
An example association table is shown in Figure 2.7. Note that all related COS exposures will be listed in an association table, with the exception of acquisitions, darks, and flats. It is possible to have an association which contains only one exposure. The association file lists the rootnames of the associated exposures as well as their membership role in the association. The exposures listed in an association table directly correspond to individual raw FITS files. For example, the association table can describe how wavecal exposures are linked to science exposures. Table 2.6 summarizes the different exposure membership types (MEMTYPES) used for COS association tables.
Table 2.6: Member Types in COS Associations.
Figure 2.7 illustrates the contents of the association table for a sequence of spectroscopic exposures for four FPPOS positions.
Figure 2.7: Sample Association Table l9v221010_asn
In the above figure, MEMTYPE describes the exposure membership type or role in the association. The column MEMPRSNT lists whether the member is present or not. A user could choose to change the association file to not include a member during processing by changing the MEMPRSNT to ‘no’.
The association table above lists the names of the eight associated exposures (four external and four calibration) that are calibrated and combined to create the various association products which will have a rootname of l9v221010. This particular association is created from a single TIME-TAG spectroscopic APT specification with FPPOS=ALL and FLASH=NO specified in the Phase II file, which leads to both a science exposure and automatic wavecal exposure taken at each FPPOS location. For example, the first entry in the table, l9v221euq, is the rootname of a single external science exposure taken with FPPOS=1. This exposure corresponds to the following rawtag files: l9v221euq_rawtag_a.fits, l9v221euq_rawtag_b.fits. The memtype of this exposure is EXP-FP which shows that it is an external exposure. The second entry in the table has a memtype of EXP-AWAVE. This denotes that the corresponding rawtag exposures, l9v221ewq_rawtag_a.fits and l9v221ewq_rawtag_b.fits, are wavecal exposures that will be used by the pipeline for wavelength calibration. Similar files correspond to the remaining three pairs of entries in the association file for data taken with the remaining three FPPOS positions. The pipeline will calibrate the members of an association as a unit, producing the calibrated data products for each individual exposure as well as the final combined association data product. For this particular association, the pipeline will produce a final combined association product, l9v221010_x1dsum.fits, which contains the final FPPOS combined, calibrated spectrum.
Trailer Files (TRL)
When COS data are processed through OTFR in the HDA, the output messages from generic conversion and the different calibration steps are stored in a FITS ASCII table known as the trailer file, with suffix trl. Each time the archive processes data before retrieval, the old trailer file is erased and a new one created using the results of the most recent processing performed. The archive will produce a trailer file for each individual exposure and association product. Association product trailer files contain the appended information from all the exposures in the association, in order of processing. The order of processing is the same as the order of exposures in the association table, with the exception of auto or GO wavecals which are always processed first.
In the trailer files from the HDA, the output messages from generic conversion appear first in the file. This section contains information relevant to the selection of the best reference files and the population of some of the header keywords. The second part of this file contains information from calcos processing. Each task in the calcos pipeline creates messages during processing which describe the progress of the calibration, and appear in the order in which each step was performed. These messages are quite relevant to understanding how the data were calibrated, and in some of the cases, to determining the accuracy of the products.
In this last section of the _trl file, the calcos steps are indicated by their module name. The calcos messages provide information on the input and output files for each step, the corrections performed, information regarding the reference files used, and in the case of FUV data, messages about the location of the stims, or shift correction applied to the data. Calcos also gives warnings when the appropriate correction to the data could not be applied. For more detailed information on the calibration steps and structure of calcos, please refer to Chapter 3.
Calcos Trailer Files (TRA)
When calcos is run in a user’s home environment, calcos redirects the output of its steps to the STDOUT and an ASCII file with name rootname.tra. Note, the level of detail included in the output messages can be modified when running calcos (see “Run Calcos” ). So, when run on a personal machine, calcos will not overwrite the trl file but rather will direct the output to STDOUT and an ASCII tra file. The tra file is formatted like the trl file but with two exceptions: the tra file will not contain the output messages from generic conversion, and the tra file is not converted to FITS format. Therefore, one must look at both the trailer (generic conversion messages) and tra (calibration messages) file when running calcos from a home environment. Each time calcos is run on a file, the STDOUT messages will be appended to the tra file if it already exists. Also, when running calcos on a personal machine there will be no tra created for the association products. Instead, the calcos messages for association products will be sent only to STDOUT.
Support Files (SPT)
The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. A COS support file contains a primary header and at least three FITS image extensions. The first extension contains a header with the proposal information and an (16-bit) image array containing the data which populate the spt header keyword values. The image array element values are used by conversion software to populate the header keywords. Following the support extension, the COS spt files contain two engineering snapshot extensions. These extensions contain a readout of several instrument and telescope parameters from telemetry data at different times during the course of an exposure. The very first snapshot extension will always contain telemetry information from the beginning of an exposure. Depending on the length of the exposure, the support file may also contain one or several “imsets” which include a support extension and two snap extensions. These intermediate imsets will have only their second snapshot extension populated with telemetry data taken during the course of an exposure, while the first snapshot will be populated with default values. The very last imset of an spt file will have all three extensions (1 support and 2 snaps) populated with telemetry values at the completion of the exposure. Figure 2.8 depicts the structure of an N extension COS support file.
With several snapshots of COS telemetry values, one may track the instrument status periodically throughout an exposure. For a schematic listing of the spt headers with detailed information about the spt header keywords, See:
http://stdatu.stsci.edu/keyword/cgi-bin/kdct-header.cgi?i=COS&s=20.1&db=Operational
Figure 2.8: COS Support File
COS support file with N extensions. The initial imset contains telemetry values at the start of the exposure. Extensions 3 through (N-3) contain imsets with telemetry values at intermediate times during the exposure. Note that the first snap extensions in these intermediate imsets are NOT populated. The final imset includes extensions N-2 through N and contains telemetry values at the end of the exposure. Both snap extensions are populated for the final imset.
Acquisition Files (RAWACQ)
All COS acquisition exposures will produce a single raw data file with suffix rawacq. Almost all COS spectroscopic science exposures are preceded by an acquisition sequence or exposure to center the target in the aperture. Keywords in the header of COS science data identify the exposure names of relevant acquisition exposures in each visit. In addition, there are several other useful keywords in the COS acquisition exposures that describe the acquisition parameters used, as well as the calculated centroid positions and slew offsets. Table 2.7 lists all the relevant acquisition keywords.
Table 2.7: ACQ/IMAGE Header Keywords.
ACQSNAME1

1
These keywords are also found in the COS science headers in addition to being in the acquisition headers.

Table 2.8: ACQ/SEARCH Header Keywords.
ACQSNAME1
SEGMENT2
TRGSUBX13

1
These keywords are also found in the COS science headers in addition to being in the acquisition headers.

2
FUV only

3
NUV only

 
ACQSNAME1
LMPSUBX12
SEGMENT3

1
These keywords are also found in the COS science headers in addition to being in the acquisition headers.

2
NUV only

3
FUV only

 
SEGMENT2
TRGSUBX13

1
These keywords are also found in the COS science headers in addition to being in the acquisition headers.

2
FUV only

3
NUV only

PEAKD and SEARCH Acquisitions:
Acquisition peakups in the dispersion direction (ACQ/PEAKD) and acquisition spiral searches (ACQ/SEARCH) both use the flux from exposures taken at different dwell points to center the target. For more information on these types of COS acquisitions see Sections 7.6.4 and 7.6.2 respectively of the COS Instrument Handbook. Data for these acquisitions contain one binary table extension which describes the acquisition search pattern dwell point locations and counts as shown in Table 2.11 and Figure 2.9
Table 2.11: Columns of an ACQ/SEARCH or ACQ/PEAKD Table
DISP_OFFSET1

1
This column is only present in ACQ/SEARCH tables

Figure 2.9: FITS File Format for ACQ/SEARCH and ACQ/PEAKD Data.
PEAKXD Acquisition:
Acquisition peakups in the cross-dispersion direction (ACQ/PEAKXD) use a TIME-TAG spectrum to center the target in the cross-dispersion direction. For more information on the ACQ/PEAKXD algorithm see Section 7.6.3 of the COS Instrument Handbook. An ACQ/PEAKXD exposure includes only a primary header and extension header. There are no data downlinked for this type of acquisition.
IMAGE Acquisition:
Acquisition images (ACQ/IMAGE) use a NUV image to center the target in the aperture. For more information on the ACQ/IMAGE algorithm see Section 7.5 of the COS Instrument Handbook. An ACQ/IMAGE exposure produces a raw data file containing two science image extensions corresponding to the initial and final pointing:
See Figure 2.10 for the FITS file format for ACQ/IMAGE data.
Figure 2.10: FITS File Format for ACQ/IMAGE Data
Jitter Files (jit)
The COS jitter files include engineering data that describe the performance of the Pointing Control System (PCS) including the Fine Guidance Sensors that are used to control the vehicle pointing. The jitter files report on PCS engineering data during the duration of the observation. The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. COS jitter files utilize the file format shown in Figure 2.11 for all science observations, excluding acquisitions.
Figure 2.11: FITS File Format for jitter Data.
The jitter tables contain PCS data for each three second interval during the observation, as listed in Table . For more information on jitter files refer to Chapter 6 of Introduction to HST Data Handbooks.
 
2-D Spacecraft Pointing Histogram (jif)
The COS jif files are a 2-D histogram of the corresponding jit file (See Section , “Jitter Files (jit)”) and have the file format shown in Figure 2.12 for all science observations excluding acquisitions.
Figure 2.12: FITS File Format for jif Data.

Table of Contents Previous Next Print