The following sections discuss the COS raw science data files, intermediate
calibration products, final calibration products, and auxiliary data files. Uncalibrated science data include all raw science data generated during Generic Conversion that have not been processed through the calibration pipeline. These raw files are the input files to the calcos
pipeline, usually as part of an association (see “Association Tables (ASN)”
). The result of the pipeline is both individual calibrated exposure files and, when appropriate, a final combined product file.
data, the raw files contain a set of images, as shown in Figure 2.1
, and have filenames with the suffix rawaccum
for NUV data, or rawaccum_a
for the two segments of the FUV detector. The SCI extension contains an image of the total accumulated counts during an exposure. For NUV data the ERR and DQ extensions have only a header with no data. For FUV data the ERR extension has only a header with no data, and the DQ extension is populated with data quality information only for pixels that are outside the subarray boundaries (defined below). The DQ extensions will be populated in the flt
files, after calibration pipeline processing. Even though FUV rawaccum_a[b]
data are 16384 x 1024 images, only portions of them contain actual data. These portions are called subarrays. Typically, three subarrays are used for each segment of an FUV ACCUM
image. Two are centered on the STIM positions and the third is a stripe 128 pixels wide which is centered on the wavecal spectrum of the object. Figure 2.2
shows these spectral region subarrays superimposed on two FUV rawtag
images. As Figure 2.2
shows, the spectrum falls outside of the subarray. Consequently, wavecals must be taken separately for ACCUM
Events Lists (rawtag
Raw events tables contain the locations and arrival times of individual photon
events collected in TIME-TAG
mode. These files have the suffix rawtag
for NUV or rawtag_a[b]
for the two FUV segments. Figure 2.3
shows the format of a rawtag
table. The first extension contains the events list, in which each row of the table corresponds to a single event in the data stream and the columns of the table contain scalar quantities that describe the event.
The second extension contains the good time intervals (GTI) table, where an uninterrupted period of time is considered as one good time interval. Interruptions in the data taking due to memory overflow could result in more than one GTI. Table 2.2
shows the columns of a rawtag
For FUV ACCUM
data only, a 7 bit pulse height amplitude histogram is accumulated in the detector electronics onboard. This information is placed in a file with the suffix pha
. The pulse-height histogram files contain a primary header with no data and a single FITS image SCI extension containing a histogram of the pulse-height distribution during the exposure. The pulse height amplitude files do not contain an ERR or DQ extension, as shown in Figure 2.4
. The pulse height distribution is an image array of length 128, corresponding to the number of photons with values from 0 to 127, corresponding to the pulse heights of 0-31 available in TIME-TAG
The COS pipeline produces corrected TIME-TAG
events lists and stores them in binary tables with suffix corrtag
. These files have a main header and three extensions: a corrected events list extension, a good time interval extension, and a time line table extension, with a format similar to the one shown in Figure 2.3
. The first extension of the corrtag file is the events table (see Table 2.3
) which includes X
event locations that have been corrected for distortion, doppler shift, and offsets due to OSM motions in both the dispersion and cross-dispersion directions. It also includes wavelengths associated with events that occur within the active area of the detectors and a data quality (DQ) flag for each event (see, Table 2.19
). The second extension gives the start and stop times of the good time intervals (as in the rawtag
file), and the third extension is the time line table. The time line table includes second by second values for spacecraft position, solar and target altitude above the horizon, and count rates for the most prominent airglow lines and the background. These observed rates might include counts from other external sources in addition to the ones from the airglow line itself. The data in this extension can be useful for reprocessing TIME-TAG
data to exclude, for example, daytime data using the Python tool timefilter, described in Chapter 5
, which is also available as an IRAF task.
data, the corrtag
files are somewhat different. All of the time stamps in the first extension are set to the median value of the observation. Each count in the rawaccum
file becomes an event so, for example, a pixel in the rawccum
that that had 100 counts, would have 100 entries in the corrtag
file. The RAWX
entries are all the same for NUV data, but can be different for FUV. In addition, RAWY
entries will have the same values. However, XFULL
can be different. In the timeline extension, the SHIFT1
, airglow and DARKRATE
entries are fixed, but all others are time dependent.
For TAGFLASH data, calcos produces an events list with suffix lampflash, that
contains the extracted wavecal lamp flashes. Each row in the events list corresponds to a different segment or stripe and flash number (the first flash is number 1, the second is number 2, etc.). The lampflash files have the format shown in Figure 2.5. The contents of the columns in a lampflash events list are listed in Table 2.4. Columns TIME, LAMP_ON, and LAMP_OFF have the same temporal zero point as the TIME column of the rawtag and corrtag tables and the same unit (seconds). The shifts contained in the SHIFT_DISP and SHIFT_XDISP columns of the lampflash table are applied to the XDOPP and YCORR columns of the corrtag file to produce the X[Y]FULL entries. When multiple TAGFLASHES are present, the shifts are interpolated in time for events occurring between the first and last flashes. Events occurring before the first flash are shifted by a value extrapolated using the slope defined by the first two flashes; events beyond the last flash are given the shift determined by the last flash. As a result, the difference between the X[Y]FULL and X[Y]CORR entries in the corrtag file can be a function of time.
The counts images are an intermediate calibrated output product for both imaging
and spectroscopic data with suffix counts
. These files contain three extensions (SCI, ERR, and DQ) as shown in Figure 2.1
. These files are constructed by summing up the events from each pixel using the XFULL and YFULL coordinates. The data are in units of counts per pixel. For FUV data the images are 16384 columns in the x (dispersion) direction by 1024 rows in the y (cross-dispersion) direction. The NUV images are 1274 columns in the x direction by 1024 rows in the cross-dispersion direction for spectroscopic data, and 1024 x 1024 for data obtained in imaging mode. The NUV spectroscopic files have more pixels in the dispersion direction than the actual NUV detector. This is necessary because Doppler shifts and shifts due to OSM motions can cause wavelengths at one time during an exposure to fall outside of those obtained at another time during the exposure. As a result, the format has to be expanded to accommodate the shifts that occur during an exposure. The FUV images are not extended since the active area is less than the size of the detector, so these effects can be incorporated into the images without the need to extend them. The FUV data are also corrected for Y-walk and geometric distortions.
data a flat-fielded image is an intermediate calibrated data file. These files have a suffix flt
, and contain three extensions (SCI, ERR, and DQ) as shown in Figure 2.1
. These files are constructed by summing up the values in the EPSILON column for each pixel using the XFULL and YFULL coordinates. The data are in units of the count rate. For FUV data the images are 16384 x 1024, and, like the counts
images, the NUV images are 1274 x 1024 for spectroscopic data and 1024 x 1024 for data obtained in imaging mode. The flt
images are corrected for deadtime effects. The NUV images are corrected for all flat-field effects and the FUV data are currently only corrected for the XDL grid wire shadows.
The initial input files to calcos
are the association tables with suffix asn
. These files provide the calibration pipeline with information about how the data files are associated. In general, only exposures taken in sequence with the same spectral element, central wavelength (if applicable), and aperture at any FPPOS
will be associated. However, the user can create associations that include data taken with different central wavelengths, and calcos
version 2.18.5 and later will calibrate such associations. For more information on COS association files see the “Association Tables (ASN)”
portion of Section 2.4.4
Processing of each individual exposure in the association produces a final
calibrated result named with exposure rootname and suffix x1d
(spectroscopy) or flt
Next, for each FPPOS
position <n> (where <n>=1,2,3, or 4), if there are multiple spectroscopic
exposures in the association that use the same FPPOS
will combine them into a file named with the association rootname and suffix x1dsum
<n>, where <n> is the integer FPPOS
Lastly, a final association product file is produced with association rootname
and suffix x1dsum
(spectroscopy) or fltsum
(imaging) by combining all science exposures
in the association.
The COS pipeline produces extracted one-dimensional spectra and stores them in
binary tables with suffix x1d, x1dsum<n>
, or x1dsum
. Figure 2.6
shows the format of the 1-D extracted spectra table.
These COS extracted spectra tables can be 1 to 3-Dimensional, with one row for
each unique segment or stripe. For FUV data there are two rows which correspond to segments A and B distinguished by “FUVA” and “FUVB” in the SEGMENT column respectively. For NUV data there are three rows, “NUVA”, “NUVB” and “NUVC” corresponding to stripes A, B, and C respectively. Each table column can contain either a scalar value or an array of values, such as WAVELENGTH or FLUX. For example, NELEM will contain a scalar number, while the WAVELENGTH column will contain an array of wavelengths. Table 2.5
shows the contents of the different columns in an extracted spectrum table. A discussion of the data in COS extracted spectra is provided in Section 3.4.15
For NUV imaging observations, the flt
images are the final data products, with the latter being a simple sum of the individuals when several exposures are processed together. They are fully linearized and flat-field corrected images. Unlike the flt
files produced for the spectroscopic data (which are intermediate data products with a format of 1274 x 1024, see Section 2.4.2
), the formats of the flt
files for imaging data are 1024 x 1024, since Doppler and OSM motions are not applied.
An association file is created for all COS observation sets, and has the suffix asn
This file holds a single binary table extension, which can be displayed with the IRAF tasks tprint
calibrates raw data from multiple science exposures and any contemporaneously obtained line lamp calibration exposures through the pipeline as an associated unit. Each individual science exposure in an association
is fully calibrated in the process. See Chapter 5 of Introduction to HST Data Handbooks
for a general explanation of HST
data associations. The information within an association table shows how a set of exposures are related, and informs the COS calibration pipeline how to process the data.
An example association table is shown in Figure 2.7
. Note that all related COS exposures will be listed in an association table, with the exception of acquisitions, darks, and flats. It is possible to have an association which contains only one exposure. The association file lists the rootnames of the associated exposures as well as their membership role in the association. The exposures listed in an association table directly correspond to individual raw FITS files. For example, the association table can describe how wavecal exposures are linked to science exposures. Table 2.6
summarizes the different exposure membership types (MEMTYPES)
used for COS association tables.
illustrates the contents of the association table for a sequence of spectroscopic exposures for four FPPOS positions.
The association table above lists the names of the eight associated exposures (four
external and four calibration) that are calibrated and combined to create the various association products which will have a rootname of l9v221010.
This particular association is created from a single TIME-TAG
spectroscopic APT specification with FPPOS=ALL
specified in the Phase II file, which leads to both a science exposure and automatic wavecal exposure taken at each FPPOS
location. For example, the first entry in the table, l9v221euq,
is the rootname of a single external science exposure taken with FPPOS=1.
This exposure corresponds to the following rawtag
files: l9v221euq_rawtag_a.fits, l9v221euq_rawtag_b.fits.
The memtype of this exposure is EXP-FP
which shows that it is an external exposure. The second entry in the table has a memtype of EXP-AWAVE.
This denotes that the corresponding rawtag
exposures, l9v221ewq_rawtag_a.fits and l9v221ewq_rawtag_b.fits,
are wavecal exposures that will be used by the pipeline for wavelength calibration. Similar files correspond to the remaining three pairs of entries in the association file for data taken with the remaining three FPPOS
positions. The pipeline will calibrate the members of an association as a unit, producing the calibrated data products for each individual exposure as well as the final combined association data product. For this particular association, the pipeline will produce a final combined association product, l9v221010_x1dsum.fits,
which contains the final FPPOS
combined, calibrated spectrum.
When COS data are processed through OTFR in the HDA, the output messages
from generic conversion and the different calibration steps are stored in a FITS ASCII
table known as the trailer file, with suffix trl.
Each time the archive processes data before retrieval, the old trailer file is erased and a new one created using the results of the most recent processing performed. The archive will produce a trailer file for each individual exposure and association product. Association product trailer files contain the appended information from all the exposures in the association, in order of processing. The order of processing is the same as the order of exposures in the association table, with the exception of auto
or GO wavecals which are always processed first.
In the trailer files from the HDA, the output messages from generic conversion
appear first in the file. This section contains information relevant to the selection of the best reference files and the population of some of the header keywords. The second part of this file contains information from calcos
processing. Each task in the calcos
pipeline creates messages during processing which describe the progress of the calibration, and appear in the order in which each step was performed. These messages are quite relevant to understanding how the data were calibrated, and in some of the cases, to determining the accuracy of the products.
In this last section of the _trl
file, the calcos
steps are indicated by their module name. The calcos
messages provide information on the input and output files for each step, the corrections performed, information regarding the reference files used, and in the case of FUV data, messages about the location of the stims, or shift correction applied to the data. Calcos
also gives warnings when the appropriate correction to the data could not be applied. For more detailed information on the calibration steps and structure of calcos
, please refer to Chapter 3
is run in a user’s home environment, calcos
redirects the output of its steps to the STDOUT and an ASCII
file with name rootname.tra
. Note, the level of detail included in the output messages can be modified when running calcos
(see “Run Calcos”
). So, when run on a personal machine, calcos
overwrite the trl
file but rather will direct the output to STDOUT and an ASCII tra
file. The tra
file is formatted like the trl
file but with two exceptions: the tra
file will not contain the output messages from generic conversion, and the tra
file is not converted to FITS format. Therefore, one must look at both the trailer (generic conversion messages) and tra
(calibration messages) file when running calcos
from a home environment. Each time calcos
is run on a file, the STDOUT messages will be appended to the tra
file if it already exists. Also, when running calcos
on a personal machine there will be no tra
created for the association products. Instead, the calcos
messages for association products will be sent only to STDOUT.
The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. A COS support file contains a primary header and at least three FITS image extensions. The first extension contains a header with the proposal information and an (16-bit) image array containing the data which populate the spt
header keyword values. The image array element values are used by conversion software to populate the header keywords. Following the support extension, the COS spt
files contain two engineering snapshot extensions. These extensions contain a readout of several instrument and telescope parameters from telemetry data at different times during the course of an exposure. The very first snapshot extension will always contain telemetry information from the beginning of an exposure. Depending on the length of the exposure, the support file may also contain one or several “imsets” which include a support extension and two snap extensions. These intermediate imsets will have only their second snapshot extension populated with telemetry data taken during the course of an exposure, while the first snapshot will be populated with default values. The very last imset of an spt
file will have all three extensions (1 support and 2 snaps) populated with telemetry values at the completion of the exposure. Figure 2.8
depicts the structure of an N extension COS support file.
With several snapshots of COS telemetry values, one may track the instrument
status periodically throughout an exposure. For a schematic listing of the spt headers with detailed information about the spt
header keywords, See:
COS support file with N extensions. The initial imset contains telemetry values at the
start of the exposure. Extensions 3 through (N-3) contain imsets with telemetry values at intermediate times during the exposure. Note that the first snap extensions in these intermediate imsets are NOT populated. The final imset includes extensions N-2 through N and contains telemetry values at the end of the exposure. Both snap extensions are populated for the final imset.
All COS acquisition exposures will produce a single raw data file with suffix rawacq
. Almost all COS spectroscopic science exposures are preceded by an acquisition sequence or exposure to center the target in the aperture. Keywords in the header of COS science data identify the exposure names of relevant acquisition exposures in each visit. In addition, there are several other useful keywords in the COS acquisition exposures that describe the acquisition parameters used, as well as the calculated centroid positions and slew offsets. Table 2.7 lists all the relevant acquisition keywords.
Acquisition peakups in the dispersion direction (ACQ/PEAKD
) and acquisition spiral searches (ACQ/SEARCH
) both use the flux from exposures taken at different dwell points to center the target. For more information on these types of COS acquisitions see Sections 7.6.4 and 7.6.2 respectively of the COS Instrument Handbook
. Data for these acquisitions contain one binary table extension which describes the acquisition search pattern dwell point locations and counts as shown in Table 2.11
and Figure 2.9
Acquisition peakups in the cross-dispersion direction (ACQ/PEAKXD
) use a TIME-TAG
spectrum to center the target in the cross-dispersion direction. For more information on the ACQ/PEAKXD
algorithm see Section 7.6.3 of the COS Instrument Handbook
. An ACQ/PEAKXD
exposure includes only a primary header and extension header. There are no data downlinked for this type of acquisition.
Acquisition images (ACQ/IMAGE
) use a NUV image to center the target in the aperture. For more information on the ACQ/IMAGE
algorithm see Section 7.5 of the COS Instrument Handbook
. An ACQ/IMAGE
exposure produces a raw data file containing two science image extensions corresponding to the initial and final pointing:
See Figure 2.10
for the FITS file format for ACQ/IMAGE
The COS jitter files include engineering data that describe the performance of the
Pointing Control System (PCS) including the Fine Guidance Sensors that are used to control the vehicle pointing. The jitter files report on PCS engineering data during the duration of the observation. The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. COS jitter files utilize the file format shown in Figure 2.11
for all science observations, excluding acquisitions.
The COS jif files are a 2-D histogram of the corresponding jit
file (See Section , “Jitter Files (jit)”
) and have the file format shown in Figure 2.12
for all science observations excluding acquisitions.