The following sections discuss the COS raw science data files, intermediate calibration products, final calibration products, and auxiliary data files. Uncalibrated science data include all raw science data generated during Generic Conversion that have not been processed through the calibration pipeline. These raw files are the input files to the
calcos pipeline, usually as part of an association (see
“Association Tables (ASN)”). The result of the pipeline is both individual calibrated exposure files and, when appropriate, a final combined product file.
For ACCUM data, the raw files contain a set of images, as shown in
Figure 2.1, and have filenames with the suffix
rawaccum for NUV data, or
rawaccum_a and
rawaccum_b for the two segments of the FUV detector. The SCI extension contains an image of the total accumulated counts during an exposure. For NUV data the ERR and DQ extensions have only a header with no data. For FUV data the ERR extension has only a header with no data, and the DQ extension is populated with data quality information only for pixels that are outside the subarray boundaries (defined below). The DQ extensions will be populated in the
flt files, after calibration pipeline processing. Even though FUV
rawaccum_a[b] data are 16384 x 1024 images, only portions of them contain actual data. These portions are called subarrays. Typically, three subarrays are used for each segment of an FUV
ACCUM image. Two are centered on the STIM positions and the third is a stripe 128 pixels wide which is centered on the wavecal spectrum of the object.
Figure 2.2 shows these spectral region subarrays superimposed on two FUV
rawtag images. As
Figure 2.2 shows, the spectrum falls outside of the subarray. Consequently, wavecals must be taken separately for
ACCUM data.
Raw TIME-TAG Events Lists (
rawtag)
Raw events tables contain the locations and arrival times of individual photon events collected in
TIME-TAG mode. These files have the suffix
rawtag for NUV or
rawtag_a[b] for the two FUV segments.
Figure 2.3 shows the format of a
rawtag table. The first extension contains the events list, in which each row of the table corresponds to a single event in the data stream and the columns of the table contain scalar quantities that describe the event
. The second extension contains the good time intervals (GTI) table, where an uninterrupted period of time is considered as one good time interval. Interruptions in the data taking due to memory overflow could result in more than one GTI.
Table 2.2 shows the columns of a
rawtag table.
For FUV ACCUM data only, a 7 bit pulse height amplitude histogram is accumulated in the detector electronics onboard. This information is placed in a file with the suffix
pha. The pulse-height histogram files contain a primary header with no data and a single FITS image SCI extension containing a histogram of the pulse-height distribution during the exposure. The pulse height amplitude files do not contain an ERR or DQ extension, as shown in
Figure 2.4. The pulse height distribution is an image array of length 128, corresponding to the number of photons with values from 0 to 127, corresponding to the pulse heights of 0-31 available in
TIME-TAG data.
The COS pipeline produces corrected TIME-TAG events lists and stores them in binary tables with suffix
corrtag. These files have a main header and three extensions: a corrected events list extension, a good time interval extension, and a time line table extension, with a format similar to the one shown in
Figure 2.3. The first extension of the corrtag file is the events table (see
Table 2.3) which includes
X and
Y event locations that have been corrected for distortion, doppler shift, and offsets due to OSM motions in both the dispersion and cross-dispersion directions. It also includes wavelengths associated with events that occur within the active area of the detectors and a data quality (DQ) flag for each event (see,
Table 2.19). The second extension gives the start and stop times of the good time intervals (as in the
rawtag file), and the third extension is the time line table. The time line table includes second by second values for spacecraft position, solar and target altitude above the horizon, and count rates for the most prominent airglow lines and the background. These observed rates might include counts from other external sources in addition to the ones from the airglow line itself. The data in this extension can be useful for reprocessing
TIME-TAG data to exclude, for example, daytime data using the Python tool timefilter, described in
Chapter 5, which is also available as an IRAF task.
For ACCUM data, the
corrtag files are somewhat different. All of the time stamps in the first extension are set to the median value of the observation. Each count in the
rawaccum file becomes an event so, for example, a pixel in the
rawccum that that had 100 counts, would have 100 entries in the
corrtag file. The
RAWX,
XCORR and
XDOPP entries are all the same for NUV data, but can be different for FUV. In addition,
RAWY and
YCORR entries will have the same values. However,
XFULL and
YFULL can be different. In the timeline extension, the
SHIFT1, airglow and
DARKRATE entries are fixed, but all others are time dependent.
For TAGFLASH data, calcos produces an events list with suffix lampflash, that contains the extracted wavecal lamp flashes. Each row in the events list corresponds to a different segment or stripe and flash number (the first flash is number 1, the second is number 2, etc.). The lampflash files have the format shown in Figure 2.5. The contents of the columns in a lampflash events list are listed in Table 2.4. Columns TIME, LAMP_ON, and LAMP_OFF have the same temporal zero point as the TIME column of the rawtag and corrtag tables and the same unit (seconds). The shifts contained in the SHIFT_DISP and SHIFT_XDISP columns of the lampflash table are applied to the XDOPP and YCORR columns of the corrtag file to produce the X[Y]FULL entries. When multiple TAGFLASHES are present, the shifts are interpolated in time for events occurring between the first and last flashes. Events occurring before the first flash are shifted by a value extrapolated using the slope defined by the first two flashes; events beyond the last flash are given the shift determined by the last flash. As a result, the difference between the X[Y]FULL and X[Y]CORR entries in the corrtag file can be a function of time.
The counts images are an intermediate calibrated output product for both imaging and spectroscopic data with suffix
counts. These files contain three extensions (SCI, ERR, and DQ) as shown in
Figure 2.1. These files are constructed by summing up the events from each pixel using the XFULL and YFULL coordinates. The data are in units of counts per pixel. For FUV data the images are 16384 columns in the x (dispersion) direction by 1024 rows in the y (cross-dispersion) direction. The NUV images are 1274 columns in the x direction by 1024 rows in the cross-dispersion direction for spectroscopic data, and 1024 x 1024 for data obtained in imaging mode. The NUV spectroscopic files have more pixels in the dispersion direction than the actual NUV detector. This is necessary because Doppler shifts and shifts due to OSM motions can cause wavelengths at one time during an exposure to fall outside of those obtained at another time during the exposure. As a result, the format has to be expanded to accommodate the shifts that occur during an exposure. The FUV images are not extended since the active area is less than the size of the detector, so these effects can be incorporated into the images without the need to extend them. The FUV data are also corrected for Y-walk and geometric distortions.
For spectroscopic data a flat-fielded image is an intermediate calibrated data file. These files have a suffix
flt, and contain three extensions (SCI, ERR, and DQ) as shown in
Figure 2.1. These files are constructed by summing up the values in the EPSILON column for each pixel using the XFULL and YFULL coordinates. The data are in units of the count rate. For FUV data the images are 16384 x 1024, and, like the
counts images, the NUV images are 1274 x 1024 for spectroscopic data and 1024 x 1024 for data obtained in imaging mode. The
flt images are corrected for deadtime effects. The NUV images are corrected for all flat-field effects and the FUV data are currently only corrected for the XDL grid wire shadows.
The initial input files to calcos are the association tables with suffix
asn. These files provide the calibration pipeline with information about how the data files are associated. In general, only exposures taken in sequence with the same spectral element, central wavelength (if applicable), and aperture at any
FPPOS will be associated. However, the user can create associations that include data taken with different central wavelengths, and
calcos version 2.18.5 and later will calibrate such associations. For more information on COS association files see the
“Association Tables (ASN)” portion of
Section 2.4.4.
Processing of each individual exposure in the association produces a final calibrated result named with exposure rootname and suffix
x1d (spectroscopy) or
flt (imaging).
Next, for each FPPOS position <n> (where <n>=1,2,3, or 4), if there are
multiple spectroscopic exposures in the association that use the same
FPPOS position,
calcos will combine them into a file named with the association rootname and suffix
x1dsum<n>, where <n> is the integer
FPPOS value.
Lastly, a final association product file is produced with association rootname and suffix
x1dsum (spectroscopy) or
fltsum (imaging) by combining
all science exposures in the association.
The COS pipeline produces extracted one-dimensional spectra and stores them in binary tables with suffix
x1d, x1dsum<n>, or
x1dsum.
Figure 2.6 shows the format of the 1-D extracted spectra table.
These COS extracted spectra tables can be 1 to 3-Dimensional, with one row for each unique segment or stripe. For FUV data there are two rows which correspond to segments A and B distinguished by “FUVA” and “FUVB” in the SEGMENT column respectively. For NUV data there are three rows, “NUVA”, “NUVB” and “NUVC” corresponding to stripes A, B, and C respectively. Each table column can contain either a scalar value or an array of values, such as WAVELENGTH or FLUX. For example, NELEM will contain a scalar number, while the WAVELENGTH column will contain an array of wavelengths.
Table 2.5 shows the contents of the different columns in an extracted spectrum table. A discussion of the data in COS extracted spectra is provided in
Section 3.4.15.
For NUV imaging observations, the flt and
fltsum images are the final data products, with the latter being a simple sum of the individuals when several exposures are processed together. They are fully linearized and flat-field corrected images. Unlike the
flt files produced for the spectroscopic data (which are intermediate data products with a format of 1274 x 1024, see
Section 2.4.2), the formats of the
flt and
fltsum files for imaging data are 1024 x 1024, since Doppler and OSM motions are not applied.
An association file is created for all COS observation sets, and has the suffix asn (e.g.,
l9v221010_asn.fits). This file holds a single binary table extension, which can be displayed with the IRAF tasks
tprint or
tread.
Calcos calibrates raw data from multiple science exposures and any contemporaneously obtained line lamp calibration exposures through the pipeline as an associated unit. Each individual science exposure in an
association is fully calibrated in the process. See Chapter 5 of
Introduction to HST Data Handbooks for a general explanation of
HST data associations. The information within an association table shows how a set of exposures are related, and informs the COS calibration pipeline how to process the data.
An example association table is shown in Figure 2.7. Note that all related COS exposures will be listed in an association table, with the exception of acquisitions, darks, and flats. It is possible to have an association which contains only one exposure. The association file lists the rootnames of the associated exposures as well as their membership role in the association. The exposures listed in an association table directly correspond to individual raw FITS files. For example, the association table can describe how wavecal exposures are linked to science exposures.
Table 2.6 summarizes the different exposure membership types
(MEMTYPES) used for COS association tables.
Figure 2.7 illustrates the contents of the association table for a sequence of spectroscopic exposures for four FPPOS positions.
The association table above lists the names of the eight associated exposures (four external and four calibration) that are calibrated and combined to create the various association products which will have a rootname of
l9v221010. This particular association is created from a single
TIME-TAG spectroscopic APT specification with
FPPOS=ALL and
FLASH=NO specified in the Phase II file, which leads to both a science exposure and automatic wavecal exposure taken at each
FPPOS location. For example, the first entry in the table,
l9v221euq, is the rootname of a single external science exposure taken with
FPPOS=1. This exposure corresponds to the following
rawtag files:
l9v221euq_rawtag_a.fits, l9v221euq_rawtag_b.fits. The memtype of this exposure is
EXP-FP which shows that it is an external exposure. The second entry in the table has a memtype of
EXP-AWAVE. This denotes that the corresponding
rawtag exposures,
l9v221ewq_rawtag_a.fits and l9v221ewq_rawtag_b.fits, are wavecal exposures that will be used by the pipeline for wavelength calibration. Similar files correspond to the remaining three pairs of entries in the association file for data taken with the remaining three
FPPOS positions. The pipeline will calibrate the members of an association as a unit, producing the calibrated data products for each individual exposure as well as the final combined association data product. For this particular association, the pipeline will produce a final combined association product,
l9v221010_x1dsum.fits, which contains the final
FPPOS combined, calibrated spectrum.
When COS data are processed through OTFR in the HDA, the output messages from generic conversion and the different calibration steps are stored in a FITS
ASCII table known as the trailer file, with suffix
trl. Each time the archive processes data before retrieval, the old trailer file is erased and a new one created using the results of the most recent processing performed. The archive will produce a trailer file for each individual exposure and association product. Association product trailer files contain the appended information from all the exposures in the association, in order of processing. The order of processing is the same as the order of exposures in the association table, with the exception of
auto or GO wavecals which are always processed first.
In the trailer files from the HDA, the output messages from generic conversion appear first in the file. This section contains information relevant to the selection of the best reference files and the population of some of the header keywords. The second part of this file contains information from
calcos processing. Each task in the
calcos pipeline creates messages during processing which describe the progress of the calibration, and appear in the order in which each step was performed. These messages are quite relevant to understanding how the data were calibrated, and in some of the cases, to determining the accuracy of the products.
In this last section of the _trl file, the
calcos steps are indicated by their module name. The
calcos messages provide information on the input and output files for each step, the corrections performed, information regarding the reference files used, and in the case of FUV data, messages about the location of the stims, or shift correction applied to the data.
Calcos also gives warnings when the appropriate correction to the data could not be applied. For more detailed information on the calibration steps and structure of
calcos, please refer to
Chapter 3.
When calcos is run in a user’s home environment,
calcos redirects the output of its steps to the STDOUT and an
ASCII file with name
rootname.tra. Note, the level of detail included in the output messages can be modified when running
calcos (see
“Run Calcos” ). So, when run on a personal machine,
calcos will
not overwrite the
trl file but rather will direct the output to STDOUT and an
ASCII tra file. The
tra file is formatted like the
trl file but with two exceptions: the
tra file will not contain the output messages from generic conversion, and the
tra file is not converted to FITS format. Therefore, one must look at both the trailer (generic conversion messages) and
tra (calibration messages) file when running
calcos from a home environment. Each time
calcos is run on a file, the STDOUT messages will be appended to the
tra file if it already exists. Also, when running
calcos on a personal machine there will be no
tra created for the association products. Instead, the
calcos messages for association products will be sent only to STDOUT.
The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. A COS support file contains a primary header and at least three FITS image extensions. The first extension contains a header with the proposal information and an (16-bit) image array containing the data which populate the
spt header keyword values. The image array element values are used by conversion software to populate the header keywords. Following the support extension, the COS
spt files contain two engineering snapshot extensions. These extensions contain a readout of several instrument and telescope parameters from telemetry data at different times during the course of an exposure. The very first snapshot extension will always contain telemetry information from the beginning of an exposure. Depending on the length of the exposure, the support file may also contain one or several “imsets” which include a support extension and two snap extensions. These intermediate imsets will have only their second snapshot extension populated with telemetry data taken during the course of an exposure, while the first snapshot will be populated with default values. The very last imset of an
spt file will have all three extensions (1 support and 2 snaps) populated with telemetry values at the completion of the exposure.
Figure 2.8 depicts the structure of an N extension COS support file.
With several snapshots of COS telemetry values, one may track the instrument status periodically throughout an exposure. For a schematic listing of the spt headers with detailed information about the
spt header keywords, See:
All COS acquisition exposures will produce a single raw data file with suffix rawacq. Almost all COS spectroscopic science exposures are preceded by an acquisition sequence or exposure to center the target in the aperture. Keywords in the header of COS science data identify the exposure names of relevant acquisition exposures in each visit. In addition,
there are several other useful keywords in the COS acquisition exposures that describe the acquisition parameters used, as well as the calculated centroid positions and slew offsets. Table 2.7 lists all the relevant acquisition keywords.
Acquisition peakups in the dispersion direction (ACQ/PEAKD) and acquisition spiral searches (
ACQ/SEARCH) both use the flux from exposures taken at different dwell points to center the target. For more information on these types of COS acquisitions see Sections 7.6.4 and 7.6.2 respectively of the
COS Instrument Handbook. Data for these acquisitions contain one binary table extension which describes the acquisition search pattern dwell point locations and counts as shown in
Table 2.11 and
Figure 2.9
Acquisition peakups in the cross-dispersion direction (ACQ/PEAKXD) use a
TIME-TAG spectrum to center the target in the cross-dispersion direction. For more information on the
ACQ/PEAKXD algorithm see Section 7.6.3 of the
COS Instrument Handbook. An
ACQ/PEAKXD exposure includes only a primary header and extension header. There are no data downlinked for this type of acquisition.
Acquisition images (ACQ/IMAGE) use a NUV image to center the target in the aperture. For more information on the
ACQ/IMAGE algorithm see Section 7.5 of the
COS Instrument Handbook. An
ACQ/IMAGE exposure produces a raw data file containing two science image extensions corresponding to the initial and final pointing:
See Figure 2.10 for the FITS file format for
ACQ/IMAGE data.
The COS jitter files include engineering data that describe the performance of the Pointing Control System (PCS) including the Fine Guidance Sensors that are used to control the vehicle pointing. The jitter files report on PCS engineering data during the duration of the observation. The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. COS jitter files utilize the file format shown in
Figure 2.11 for all science observations, excluding acquisitions.
The COS jif files are a 2-D histogram of the corresponding
jit file (See
Section , “Jitter Files (jit)”) and have the file format shown in
Figure 2.12 for all science observations excluding acquisitions.
The COS pipeline produces corrected TIME-TAG events lists and stores them in binary tables with suffix corrtag. These files have a main header and three extensions: a corrected events list extension, a good time interval extension, and a time line table extension, with a format similar to the one shown in Figure 2.3. The first extension of the corrtag file is the events table (see Table 2.3) which includes X and Y event locations that have been corrected for distortion, doppler shift, and offsets due to OSM motions in both the dispersion and cross-dispersion directions. It also includes wavelengths associated with events that occur within the active area of the detectors and a data quality (DQ) flag for each event (see, Table 2.19). The second extension gives the start and stop times of the good time intervals (as in the rawtag file), and the third extension is the time line table. The time line table includes second by second values for spacecraft position, solar and target altitude above the horizon, and count rates for the most prominent airglow lines and the background. These observed rates might include counts from other external sources in addition to the ones from the airglow line itself. The data in this extension can be useful for reprocessing TIME-TAG data to exclude, for example, daytime data using the Python tool timefilter, described in Chapter 5, which is also available as an IRAF task.For ACCUM data, the corrtag files are somewhat different. All of the time stamps in the first extension are set to the median value of the observation. Each count in the rawaccum file becomes an event so, for example, a pixel in the rawccum that that had 100 counts, would have 100 entries in the corrtag file. The RAWX, XCORR and XDOPP entries are all the same for NUV data, but can be different for FUV. In addition, RAWY and YCORR entries will have the same values. However, XFULL and YFULL can be different. In the timeline extension, the SHIFT1, airglow and DARKRATE entries are fixed, but all others are time dependent.
COS support file with N extensions. The initial imset contains telemetry values at the start of the exposure. Extensions 3 through (N-3) contain imsets with telemetry values at intermediate times during the exposure. Note that the first snap extensions in these intermediate imsets are NOT populated. The final imset includes extensions N-2 through N and contains telemetry values at the end of the exposure. Both snap extensions are populated for the final imset.All COS acquisition exposures will produce a single raw data file with suffix rawacq. Almost all COS spectroscopic science exposures are preceded by an acquisition sequence or exposure to center the target in the aperture. Keywords in the header of COS science data identify the exposure names of relevant acquisition exposures in each visit. In addition, there are several other useful keywords in the COS acquisition exposures that describe the acquisition parameters used, as well as the calculated centroid positions and slew offsets. Table 2.7 lists all the relevant acquisition keywords.Table 2.7: ACQ/IMAGE Header Keywords.
The COS jitter files include engineering data that describe the performance of the Pointing Control System (PCS) including the Fine Guidance Sensors that are used to control the vehicle pointing. The jitter files report on PCS engineering data during the duration of the observation. The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. COS jitter files utilize the file format shown in Figure 2.11 for all science observations, excluding acquisitions.The jitter tables contain PCS data for each three second interval during the observation, as listed in Table . For more information on jitter files refer to Chapter 6 of Introduction to HST Data Handbooks.
