[Top] [Prev] [Next] [Bottom]

30.1 Contents of Delivered Data

Unless you have read your FOS data from very old data tapes, the data you have on your machine will be in FITS format. These files need to be converted to GEIS format with the STSDAS routine strfits, as is described in Chapter 2 (Volume 1). The resulting GEIS format files will have default suffixes, as described in Table 30.1.

You will be most interested in the contents of the .c1*, .c0*, and .c2* files as these contain the flux-calibrated data, wavelengths and errors for an exposure. The easiest way to get a quick glance at your spectra is by using the routine splot or fwplot if you want to plot wavelength vs. flux (see "FOS and GHRS Spectra" on page 3-17). We recommend that you generate the FOS paper products for your data with the STSDAS routine pp_dads and obtain a variety of displays and tables which, as we shall see in "FOS Paper Products" on page 30-15, can facilitate the assessment of your data quality.



FOS File Name Suffixes

Suffix

File Contents

Suffix

File Contents

Raw Data Files (input for calfos)

Calibrated Data Files (output from calfos)

.d0h/.d0d

Science data image

.c0h/.c0d

Calibrated wavelengths

.q0h/.q0d

Science data quality

.c1h/.c1d

Calibrated fluxes

.shh/.shd

Standard header packet

.c2h/.c2d

Propagated statistical error

.d1h/.d1d

Science trailer line

.c3h/.c3d

Special statistics

.ulh/.uld

Unique data log

.c4h/.c4d

Count rate

.x0h/.x0d

Science header line

.c5h/.c5d

Flat-fielded object spectra

.xqh/.xqd

Science header line data quality

.c6h/.c6d

Flat-fielded sky spectra

.q1h / .q1d

Science trailer line data quality

.c7h/.c7d

Background spectra

.c8h/.c8d

Flat-fielded object minus smoothed sky spectra

.cqh/.cqd

Calibrated science data quality

Additional Files

.jih/.jit

Jitter files (.cmh, .cmj, .cmi)

.trl

Trailer file

.pdq/.ocx

PODPS/OPUS data quality files

30.1.1 Uncalibrated (Raw) Data Files

Table 30.1 lists the science files that are used as required input to calfos. These files are described briefly below.

Science Data Files (.d0h/.d0d)

Science data files contain single-precision floating point values that represent the number of detected counts accumulated in each pixel. The number of data elements in the one-dimensional science data array depends on the observation mode; specifically, the number of diodes, the number of substeps, the number of Y steps, and the number of repeats (sometimes called slices or bins) used in the observation. The maximum number of data elements is 12288. The associated header file also provides information on the different steps to be performed during pipeline calibration processing, and the reference files and tables to be used in the calibration.

Science Header Line (.x0h/.x0d)

The science header line (SHL) file is a one dimensional array with a length equal to a line of the science data. It contains a partial copy of the unique data log.

Science Trailer Line (.d1h/.d1d)

The science trailer line (STL) file is also a one dimensional array containing the number of measurements rejected from the various combinations of x substeps, y steps, repeats, etc. The information in these files is used to compute the total effective exposure time per pixel which is later used to convert the counts into count rates.

Data Quality Files (.q1h/.q1d)

The science data files, science header line files, and the science trailer files have corresponding data quality files that contain the flags for bad or suspect data. These raw data quality files have quality flags as follows:

Standard Header Packet (.shh/.shd)

The standard header packet (SHP) contains the telemetry values for engineering data and some FOS-unique data. The engineering data include temperatures, currents, and voltages at various points in the instrument. The FOS-unique data varies depending on the onboard processing used for a given observation. The header packet also contains information used in the operation of the spacecraft, such as target name, position and velocity of the telescope, the right ascension and declination of the target, the sun, and the moon, and other proposal information used in the observation which was provided in Phase II of the proposal process.

Unique Data Log (.ulh/.uld)

The unique data log (UDL) contains the mechanism control blocks used to control the entrance aperture, entrance port, polarizer, and filter grating wheel assembly. This file also contains the discriminator level, disabled diode table, serial engineering data, instrument configuration, and exposure parameters.

Trailer File (.trl)

The trailer file contains many messages generated by the so-called "generic conversion" of the data from what was onboard the spacecraft into STSDAS images. These messages may contain information on missing or filled data packets. Informational messages produced by calfos during calibration are also stored in the trailer file. The trailer file is identified by the suffix .trl.

30.1.2 Calibrated Data Files

Several types of calibrated output files are produced by calfos. These are listed in Table 30.1. More extensive descriptions of each type of file are provided below.

Calibrated Wavelength Files (.c0h/.c0d)

These files contain single-precision floating point calibrated vacuum wavelengths corresponding to the center of each pixel of the science data. All FOS wavelength solutions assume the first pixel is pixel 0.

Calibrated Flux Files (.c1h/.c1d)

These files contain single-precision floating point calibrated fluxes (in ergs sec-1 cm-2 Å-1) corresponding to each pixel of the science data.

Calibrated Statistical Error Files (.c2h/.c2d)

These files contain the statistical errors of the original data. These files are calibrated in lock-step with the science data files. Errors caused by sky and background subtraction, flatfields, and sensitivity are not included in the error estimates.

Calibrated Special Mode Data Files (.c3h/.c3d)

Data acquired in the rapid-readout, time-resolved, or spectropolarimetry modes require processing steps in addition to (or complementing) those used for standard ACCUM data. The calibrated data are then stored in special mode data files. For RAPID mode, the files contain the total flux, integrated over all pixels, and the associated statistical error for each readout. For TIME RESOLVED mode, the files contain the pixel-by-pixel average of all slices or bins, the difference between each slice or bin and the average, and the average propagated statistical errors. For POLARIMETRY mode, the file contains the Stokes I, Q, U, and V parameters, the linear and circular polarization, and the polarization position angle. The polarimetric quantities and the propagated errors are calculated for each of the separate pass directions, the combined pass direction data, and the combined pass direction corrected for interference and instrumental orientation.

Calibrated Data Quality Files (.cqh/.cqd)

The quality flags in these files flag the bad pixel values in the calibrated files. The quality flags from the raw data are updated and additional flags are added for problems detected in the calibration process. The data quality flags are defined in Table 30.2.


FOS Data Quality Flag Values

Flag

Value

Description

Category 1: Data not useful. Data values set to zero.

800

Data filled

700

Data filled due to GIM correction

400

Disabled channel

300

Severe saturation (uncertainty greater than 50%)

200

Inverse sensitivity invalid ( < 1100 Å or > 7000 Å)

Category 2: Data Uncertain. Uncertainty not Indicated in Error Calc.

190

Large saturation correction (uncertainty greater than 20%)

170

Intermittent noisy channel

160

Intermittent dead channel

130

Moderate saturation correction (uncertainty greater than 5%)

120

Sky or background fixed or extrapolated

100

Reed-Solomon decoding error

Category 3: Data uncertain. Uncertainty in propagated error file

50

Sampling less than 50% of nominal

Intermediate Calibrated Output Data Files (.c4* - c8*)

At most, six sets of intermediate calibrated output files are produced depending on the observation mode.

Trailer File (.trl)

We list the trailer file again as any log comments written by the pipeline calfos procedures are appended to spacecraft and generic conversion information already in the file. The trailer file is identified by the suffix .trl.

30.1.3 Paired-Aperture File Structures

The FOS paired apertures were sampled in two different ways which lead to very different internal file structures and data products:

30.1.4 Other Important Observation-related Files

Observation Log Files (.jih/.jit)

As of February 1995, the observation log (or jitter) files (.jih, .jit ,etc), which are provided by the Observatory Monitoring System (OMS), became routinely available (some files from the October 1994 through February 1995 time period are also available). These can give useful information, such as telescope performance during the observation and the position of the target within the aperture. A so-called jitter ball is a routine part of all FOS paper products for which jitter files exist (see Figure 30.8). This plot shows (V2,V3) motions of the FGSs needed to keep the guide stars centered, recorded every six seconds throughout the exposure. See Appendix C for a description of the observation logs and jitter files.

PODPS Data Quality File (.pdq)

STScI staff performed a quick Data Quality Assessment (DQA) of the target acquisitions and the science observations. This assessment identified any target acquisition failures, guide star acquisition failures, other spacecraft anomalies, or instrumental problems; the assessment was recorded in the data quality keywords of the PDQ (Procedural Data Quality) file. PDQ files are archived to data class PDQ and can be retrieved from the HST Archive using StarView. The FOS paper products list each relevant PDQ file comment for every exposure.

OCX files (Observatory Support System (OSS) Observer Comment files) contain information about the success or failure of target acquisitions and science observations. Before April 17, 1992, OCX files were not always archived separately, and, in some cases, were prepended to the RSDP pipeline history file, the .trl trailer file. After February 1995, OCX files were produced only to document real-time activity (e.g., interactive target acquisition) in support of an observation. After October 1996, OCX files may contain information about observation execution problems.

The PDQ files and the comments they contain should not be over-interpreted. The remarks were based upon a visual examination of the data and the experience of the OPUS staff. Technical remarks are usually sound, but some comments about the "quality" of the observations can arise from an incomplete knowledge of the science goals of the original proposer. Unless the object was significantly miscentered, for example, the achieved signal-to-noise is probably close to what was expected, even if it may appear low in a single exposure. To assess whether or not a low or varying flux is meaningful, you should examine the jitter file or other records of spacecraft performance. The lack of a comment is also not a guarantee that there were not some problems with the data.

30.1.5 Reference Files and Tables

The calibration pipeline used reference files and tables to flux calibrate FOS data. Data in the archive and the data that were sent to GOs were calibrated with the reference files that were available at the time of the observation. These are generally not the "best" files to use to calibrate these data today. StarView allows you to retrieve either the "used" or the "best" reference files. The "best" reference files reflect our accumulated experience of FOS calibrations and derive from the late 1997 FOS closeout calibration analyses.

Naming and Structure of FOS Reference Files and Tables

The reference files and tables are typically referred to by the name of the Calibration Data Base System (CDBS) reference relation that holds their names. The suffixes of the reference tables and files are of the form .cyn, .rnh and .rnd where n represents a value from 0 to 9 and A to D (see Table 30.3). These files are maintained in the CDBS. The STScI web site also includes a catalog of these tables and files.

A thorough description of the internal header and data formats of FOS reference files and tables is beyond the scope of this document, but all this information can be found in rigorous detail in STScI document ICD-47 which is available from the FOS WWW page and in paper format from help@stsci.edu. A description of what each FOS reference file and table is used for is found in "Details of the FOS Pipeline Process" on page 31-13.

Except for some spectropolarimetric reference files, which are twice this length (for two pass directions), all reference files contain a vector of length:

Where



[Top] [Prev] [Next] [Bottom]

stevens@stsci.edu
Copyright © 1997, Association of Universities for Research in Astronomy. All rights reserved. Last updated: 01/14/98 14:29:35