Space Telescope Science Institute
Intro to HST Data Handbooks 8.0 May 2011
help@stsci.edu
Table of Contents Previous Next Index Print


Introduction to the HST Data Handbooks > Chapter 2: HST File Formats > 2.2 Multi-Extension FITS File Format

2.2
Flexible Image Transport System (FITS)1 is a standard format for exchanging astronomical data, independent of the hardware platform and software environment.
A file in FITS format consists of a series of Header Data Units (HDUs), each containing two components: an ASCII text header and the binary data. The header contains a series of keywords that describe the data in a particular HDU and the data component may immediately follow the header.
For HST FITS files, the first HDU, or primary header, contains no data. The primary header may be followed by one or more HDUs called extensions. Extensions may take the form of images, binary tables, or ASCII text tables. The data type for each extension is recorded in the XTENSION header keyword. Figure 2.1 schematically illustrates the structure of a FITS file and its extensions.
Figure 2.1: FITS File Structure
Each FITS extension header contains the required keyword XTENSION, which specifies the extension type and has one of the following values: IMAGE, BINTABLE, and TABLE, corresponding to an image, binary table, and ASCII table, respectively.
Table 2.2: HST Header Keyword Descriptions
SAMP1 (number of sample)
TIME1 (exposure time)
EVENTS2 (photon event list)
GTI2 (good time interval)
Allowed for any extension except SCI, and used for an image with uniform value for all pixels

1
Only found in NICMOS and WFC3 data.

2
Only found in COS and STIS data.

3
When an image has the same value at all pixels (e.g., data quality value), the extension has no data component. Instead, the constant pixel value is stored in the header keyword PIXVALUE.

A set of FITS extension images which are logically related to one another is called an imset. For example, the error image and the data quality image are in the same imset as the science image itself. The keyword EXTNAME is used to specify the extension names of different images in the same imset.
2.2.1
The FITS image kernel included in IRAF version 2.12.2 and higher treats each extension like a standard IRAF image. This subsection provides an overview of Multi-Extension FITS file syntax and keyword options needed to extract specific types of data. The following discussion describes how to specify the image extensions in FITS files that you would like to process with IRAF/STSDAS tasks and presumes that you are using IRAF 2.12.2 or higher. It covers how to:
 
Retaining the .fits suffix at the end of every FITS file name in your file specifications will ensure that IRAF both reads and writes these images in FITS format.
 
In order to work with ACS, NICMOS, or STIS data, users will need to upgrade to at least IRAF 2.12.2 and STSDAS 3.2. Processing of WFC3 data will require instead STSDAS 3.8 or higher. COS will require STSDAS 3.9 or later and its pipeline runs only under PyRAF (Section 3.2).
Generating a FITS File Listing
Once you have downloaded any FITS files from the HDA, you may want an inventory of their contents. To generate a listing of a FITS file’s extensions, you can use the catfits task in the tables package. For example, Table 2.3 illustrates the first 11 lines generated by catfits from a NICMOS MultiAccum FITS file containing only images. The output columns from catfits contain the following information:
Several STSDAS tasks can work with entire imsets (see Section 3.4.3), but most operate on individual images. See the “Data Structure” chapters in Part II of this Data Handbook for more information on the contents of a particular instrument’s imsets.
Table 2.3: NICMOS MULTIACCUM Listing from catfits
tt> catfits n3t501c2r_raw.fits

EXT# FITSNAME FILENAME              EXTVE  DIMENS BITPI OBJECT

0 n3t501c2r_raw n3t501c2r_raw.fits 16 n3t501c2r_raw.f
Accessing FITS Images
After you have identified which FITS image extension you wish to process, you can direct an IRAF/STSDAS task to access that extension using the following syntax:
fitsfile.fits[extension number][image section]
or
fitsfile.fits[keyword options][image section]
Specifying the extension number is the most basic method of accessing an HDU in a FITS file, but it is not necessarily the most useful syntax. Referring to an extension’s EXTNAME and EXTVER in the [keyword options] is often more convenient. If a number follows an EXTNAME, IRAF interprets the number as an EXTVER. For example, if extension number 6 holds the science image belonging to the imset with EXTVER = 2, as in the catfits listing above, it can be specified in two equivalent ways:
fitsfile.fits[6]
fitsfile.fits[sci,2]
Designating an EXTNAME without an EXTVER refers to the first extension in the file with the specified value of EXTNAME. Thus, fitsfile.fits[sci] is the same as fitsfile.fits[sci,1].
The syntax for designating image sections follows the IRAF standard. So, in the current example, the specifications
fitsfile.fits[6][100:199,100:299]
fitsfile.fits[sci,2][100:199,100:299]
both refer to a 100 by 200 pixel subsection of the same science image in fitsfile.fits.
Header Keywords and Inheritance
ACS, COS, NICMOS, STIS, and WFC3 data files use an IRAF image kernel convention that allows HDU extensions, under certain circumstances, to inherit keywords from the primary header. When this inheritance takes place, the primary header keywords are practically indistinguishable from the extension header keywords. This feature circumvents the large scale duplication of keywords that share the same value for all extensions. The primary header keywords effectively become global keywords for all image extensions. Note, the FITS standard does not include keyword inheritance, and while the idea itself is simple, its consequences are often complex and sometimes surprising to users.
In general, keyword inheritance is the default, and IRAF/STSDAS applications will join the primary and extension headers and treat them as one. For example, using imheader as follows on a FITS file will print both primary and extension header keywords to the screen:
Using imcopy on such an extension will combine the primary and extension headers in the output HDU, even if the output is going to an extension of another FITS file. Once IRAF has performed the act of inheriting the primary header keywords, it will normally turn the inheritance feature off in any output file it creates unless specifically told to do otherwise.
If you need to change the value of one of the global keywords inherited from the primary header, you must edit the primary header itself (i.e., “extension” [0]).
Keyword inheritance is not always desirable. For example, if you use imcopy to copy all the extensions of a FITS file to a separate output file, IRAF will write primary header keywords redundantly into each extension header. You can suppress global keyword inheritance by using the NOINHERIT keyword option in the file specification. For example:
im> imcopy fitsfile.fits[6][noinherit] outfile.fits
im> imcopy fitsfile.fits[sci,2,noinherit] outfile.fits
The resulting outfile.fits contains no global keywords from fitsfile.fits, except for keywords which were present in the extension header. In the first example, where the FITS image uses an absolute extension number, noinherit is the only entry needed in the FITS option field. In the second command, the noinherit option is bracketed with the EXTNAME and EXTVER keyword. For a complete explanation of FITS file name specifications, see:
http://iraf.noao.edu/iraf/web/docs/fitsuserguide.html
Appending Image Extensions to FITS Files
IRAF/STSDAS tasks that produce FITS images as output can either create new FITS files or append new image extensions to existing FITS files. You may find the following examples useful if you plan to write scripts to reduce MEF FITS files:
If the specified output file does not yet exist, it is created containing only the specified extension of the original file. For example, to copy the contents of the primary header of fitsfile.fits to a new FITS file named outfile.fits:
Note that imcopy will yield an error if an extension is not specified in the command. If the specified output file already exists and you want to append a new extension to it, you must include the APPEND option in the output file specification. The example below appends extension [sci,2] of fitsfile.fits to the existing file outfile.fits, while retaining the original EXTNAME and EXTVER of the extension. The noinherit keyword option prevents the copying of the primary header keywords from the input file into the output extension header:
     cl> imcopy fitsfile.fits[sci,2,noinherit] \
     >>> outfile.fits[append]
Note that the backslash is added to indicate that the remainder of the command follows on the next line, after the “>>>” prompt.
To change the EXTNAME or EXTVER of the appended extension, you can specify new values for these keywords in the output extension:
     cl> imcopy fitsfile.fits[sci,2,noinherit] \
     >>> outfile.fits[sci,3,append]
For obvious reasons, it is generally not advisable for two file extensions in the same FITS file to share the same EXTNAME and EXTVER values. However, if you must append an extension to an output file already containing an extension with the same EXTNAME/EXTVER pair you can do so with the DUPNAME option:
     cl> imcopy fitsfile.fits[7] \
     >>> outfile.fits[append,dupname]
If you need to replace an existing extension with a new output extension, you can use the OVERWRITE option as follows. Overwriting can cause a lengthy rewrite of the whole file to insert the new extension, if its size is not the same as the extension it replaces.
     cl> imcopy fitsfile.fits[sci,2,noinherit] \
     >>> outfile.fits[sci,2,overwrite]
2.2.2
FITS tables are used to store certain types of data from ACS, COS, NICMOS, STIS, WFC3, and WFPC2. For these instruments, OPUS produces association tables that list the exposures used to construct association products. In addition, reference data may be stored in FITS tables. This section describes:
This discussion assumes you are using STSDAS 3.2 or later. (The IRAF FITS kernel deals only with FITS images. The tables package in, STSDAS handles FITS table extensions.)
Accessing FITS Tables
You can access data in FITS table extensions using the same tasks appropriate for any other STSDAS table. The syntax for accessing a specific FITS table is similar to the syntax for accessing FITS images (see Section 2.2.1), with the following exceptions:
FITS tables must reside in a FITS table extension, in either ASCII form (XTENSION=TABLE) or binary form (XTENSION=BINTABLE).
For example, running catfits on the NICMOS association table n3tc01010_asn.fits provides the following output:
Extension number 1 holds the association table, which has EXTNAME=ASN and EXTVER=1. You can use the tprint task in the STSDAS tables package to print the contents of this table, and the following commands are all equivalent:
tt> tprint n3tc01010_asn.fits
tt> tprint n3tc01010_asn.fits[1]
tt> tprint n3tc01010_asn.fits[asn,1]
STSDAS tables tasks can read both FITS TABLE and BINTABLE extensions, but they can write tabular results only as BINTABLE extensions. Tasks that write to a table in-place (e.g., tedit) can modify an existing FITS extension, and tasks that create a new table (e.g., tcopy) will create a new extension when writing to an existing FITS file. If the designated output file does not already exist, the task will create a new FITS file with the output table in the first extension. If the output file already exists, the task will append the new table to the end of the existing file; the APPEND option necessary for appending FITS image extensions is not required. As with FITS images, you can specify the EXTNAME and EXTVER of the output extension explicitly, if you want to assign them values different from those in the input HDU. You can also specify the OVERWRITE option if you want the output table to supplant an existing FITS extension. For example, you could type:
This command would copy the table in the first extension of n3tc01010_asn.fits into the third extension of out.fits, while reassigning it the EXTNAME/EXTVER pair [asn,2] and overwriting the previous contents of the extension. Note that overwriting is the only time when it is valid to specify an extension, EXTNAME, and an EXTVER in the output specification.
Specifying Arrays in FITS Table Cells
A standard FITS table consists of columns and rows forming a two-dimensional grid of cells; however, each of these cells can contain a data array, effectively creating a table of higher dimensionality. Tables containing extracted STIS and COS spectra take advantage of this feature. Each column of a STIS or COS spectral table holds data values corresponding to a particular physical attribute, such as wavelength, net flux, or background flux. For STIS, each row contains data corresponding to one spectral order, and tables holding echelle spectra can contain many rows. Each cell of such a spectral table can contain a one-dimensional data array corresponding to that cell’s physical attribute and spectral order.
In order to analyze tabular spectral data with STSDAS tasks other than sgraph and igi (which have been appropriately modified to handle three-dimensional data tables), you will need to extract the desired arrays from the three-dimensional table. Two IRAF tasks, named tximage and txtable, can be used to extract the table-cell arrays. Complementary tasks, named tiimage and titable, will insert arrays back into table cells. The task tscopy will copy rows, columns, and subsets of tables. To specify the arrays which should be extracted from or inserted into the table cells, you will need to use the selectors syntax to specify the desired row and column. The general syntax for selecting a particular cell is:
intable.fits[extension number][c:column_selector][r:row_selector]
intable.fits[keyword options][c:column_selector][r:row_selector]
A column selector is a list of column patterns separated by commas. The column pattern is either a column name, a file name containing a list of column names, or a pattern using the IRAF pattern matching syntax (type “help system.match for a description of the IRAF pattern matching syntax). To obtain a list of the column names, you can run the tlcol task (type “tlcol infile.fits”).
A row selector parameter can be used to specify a certain row in the table. For example, if you specify:
IRAF will extract data from the table stored in the third extension of infile.fits, specifically from the columns labeled WAVELENGTH and FLUX, and will restrict the extraction to the rows where the spectral order (SPORDER) is within the range 68–70, inclusive. Alternatively, if you specify:
IRAF will obtain data from the table stored in the FITS file extension with an EXTNAME=SCI and EXTVER=2. The data will come from the column FLUX and be restricted to the row numbers 20–30, inclusive. All STSDAS and TABLES tasks are now able to use row and column selection. For a complete explanation of the table selector syntax, type “help selectors”.

1
A description of FITS format and various supporting documents can be found at the Web site http://fits.gsfc.nasa.gov/fits_home.html


Table of Contents Previous Next Index Print