HST Data Handbook for COS

1.1 Archive Overview

---

All Hubble Space Telescope (HST) data files are stored in the Hubble Data Archive (HDA), which forms part of the Multimission Archive at STScI (MAST)¹. HST Guaranteed Time Observers (GTOs), Guest Observers (GOs), and Archival Researchers can retrieve data in one of the following ways:

• Via ftp/sftp, either directly from the HDA, or by accessing a staging disk set up upon request.
• Via data storage media (CDs or DVDs) which are shipped to the user.
• Via the use of external hard drives loaned by STScI: Users requesting large amounts of data (>100GB) over a short period of time may request the data be written to hard disk. The disk will be delivered to the user for reading, and then the disk must be returned. Use of this retrieval method must be coordinated through the MAST Help Desk (archive@stsci.edu). The number of external hard drives is limited, so users are strongly encouraged to make arrangements in advance. For reference, a DVD can hold 4.7 GB.

All datasets retrieved from the HDA, regardless of the method used, will be in FITS (Flexible Image Transport System) format. Further information on HST file formats is presented in Chapter 2.

Non-proprietary data in the HDA can be retrieved electronically either by registered HDA users or via anonymous login. Proprietary data may not be retrieved except by a registered HDA user who has the permission of the program's Principal Investigator (PI). Note that HST PIs are not automatically registered. PIs should register before their first observations have been taken. All calibration observations as well as observations made as part of the GO Parallel programs are immediately public. All observations made as part a Treasury Program will either be immediately public or have only a brief proprietary period. The High-Level Science Products (HLSP) section of MAST also contains several sets of publicly available and fully reduced HST data such as the Ultra Deep Field and the GEMS survey data. See http://archive.stsci.edu/hlsp/index.html for a complete listing.

The archive recommends to ask for compressed data, which distinctly shortens the retrieval times without any information loss.

Advanced Camera for Surveys (ACS), Cosmic Origins Spectrograph (COS), and Wide Field Camera 3 (WFC3) users should note that the preferred option for data retrieval is from the HDA staging disk, via ftp/sftp. Users retrieving large numbers of ACS, COS, or WFC3 files should also consider requesting them on DVDs.

1.1.1 Archive Processing

The HDA contains all observations ever made by HST and a catalog that describes the data. Each time a user makes a data request, the HDA delivers data which are either processed with the On-The-Fly-Reprocessing (OTFR) system or simply retrieved from a set of final calibrated and statically stored data, as listed in Table 1.1.

Table 1.1: HST Instrument Data Processing.

Instrument	HDA Processing
ACS	OTFR
COS	OTFR
FGS	Static
FOC	Static
FOS	Static
GHRS	Static
HSP	Static
NICMOS	OTFR
STIS	Static & OTFR1
WFC3	OTFR
WF/PC-1	Static
WFPC2	Static1

1STIS and WFPC2 data taken before SM4 will not be processed with OTFR but will instead be fully calibrated and statically stored in the HDA.

The OTFR system reconstructs and calibrates the data at the time of retrieval. This allows users to obtain data calibrated with up-to-date reference files, parameters, and software. OTFR makes use of the OSS/PODPS Unified System (OPUS) system to orchestrate the processing of the data. OPUS retrieves the HST telemetry data (POD files) from the Data Archiving and Distribution System (DADS) and creates raw data files in a step called Generic Conversion. Calibration reference files specific to the different modes of HST operations are prepared and archived in the Calibration Data Base System (CDBS). During Generic Conversion, CDBS is queried to determine which reference files apply to the specific observation being processed. OPUS then calibrates the data using the specific instrument's calibration software and reference files. After calibration, the files are returned to DADS for distribution to the user, as calibrated and/or raw files.

All STIS data collected before its failure in August 2004 have been fully reprocessed and calibrated, and are now stored in a static archive. (STIS data taken after SM4 will be processed via OTFR.) A similar effort is currently underway to reprocess and calibrate WFPC2 data. All data taken with WFPC2 until its decommissioning will be reprocessed and permanently stored in a static archive.

Data for FOC, FOS, GHRS, HSP, and WF/PC-1 do not pass through OTFR or any pipeline. For FOC, FOS, and GHRS, final calibrated archives have been produced, since no further improvements in the calibration for these instruments are expected. The user is provided with a copy of the raw and final calibrated data from the static (final) archive once a request is made. For HSP and WF/PC-1, no reprocessing or recalibration has been done nor is any planned. Once raw data from these instruments are retrieved from the HDA, they need to be calibrated locally by the users.

Searches and retrievals are available through the MAST Web site, http://archive.stsci.edu. The MAST Web site allows cross correlation of HDA searches with other MAST mission archives. It offers simple previews of HST datasets when available, as well as links to references citing a given dataset using the Astrophysics Data System (ADS). The MAST Web site is discussed in more detail in Section 1.2.

1.1.2 Archive Registration

The simplest way to register for HST data is to complete the form on the Web page at: http://archive.stsci.edu/registration. If problems occur, registration requests may also be sent to the HDA Help Desk, at archive@stsci.edu. The PI of each HST proposal must request access (i.e., authorization) to their proprietary data for themselves, and for anyone else whom the PI wants to have access to them. PI retrieval permission is not granted automatically, for security reasons. PIs wishing to allow access to their proprietary data should make that request to archive@stsci.edu. When registration is granted, your account will be activated automatically, and you will receive your username and password via e-mail.

1.1.3 Archive Documentation and Help

The MAST Web site provides a wealth of useful information, including an online version of the HST Archive Manual available at http://archive.stsci.edu/hst/manual. Investigators expecting to work regularly with HST and other datasets supported by MAST should also subscribe to the MAST electronic newsletter by sending an e-mail message to archive_news-request@stsci.edu with the word "subscribe" in the body of the message. Questions about the HDA can be directed to archive@stsci.edu, or by phone to (410) 338-4547.

¹MAST currently includes data from HST, FUSE, XMM-OM, GALEX, IUE, EUVE, ASTRO (HUT, UIT, WUPPE), ORFEUS (BEFS, IMAPS, TUES), Copernicus, and ROSAT. Data from the FIRST radio survey, Digitized Sky Survey (DSS) and Sloan Digital Sky Survey (SDSS), the HPOL spectropolarimeter, and the Guide Star Catalog (GSC) are also available.