7.2 The HST Data Archive

7.2.1 Web Access to the HST Data Archive

All science and calibration data are placed in the HST Data Archive. Science data become immediately available (after pipeline processing) to a program’s Principal Investigator (PI), as well as those designated by the PI. These data may be retrieved after the PI has registered as an Archive user and are normally proprietary for a period of one year from the date of observation (see Section 5.1 of the Call for Proposals for information on data rights).

On average, the science data from HST flow through the production pipeline and into the Archive within one day after observation on the telescope. Some data may take as long as three to five days. The observer is notified by e-mail when the first datasets reach the archive and is provided with Web tools to track a visit’s completeness and to retrieve the data generated by the pipeline. The time required for retrieving data from the Archive is typically a few minutes. However, occasional software or system failures may lengthen the processing and retrieval times.

If there are strict scientific requirements for data receipt within days after execution, such as to provide pointing corrections for tightly scheduled visits, there are resource-intensive methods to expedite delivery of data. Such requirements must be stated in the proposal so that the resource needs can be determined and reviewed (see Section 9.3 of the Call for Proposals).

As of October 2012, the HST Data Archive contains over 68.76 Tbytes of data. The heart of the Archive is the Data Archive and Distribution Service (DADS) where data are stored in hard disk arrays, in a collection of Ultra Density Optical disks for off-site backup storage, and in databases that comprise the Archive catalog. Hardware and software that support ingest and distribution of HST data are also managed by DADS.

7.2.1 Web Access to the HST Data Archive

Most of the data in the HST Data Archive are public and may be retrieved by any user. The archive can be accessed through the MAST Web page. The Web interface can perform simple searches by object name or location, or by lists of names or locations, and it can retrieve data and calibration files. Users can also filter their data retrievals to obtain only the files they need. The Archive has a Web-based interface for accessing instrument-specific databases, called Starview. This Web interface is available at:

http://starview.stsci.edu/

The MAST Web site allows previews of most publicly available images and spectra. Its interface includes integrated access to the Digitized Sky Survey (DSS) and allows the user to access the Set of Identifications, Measurements and Bibliography for Astronomical Data (SIMBAD) or NASA/IPAC Extragalactic Database (NED) to look up the coordinates of an object by name.

All ACS, COS and WFC3 data requested from the Archive, as well as STIS data acquired after Servicing Mission 4, are reprocessed in the pipeline at the time of the request. On-The-Fly Reprocessing (OTFR) takes the science data in spacecraft package format and generates calibrated files. It prepares new FITS files with each request, which allows a clean integration of new headers with updated keywords that are fully compatible with the latest version of the pipeline and calibration software.

Data from legacy instruments, NICMOS, along with STIS data acquired before Servicing Mission 4, were reprocessed with optimized software and reference files, and are available from the HST Static Archive.

While STIS was inoperable after its failure in August 2004, all pre-failure archival STIS data were recalibrated. Since the Summer of 2007, these data are served statically from DADS rather than being reprocessed through OTFR, allowing for significantly faster data retrievals. A description of the improvements made as part of this recalibration is available at the STIS Closeout Web page. Concurrently with Servicing Mission 4, OTFR was re-enabled for all post-SM4 STIS data. Archive requests for the older pre-SM4 STIS data continue to be filled using the Static Archive, and can also be accessed directly via FTP from archive.stsci.edu, in the directory /pub/hstonline.

All WFPC2 observations have been recalibrated and are served statically from DADS. A description of the improvements made as part of this recalibration is available at the WFPC2 Reprocessing Web page: 

http://www.stsci.edu/hst/wfpc2/wfpc2_reproc.html.

STScI maintains an “Archive Hotseat” to which all archive-related questions, problems, or comments should be referred. The Archive Hotseat can be reached by e-mail at archive@stsci.edu or by phone at 410-338-4547.