All science and calibration data, along with a large fraction of the engineering data, are placed in the
HST Data Archive. Science data become immediately available to a program’s Principal Investigator (PI) and those designated by the PI. These data may be retrieved after the PI has registered as an archive user and are normally proprietary for a period of one year (see
Section 5.1 of the Call for Proposals for information on data rights).
On average, the science data from HST flow through the production pipeline and into the archive within one day after observation on the telescope. Some data may take as long as three to five days. The observer is notified by e-mail when the first datasets reach the archive and is provided with Web tools to track a visit’s completeness and to retrieve the data generated by the pipeline. The time required for retrieving data from the archive is typically a few hours. However, occasional software or system failures may lengthen the processing and retrieval times.
If you have strict scientific requirements for data receipt within days after execution, such as to provide pointing corrections for tightly scheduled visits, there are resource-intensive methods to expedite delivery of data. If you have such requirements, they must be stated in your proposal so that the resource needs can be determined and reviewed (see
Section 9.3 of the Call for Proposals).
As of October 2009, HST Data Archive contained over 903,000 individual observations. These observations, along with engineering and other supporting information, comprise over 42 Terabytes of data. About 180 new observations are archived every day. The heart of the archive is the Data Archive and the Distribution Service (DADS), hard disk arrays where the data are stored, a collection of Ultra Density Optical disks for off-site back-up storage, databases that comprise the archive catalog, and the hardware and software that support the ingest and distribution of
HST data.
Most of the data in the HST Data Archive are public and may be retrieved by any user. The archive can be accessed through the
MAST Web page. The Web interface can do simple searches by object name or location or by lists of names or locations, and it can retrieve data and calibration files. The archive is offering a new Web based Starview interface to replace the Starview Java application. The new Starview supports much of the same functionality of the old Starview, permitting users access to instrument specific database tables. Users are able to choose the metadata of interest to construct their own, personalized queries and output. The application is available at:
The MAST Web site allows you to preview most of the publicly available images and spectra. The interface also offers integrated access to the
Digitized Sky Survey (DSS) and allows the user to access the
Set of Identifications, Measurements and Bibliography for Astronomical Data (SIMBAD) or
NASA/IPAC Extragalactic Database (NED) to look up the coordinates of an object by name.
All ACS, COS and WFC3 data requested from the archive, as well as all new NICMOS and STIS data acquired after SM4, will be reprocessed in the pipeline at the time of the request. On-The-Fly Reprocessing (OTFR) takes the science data in spacecraft package format and generates calibrated files. On-The-Fly Reprocessing prepares a new FITS file with each request, which allows a clean integration of new headers with updated keywords that are fully compatible with the latest version of the pipeline and calibration software. Data from the legacy instruments and data from the NICMOS and STIS acquired before SM4 were reprocessed with optimized software and reference files, and are available from the static archive. DADS allows users to filter their retrievals to obtain only the files they need (as well as allowing secure ftp retrievals for most users and retrieval of compressed data).
While STIS was inoperable after its failure in August 2004, all pre-failure archival STIS data were recalibrated, and starting in Summer of 2007, these data were served from a static archive rather than being reprocessed through OTFR, allowing for significantly faster data retrievals. A description of the improvements made as part of this recalibration are available at the
STIS Closeout Web page. Concurrently with SM4, OTFR was re-enabled for all post-SM4 STIS data. Archive requests for the older pre-SM4 STIS data continue to be filled using the static archive.
Similarly, while NICMOS was inoperable after the SI C&DH failure in September 2008, all pre-SM4 archival NICMOS data were recalibrated, and these data are served from a static archive rather than being reprocessed through OTFR, allowing for significantly faster data retrievals. OTFR is enabled for all post-SM4 NICMOS data.
Finally, all observations from the WFPC2 have been recalibrated and are available in the static archive. A description of the improvements made as part of this recalibration are available at the WFPC2 Reprocessing Web page:
STScI maintains an “Archive Hotseat” to which all archive-related questions, problems, or comments should be referred. The Archive Hotseat can be reached by email at
archive@stsci.edu or by phone at 410-338-4547.
