Space Telescope Science Institute
Call for Proposals and HST Primer
help@stsci.edu
Table of Contents Previous Next Print


Hubble Space Telescope Primer for Cycle 22 > Chapter 7: Data Processing and the HST Data Archive > 7.2 The HST Data Archive

7.2 The HST Data Archive
All science and calibration data are placed in the HST Data Archive. Science data become immediately available (after pipeline processing) to a program’s Principal Investigator (PI), as well as those designated by the PI. These data may be retrieved after the PI has registered as an Archive user and are normally proprietary for a period of one year from the date of observation (see Section 5.1 of the Call for Proposals for information on data rights).
As of September 1, 2013, the HST Data Archive contains over 96.92 Tbytes of data. The heart of the Archive is the Data Archive and Distribution Service (DADS) where data are stored in hard disk arrays, in a collection of Ultra Density Optical disks for off-site backup storage, and in databases that comprise the Archive catalog. Hardware and software that support ingest and distribution of HST data are also managed by DADS.
On average, science data from HST flow through the production pipeline and into the Archive within one day after observation on the telescope. Some data may take as long as three to five days. Observers are notified by e-mail when the first datasets reach the Archive. They are also provided with Web tools to track a visit’s completeness and to retrieve the data generated by the pipeline.
The time required for retrieving data from the Archive is dependent on the number of products/exposures being requested and the instrument that produced the data. For a small request consisting of STIS data, response time should be a few minutes. For a large request consisting of ACS and/or WFC3 data, the response time may be longer than a day. In addition, if the Archive is being heavily used when the request is made, response time will be longer than usual.
If there are strict scientific requirements for data receipt within days after execution, such as to provide pointing corrections for tightly scheduled visits, there are resource-intensive methods to expedite delivery of data. Such requirements must be stated in the proposal so that the resource needs can be determined and reviewed (see Section 9.3 of the Call for Proposals).
7.2.1 Web Access to the HST Data Archive
Most of the data in the HST Data Archive are public and may be retrieved by any user.
The Archive can be accessed through the MAST webpage, using a Web interface to perform simple searches by object name or location, or by lists of names or locations. From that interface, data and calibration files of interest can be marked for retrieval and submitted to the Archive. It is also possible to request specific data products instead of the entire complement of files in a standard delivery. Retrievals can also be made using Starview, a Web interface using instrument-specific databases, available at: http://starview.stsci.edu/
The MAST website allows previews of most publicly available images and spectra. Its interface includes integrated access to the Digitized Sky Survey (DSS) and enables users to access SIMBAD (Set of Identifications, Measurements and Bibliography for Astronomical Data) and NED (NASA/IPAC Extragalactic Database) to look up the coordinates of an object by name.
When a request is placed for data from ACS, COS, and WFC3, as well as STIS data acquired after Servicing Mission 4, the On-The-Fly Reprocessing (OTFR) pipeline begins its processing with science data in the "spacecraft package format" (known as POD files) to produce raw and calibrated data products. New FITS files are created with each request, allowing a clean integration of science data with new headers containing updated keywords that are fully compatible with the latest versions of pipeline and calibration software.
Data from legacy instruments (including WFPC21) and NICMOS, along with STIS2 data acquired before Servicing Mission 4, were reprocessed with optimized software and reference files, and stored in the Archive. When a request is made, data from these instruments are already available for immediate delivery and do not pass through OTFR.
STScI maintains an “Archive Hotseat” to which all archive-related questions, problems, or comments should be referred. The Archive Hotseat can be reached by e-mail at archive@stsci.edu or by phone at 410-338-4547.
1
All WFPC2 observations have been recalibrated and are served statically from DADS. A description of the improvements made as part of this recalibration is available at the WFPC2 Reprocessing webpage.
2
STIS data taken before its August 2004 failure were recalibrated, and since the summer of 2007, have been served statically from DADS instead of being run through OTFR, allowing for significantly faster retrievals. These data can be obtained from the Archive, or via ftp from archive.stsci.edu in the directory /pub/hstonline/. More information is available at the STIS Closeout webpage. Post-SM4 STIS data, however, are processed in the OTFR pipeline.


Hubble Space Telescope Primer for Cycle 22 > Chapter 7: Data Processing and the HST Data Archive > 7.2 The HST Data Archive

Table of Contents Previous Next Print