Date: Mon, 14 Jan 2002 09:46:51 -0500
From: Jim Rose <rose@stsci.edu
X-Accept-Language: en
MIME-Version: 1.0
To: Gerry Kriss <gak@stsci.edu, Daryl Swade <swade@stsci.edu, Stefi Baum <sbaum@stsci.edu
Subject: SHARE and OPUS
Content-Transfer-Encoding: 7bit

I have been asked to imagine the future of OPUS.  My speculations have
been an attempt to integrate our current capabilities with those demanded
by a possible NVO, and the opportunities presented by upcoming AURA-wide
collaborations.  There have been a number of alternative futures recommended,
but one specific possibility calls out for immediate attention.

The bottom line is that we should be prepared to open up the Institute's
OTFR pipeline to plug-and-play post calibration analysis which can build
extended relations in the HST archive.

SHARE has probably dealt with this idea before, but just to make sure
we're talking about the same thing, let me give an example.

 COS spectra in the FUV will detail the Lyman-Alpha forest.  Dr. PDQ
 writes a specialized task which automatically calculates specific velocity
 adusted line ratios as well as a number of other characteristics
 of each spectrum.  The output is written as a FITS table (of perhaps
 a single row).
 
 That task will run for each COS FUV exposure with a limiting S/N 
 ratio processed through OTFR.  The resulting statistics will populate
 a private table (which will become public in a year), and updates
 will be ftp'd to Dr. PDQ each (day/week/month).
 
The advantage to the individual archive researcher (AR) is clear: his task
will be run against all appropriate exposures flowing through OTFR, without
having to do a StarView query, OTFR request, FTP the data, perform the
analysis locally and obtaining the result. 

The advantage for the HST archive is two-fold.  First plug-and-play analysis
could significantly reduce the amount of network traffic, allowing improved
response times for all other archive requests.  But more importantly the
resulting tables could significantly enrich the archive, giving all future
archive researchers more information about the nature of HST exposures in
the extended catalog.

One of the goals of the NVO is to support distributed computing, and the
Sloan experiments which support complex queries of gigantic databases is
a dramatic achievement.  OTFR support of plug-and-play is a parallel
demonstration of the win-win strategies for the NVO. 
   
The role of OPUS, or really the whole of DST, would be to make such an 
OTFR extension an easy process for the independent scientist to handle.
Much of the setup might be handled with simple web applications which
provide test suites and documentation tools.  The OPUS team needs to
carefully construct an OTFR sandbox in which it is safe to run such
analysis application.  And the pipeline environment needs to be enhanced
to adapt to a fan-out to an arbitrary number of tasks.

I see this as a terrific opportunity for the Institute to, once again,
provide the Astronomy community with a significant value.  OPUS can already
handle most of what is required, but there is still work to be done.

We need to know if SHARE has considered the consequences of this policy,
whether SHARE has already made such a recommendation, whether SHARE can
support us in opening up the Institute's pipeline to the scientific
community.

-- 
James F. Rose                           rose@stsci.edu
Computer Sciences Corporation           (410) 516-7651
Space Telescope Science Institute       (410) 516-8615 (FAX)
3700 San Martin Drive                   http://www.stsci.edu/opus/
Baltimore, MD 21218                     http://www.arborworks.org/