Date: Tue, 18 Dec 2001 15:58:09 -0500 (EST)
From: "Howard A. Bushouse" <bushouse@stsci.edu>
Subject: New "Customized Reprocessing" SHARE writeup
To: bushouse@stsci.edu, dickinson@stsci.edu, donahue@stsci.edu, fraquelli@stsci.edu, gaffney@stsci.edu, gak@stsci.edu, keyes@stsci.edu, koekemoe@stsci.edu, mauro@stsci.edu, meurer@pha.jhu.edu, padovani@stsci.edu, sparks@stsci.edu, swade@stsci.edu
MIME-Version: 1.0
Content-MD5: VePBcPAeXUtp1GoUXnaQ1w==


Customized Reprocessing of Archived Data
========================================

Description:

Allow users to specify customized parameters to be used when running OTFR
to tailor the pipeline calibration to their scientific needs.

Scientific Case:

The standard processing applied in instrument calibration pipelines is
not always optimized for individual observations.  A typical routine
example is the extraction of a spectrum for a STIS spectral observation.
The pipeline assumes the target is a point source in all cases and uses
an extraction aperture based on the point-source assumption.  Summed or
spatially resolved spectra of extended sources require extraction apertures
that are customized in size and placement along the slit.  Another example
would be choosing different parameters for performing cosmic-ray rejection.

Unique STScI capabilities:

The calibration pipelines are STScI developed software, and the resident
instrument groups in the HST Division are the best source of knowledge for
what aspects of customized reprocessing would have the greatest utility.

Drawbacks:

Calibration pipeline software is already distributed to the community in
STSDAS, and this currently allows users to reprocess their data locally
in a customized fashion when necessary.  As described, this capability
would only add a special archive interface and offload the computations
from the user's site to STScI.  This may cause performance problems for
local STScI servers, depending upon the load.

Assumptions:

	Adding user-selectable processing options to StarView retrieval
	screens gives users access to only those processing options that
	are already available and built into the various HST instrument
	processing pipelines (calxxx routines). Furthermore, the only
	types of options can that be enabled via this interface are 
	those that do not require interactive input or guidance from
	the user, nor can the user provide any type of input data file.

Required Decisions:

	A decision must be made as to whether or not any particular
	processing option in a pipeline should be made available via
	OTFR processing and retrieval methods. Options that significantly
	increase the required processing time or require iterative
	settings of options to achieve optimum results, for example,
	may be best left to the individual user to perform on their own.

Min and Max Goals:

	The range of goals for this particular project is quite narrow,
	because implementing the necessary mechanism for any processing
	option essentially meets the needs for access to all options.
	The goal is to implement the ability for allowing users to set all
	available pipeline options that meet the assumptions stated above.
	
Implementation Plan:

	This work is already planned as a follow-on to the basic DADS
	redesign that is currently in progress. The enhancements listed
	below will be implemented after basic functionality of the new
	system has been finished.

	Necessary steps in DADS distribution:

	1. Modify the distribution DTD by adding to OTFR processing options
	   the ability to specify lists of "keyword = value" items, attaching
	   them either globally or to specific rootnames.

	2. Modify the XML parser to pick up these lists as part of the OTFR
	   processing options.

	3. Modify the transaction generator to pass these lists along to the
	   OTFR transactions that are generated.

	4. Modify the OTFR transaction and the OTFR message format to include
	   these "keyword = value" lists.

	Necessary steps in OTFR processing:

	1. Modify OTFR processing to accept "keyword = value" lists and give
	   them to the appropriate calibration software step (which could be
	   done via the command line).

Required Resources:

	DADS: 1 FTE month for coding and unit testing
	OTFR: 1 FTE month for coding and unit testing
	Combined system: 1 FTE month for testing
	Total: 3 FTE months

Time Scales:

	Expect that work can begin in late 2002, around Oct. 1. Operational
	implementation would then occur in early 2003.