Combining Spectral Data ======================= Description: This enhancement would permit the user to specify data sets from multiple observations of a target and request that they be optimally combined into a single, summed spectrum. Many STIS CCD spectral observations are obtained in a dither pattern to optimize CR rejection, to avoid hot pixels, and to completely sample the spatial domain. These dithered observations could be combined in an OTFR process. Science Case: Many HST archival spectral observations consist of multiple data sets. In some cases these are merely repeated, unrelated observations, and in others it is a result of a deliberate observing strategy. The combinations produced via OTFR could be multiple observations in a single wavelength region whose combination increases the signal-to-noise ratio, or it may consist of several observations at different grating settings that could be combined to obtain broader wavelength coverage in the product. Automated combination of dithered STIS CCD spectral observations would automatically produce a better total product for the archive user. Unique STScI capability: The instrument groups at STScI have the specific knowledge of the instruments and the associated pointing data (WCS information and jitter files) that would be needed to perform this task routinely. Drawbacks: Blindly combining spectra obtained at different epochs could produce errant results for time-variable objects. Judicious scientific input on the part of a careful observer is generally required to assure the validity of any result. An automated system could circumvent the careful scrutiny usually applied when users combine data sets on their own. In the case of merging dithered long-slit spectral observations, we note that combining spectral images has the same (if not more) problems with alignment and registration that affect combining images. One often must tweak parameters manually in an iterative process to achieve an optimum result. Assumptions The data headers need to contain contain accurate spatial information. This could be either accurate WCS information (another SHARE task listed at higher priority), or accurate relative spatial information as given by POS TARG or Pattern_Type keywords in the data header should they exist in certain specific cases. The implementation plan described below assumes sufficient coordinate accuracy offset info in header to perform either a) corrections to wavelength scale at integral pixel scale or b) combination in spatial direction (i.e. in absence of any cross-correlation techniques). Required Decisions Decisions must be made concerning whether particular options be made available to the user. For example, should the level of flux variablility allowed in a combined dataset be left to the user. Also, choices potentially abound for specification of output wavelength grids in overlap regions. The complexity of the front-end interface development, especially the testing component, scales directly with the number of such options. We note that it is already possible to simply overplot preview spectra from different instruments/missions archived in MAST using a tool which can be accessed from the MAST scrapbook pages (http://archive.stsci.edu/scrapbook.html) and any other returned mission page. We also note that work is on-going within MAST and SSG to expand on the current capabilities of SpecView, a tool which allows users to overplot and manipulate spectra from different instruments/missions interactively. The results of such efforts should be taken into account when implementing the enhancement discussed here. As with other SHARE tasks we should confirm community interest before implementing this capability. We listed this at moderate priority among the SHARE tasks. Before investing substantial resources, we should be satisfied that this would benefit a signficant number of users. (First step: identify the number of STIS observations that employ along slit dithering.) We might also explore ST/ECF interest in carrying out some portion of the work described here. Min and Max Goals We define minimum, intermediate, and maximum goals. The minimum and intermediate goals correspond to activities that can be completed by relatively straightforward adaptation of existing tasks or tasks that are presently being developed. The terminology, minimum and intermediate, is intended to describe relative prioritization of these tasks. minimum goals: I. no spatial dithering: a) combine spectra taken with same grating / same central wavelength; b) combine spectra taken with same grating / different central wavelengths c) combine spectra taken with different gratings / similar resolution d) combine spectra taken with different resolutions (e.g. STIS L, M, or E modes or STIS M / GHRS, STIS L / GHRS or FOS, etc) Intermediate goals: II. integral pixel spatial dithers (STIS only) STIS tool is presently in development to use existing STSDAS tasks imshift, imcalc, ocrreject) to perform integral pixel shifts, combines, and CR rejection (Dressel, Busko). Completion of the scripting for this capability is presently expected not before late spring, 2002 (personnel resources are driver, not algorithm development). This tool could be driven by our interface to produce combined / CR-rejected datasets from STIS observations that were taken with integral pixel dither patterns. To implement this minimum goal, only accurate *relative* spatial information need be present in the data headers. Accurate absolute WCS solutions are not required. Maximum goals: III. fractional pixel spatial dithers (STIS only) This goal envisions utilization of drizzling or some other image combination process (e.g., super-pixilization or interlacing) to combine spectroscopic observations that have been intentionally dithered - usually perpendicular, but occasionally parallel, to dispersion. This process would allow the best possible sampling of the spatial component of the PSF and/or give the best sampling of the LSF. Existing drizzle tasks might be adapted, however the handling and population of header information must be researched and modified. Additionally, substantial effort would be required to correct STIS geometric distortion mapping to a form that drizzle could use. The drawbacks in attempting to automate an often iterative procedure have been discussed above. Again, we assume that the headers provide sufficient WCS information concerning the dithered offsets. Implementation Plan For each of the three goals, the following sequence applies: - Develop requirements document. Include algorithmic description and prototype software. - Programming team develops required software and interfaces. - Archive team develops data set selection method (same for all) - Test - Document (external, for user support; internal, for maintenance). - Deploy I. minimum goal: Splicer tool can be used to combine spectra on same or different wavelength grids. Splicer allows specification of output wavelength grid, so choices can be offered concerning overlap regions. Splicer also allows weighting vectors to be specified for all spectra to be combined so choices can be offered concerning combination in overlap regions, as well as exposure time weighting if desired. Can use existing splice task which is designed to combine multiple datasets with options for specification of weight, quality, and wavelength sampling vectors. a) use splicer, place on common wavelength grid b) use splicer, establish treatment for wavelength grid and combination in overlap region (user-input: finest, coarsest wavelength grid or some ramp or discontinuity or simply default to finest wavelength grid); also establish combination mechanism in overlap regions (average, special weighting). c) as b) d) as c) and b) with particular concerns about combination of radically different wavelength grids. II. intermediate goal: integral pixel offsets in spatial direction This is a STIS-only application. Use scripts currently in development. Allow combination into single file with CR rejection. Wavelength gridding is assumed accurate and is not re-sampled. III. maximum goal: fractional pixel offsets in spatial direction Again, a STIS-only application. Modify existing drizzle tasks to properly populate output headers. Must create geometric distortion maps that are compatible with drizzle process requirements. Explore adaptation of existing pipeline processing to utilize _flt file or counts-image produced by x2d as input to the process. For higher S/N cases appropriate cross-correlation tools also exist. Output is a single combined file. common to all: For purposes of candidate image selection, must establish limit on flux level variations (either real variability or possible calibration differences) that will be combined (user-defined or default); wavelength overlap treatment options which imply weighting vectors that will be passed to splicer; infrastructure to pass identification of files selected for combination (much overlap with selection infrastructure required for other SHARE tasks) Required Resources For interface development we include below under "archive team" only the resources specific to this task (requirements, design, prototype, testing). The interface should be quite similar to those developed for other SHARE components or addition of a new screen to StarView. We will need to establish actions to pass user choices or defaults to splice utility. - For Min Goals (splicing and integral-pixel offsets) - Develop requirements document. Include algorithmic description and prototype software. splicing: 2 FTE weeks; integral: 2 FTE weeks - Programming team develops required software and interfaces. splicing: 1 FTE month integral: 1 FTE month - Archive team develops data set selection method (same for all) splicer-specific: 1 FTE month generic interface: 1 FTE month - Test 1 FTE month total - Document (external, for user support; internal, for maintenance). 1 FTE month total - Deploy 1 FTE month total Total FTE for Min goals: 8 months - For Max Goals Effort required to establish fractional pixel interpolation procedures. - Develop requirements document. Include algorithmic description and prototype software. 3 FTE months - Programming team develops required software and interfaces. 2 FTE months - Archive team develops data set selection method (same for all) marginal additional to minimum - Test 2 FTE months - Document (external, for user support; internal, for maintenance). 1 FTE month - Deploy 1 FTE month Total for 9 months Time Scales (for min and max goals) - For Min Goals Minimum level implementation can proceed with existing tools; apart from user interface, little development is required; FTE expenditure is essentially same as calendar time needed for all but documentation and testing where typical multi-tasking of developer and IS times applies. Estimate testing and documentation require 2 months each (1 FTE month each); yields total time estimate of 10 months. - For Max Goals Requirements and algorithm development will require substantial interaction between IS and developers. Estimate 4-6 months to yield requirements document. Given typical multitasking environment, testing will require 3-4 months. We assume little time is required on archive interface development as selection screens will have been developed for minimum level implentation and/or other SHARE tasks. Remaining FTE estimates translate to equivalent calendar time estimates; yields total time estimate of 11-14 months.