OPUS: The FUSE Science Data Pipeline

J.F. Rose^a,
C. Heller-Boyer, M.A. Rose, M.Swam, W.Miller^b,
G.A. Kriss and W. Oegerle^c

^aComputer Sciences Corporation, Inc. (CSC)
Space Telescope Science Institute
3700 San Martin Drive
Baltimore, MD 21218

^bAssociation of Universities for Research in Astronomy (AURA)
Space Telescope Science Institute
3700 San Martin Drive
Baltimore, MD 21218

^cThe Johns Hopkins University
Department of Physics and Astronomy
Far Ultraviolet Spectroscopic Explorer (FUSE) Project
3400 North Charles Street
Baltimore, MD 21218

1. ABSTRACT

This paper describes how the OPUS pipeline, currently used for processing science data from the Hubble Space Telescope (HST), was used as the backbone for developing the science data pipeline for a much smaller mission. The Far Ultraviolet Spectroscopic Explorer (FUSE) Project selected OPUS for its data processing pipeline platform and selected the OPUS team at the STScI to write the FUSE pipeline applications.

A total of 105 new modules were developed for the FUSE pipeline. The foundation of over 250 modules in the OPUS libraries allowed development to proceed quickly and with considerable confidence that the underlying functionality is reliable and robust. Each task represented roughly 90% reuse, and the project as a whole shows over 70% reuse of the existing OPUS system.

Taking an existing system that is operational, and will be maintained for many years to come, was a key decision for the FUSE mission. Adding the extensive experience of the OPUS team to the task resulted in the development of a complete telemetry pipeline system within a matter of months. Reusable software has been the siren song of software engineering and object-oriented design for a decade or more. The development of inexpensive software systems by adapting existing code to new applications is as attractive as it has been elusive. The OPUS telemetry pipeline for the FUSE mission has proven to be a significant exception to that trend.

Keywords: HST, OPUS, STScI, Hubble Space Telescope Science Institute, FUSE, Far Ultraviolet Spectroscopic Explorer, Distributed processing, Operational pipeline, Telemetry processing, Software reuse.

2. INTRODUCTION

OPUS is the production science data pipeline at the Hubble Space Telescope Science Institute. As a dynamic event-driven distributed processing system, OPUS provides an environment designed to handle a large number of observations processed through many steps across a network of computers. OPUS is also an automated environment that monitors that processing, and allows an operator to quickly identify what actions need to be taken. A recently enhanced system, OPUS has been ported to the UNIX platform and supports mixed UNIX-VMS platform operations.

While OPUS was developed in an environment that required a great deal of attention to throughput, speed, efficiency, flexibility, robustness and extensibility, it is not only suited for high volume projects. The OPUS platform is a small compact system designed to solve a specific problem in a robust way, distributed processing with controlled monitoring.

This paper specifically addresses the use of OPUS for a smaller mission. The Far Ultraviolet Spectroscopic Explorer mission selected OPUS for its pipeline platform and selected the OPUS team at the STScI to write the FUSE pipeline applications. That pipeline development project is nearly complete, and well within budget. The obvious savings are directly related to the great extent we were able to reuse the existing OPUS libraries, limiting the amount of new code that had to be developed and tested.

3. FUSE

The primary objective of the FUSE mission is to obtain high-resolution spectra of faint objects throughout our galaxy and beyond in the wavelength region of 910 to 1190 Angstroms. The far ultraviolet region contains a wealth of information on the density, pressure, temperature, and composition of astrophysical plasmas and neutral gases in a broad array of environments, yet is one of the least explored parts of the electromagnetic spectrum.

FUSE is a pathfinder for NASA's Mid-Class Explorer (MIDEX) missions. NASA, in an effort to reduce mission cost and the development cycle without compromising the science, has developed new guidelines and techniques that will be proven by the MIDEX programs. This is accomplished primarily by issuing strict cost caps, defining clear high level science objectives, and empowering the Principal Investigator (PI) to be responsible for all program decisions.

It is these guidelines which, in part, led the FUSE team to select OPUS as the science pipeline platform for the mission. As a proven system already operational at the Space Telescope Science Institute (STScI), the OPUS platform is based on many years of processing experience. The goal of the OPUS platform is to reduce the amount of operational `hands-on' effort required to process the large volume of data arriving from the spacecraft. Such a `lights-out' operational scenario is consistent with the long term cost concerns of the FUSE project.

In addition to selecting the OPUS platform, the FUSE project selected the OPUS team at STScI to implement the FUSE pipeline applications. With a great deal of experience in developing the pipeline applications for each of the various instruments on the Hubble Space Telescope (HST), the OPUS team brought both an understanding of the pipeline requirements, and a body of reusable software which can be applied to the task of processing the FUSE telemetry.

Some of the key members of the FUSE team have prior experience with space missions and are familiar with the problems involved with managing the mechanics of data processing. Rather than devoting scarce scientific resources to the development of routine data processing tasks, the FUSE team chose to focus on the requirements for calibration and the scientific interpretation of the data. By selecting an existing system, the OPUS platform, to help manage the data during operations, and an experienced team to develop the FUSE pipeline applications, costs have been carefully controlled and the FUSE science staff freed to focus on the science.

4. OPUS

Seen from different vantage points OPUS is not one but four different things:

1. OPUS is a distributed processing platform, independent of the applications which it runs and monitors;
2. OPUS is a library of software packages that support both the blackboard architecture for the distributed platform as well as the standard applications;
3. OPUS consists of a collection of instrument specific pipeline applications which have been developed primarily for the instruments aboard the HST;
4. OPUS is a team of experienced software engineers who collectively have many years of experience building telemetry pipelines for spacecraft missions.

4.1 OPUS Platform

The OPUS platform is a distributed pipeline system that allows multiple instances of multiple processes to run on multiple nodes over multiple paths. While OPUS was developed to support the telemetry processing for the HST instruments, it is a generic pipeline system, and is not tied to any particular processing environment, or to any particular mission. From this point of view the OPUS platform does not provide the mission specific applications themselves. Instead OPUS provides a fully distributed pipeline processing environment structured to help organize the applications, monitor the processing and control what is going on in the pipeline.

The basic architecture of the OPUS system was described in an earlier paper (Rose 1994). Based on a blackboard model where processes do not communicate with one another, but simply read and write to a common "blackboard", OPUS has succeeded in decoupling a large, highly integrated system. Instead of having a single "controller" process which must be continuously aware of what other processes are active, what observations are being analyzed, and what the next analysis step must be, OPUS decentralizes that control allowing each individual process to determine what to do next. This simple understanding allows us to move from a procedural, linear, controlled, sequential processing system to one that is distributed, event-driven, parallel, extensible and robust.

In the OPUS implementation the blackboard is simply a directory on a commonly accessible disk. In a cluster of workstations and larger machines, if the protections are set appropriately, any process can "see" any file in the blackboard directory: the "posting" of blackboard messages consists of either creating or renaming an empty file in that directory. Monitoring is done with two Motif applications: the Process Manager (PMG), and the Observation Manager (OMG). The first gives you the tools you need to control the individual tasks in the pipeline. The second gives you a quick view of the status of datasets in the pipeline and allows you to control individual datasets.

But more than monitoring, the OPUS platform makes it easy to distribute processing over multiple nodes, running several instances of the same task either on a single machine, or on several machines. OPUS is designed to be used for a variety of purposes simultaneously. A production OPUS pipeline can be established, for example, while running a reprocessing OPUS pipeline at the same time.

The OPUS platform is now being distributed on CD-ROM to NASA or ESA supported projects. Originally designed in a VMS environment, the OPUS system has been successfully ported to UNIX environments. The binaries are built and tested on all environments, and the Solaris and Linux system versions are distributed on the CD-ROM. Visit www.stsci.edu/opus/ or contact opushelp@stsci.edu for more information.

4.2 OPUS Support Libraries

The software libraries which support both the OPUS platform and the applications which run in the pipeline are the result of several years of development, testing and operation at the STScI and elsewhere. There are three categories of support software which form the foundation of OPUS and its applications across all operating systems. Utility packages such as string manipulation and time packages are designed to work generically across the various operating systems. Packages that specifically support the blackboard architecture and the operation of the two Motif managers are unique to the OPUS environment. And application packages such as the keyword package, the Open Space Telescope Database (OSTDB) and the Flexible Image Transport System (FITS++) which are designed to support pipeline applications directly.

These packages provide a true resource for NASA and ESA projects. In a cost-conscious era where the software development cycle can and should be better controlled, the OPUS platform and the OPUS software libraries can be reused, forming the basis for the rapid development of robust applications.

4.3 OPUS Applications

While the OPUS platform can support many different types of pipelines, the first applications developed were the HST instrument-specific telemetry pipelines at the STScI. In that environment OPUS has managed the large volumes of production data produced by nine different instruments ranging from the astrometry data produced by the Fine Guidance Sensors (FGS) to the high volume time tagged photon data produced by the Space Telescope Imaging Spectrograph (STIS). The volume of data from the HST has grown over time: the pipeline that used to handle 50 exposures a day now produces nearly 500 exposures a day, and in two years it will face another doubling of the data volume.

The OPUS applications for each instrument focus on interpreting the telemetry stream coming from the spacecraft, extracting critical parameters from the data stream, accessing exposure data contained in the planning database, and reformatting the data in a standard way so that calibration algorithms written by the Instrument Scientists can easily operate on the data.

5. THE FUSE PIPELINE

A telemetry pipeline for the FUSE mission must convert the spacecraft telemetry into usable raw science data in a standard format. That format must include planned information about the exposure, engineering information about the state of the spacecraft and of the science instrument, and information about the telemetry processing itself. It must do this in a robust way, handle missing data, and respond sensibly to unexpected conditions.

A definition of the OPUS portion of the FUSE pipeline is clarified by noting what it doesn't do. Specifically OPUS is not part of the FUSE Spacecraft Control Center (SCC) and is not involved with the commanding of the spacecraft or with the direct data receipt from the spacecraft. OPUS does not play a part in the planning and scheduling of observations although access to that information is critical for the pipeline functions. Nor is OPUS part of the archive and distribution functions for the FUSE mission, although OPUS does feed that archive. Furthermore, while OPUS provides the pipeline shells for the FUSE detector calibration steps, it does not provide the calibration code itself.

The OPUS portion of the FUSE pipeline consists of the following steps:

• Ingest of exposures into the pipeline
• Telemetry validation and data partitioning
• Support Schedule processing
• Telemetry decoding and conversion
• Science data conversion to FITS files
• ..a series of calibration steps..
• Generation of an archive request
• Archive response processing



The claim that the use of OPUS reduces both the cost and the time of development can be supported, in part, by documenting the amount of code reused from the HST pipeline, which will be done in the following sections. While there is a great deal of FUSE-specific processing inherent in the nature of the data, each of these tasks is built upon existing OPUS libraries of functions. This design drastically reduced the amount of new code that had to be developed and tested.

Not only is the use of OPUS beneficial for the FUSE project, but this exercise has proven valuable to the OPUS software as well. HST-specific code in the OPUS libraries was clearly identified, and the libraries considerably generalized. In addition, a few new capabilities were identified and added to the OPUS libraries and are thus available for future project use.

5.1 Ingest

The FUSE pipeline is activated when the Spacecraft Control Center (SCC) begins receiving data from the instrument. When a new exposure is received, the SCC will notify OPUS that processing has begun. OPUS, in turn, will log that message and generate an observation status file (OSF) on the OPUS blackboard for that exposure. The processing status for each exposure will be indicated by that OSF which, in turn, will be displayed on the OPUS Observation Manager's (OMG) graphical display.

When the SCC has completed collection of the data for an exposure, OPUS will be notified, and the OPUS processing of the exposure will begin. The data receipt message is logged to a database, and the OSF for this exposure is modified to trigger the next pipeline task. If the data received is engineering telemetry, the ingest task has the responsibility to determine the base type of the data received, in order to associate the engineering telemetry with the correct science exposure.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
DRF_ingest	8	392	205	96%

As for all tasks in this project, the OPUS blackboard software functions are used directly to manage task start-up. In this case three separate start-up triggers were used: two for the different SCC messages, and another to allow the operator to modify the OSF to send a message from OPUS to the SCC requesting the dataset be sent again. Another major package, the Institute's Open Space Telescope Database (OSTDB) interface to Sybase, was used to log activity to the database. That information is not only used to record when exposures are initially received by the system, but also to determine valid "instance numbers" for exposures received multiple times.

5.2 Validation & Partitioning

The ingest step above deals solely with messages and naming conventions. The validation and partitioning step is the first task to inspect the data is designed to determine whether the file sent to the OPUS pipeline is either valid science or engineering data. What constitutes valid data is, of course, unique to each mode of the instrument so three relatively independent tasks use this step to perform their own series of tests. Separate tasks were designed for the Fine Error Sensor (FES) data, the photon address mode FUV data, and the histogram mode FUV data streams.

In each mode the task validates the information in the file headers and the individual block headers. In the case of FES data the input file is partitioned, separating the science images from the engineering snapshots. The engineering data is put into a format similar to other such data received from the SCC. This enables the downstream processes to handle a single format of engineering data.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
SSF_fuse	16	836	141	89%

Since this process is highly table-driven, most of the modifications to the code stemmed from an effort to remove HST-specific logic from the task. This effort has made the code both more general and more portable to other missions. For example, HST had an historical misuse of the concept of instrument-specific keywords. Removing this complication and simplifying the algorithms is expected to lead to even greater code reuse.

5.4 Telemetry Decoding

One of the challenges in any telemetry processing pipeline is the extraction of information that is encoded in a stream of bits. Similar to the task that extracts text and value data from the MPDB, this component of the OPUS pipeline is highly table-driven. It is essential in any mission to ensure that these tables are defined early and correctly. For example, the bit location and bit size of a telemetry component (e.g. temperature) must be well known early in the planning for the mission. In addition, how those bits are converted into physical units (e.g. degrees) must be established early in the program. This information is stored in the MPDB and drives the decoding of the telemetry.

For the FUSE mission each FES image, and at periodic intervals for each FUV exposure, the ground system will receive engineering snapshot data containing, among other things, the current pointing of the spacecraft as well as the configuration of the instrument. OPUS extracts and converts the telemetry to physical units, and stores this information into the database in a time-series order. Later queries into this database will be invaluable to scientists in their investigation of suspected anomalies.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
DVF_fuse	9	482	205	95%

Insofar as this task is based on the architecture of the telemetry database tables, few changes were required for porting the code to the FUSE mission. However the HST packages which deal directly with telemetry streams proved much too specific to the byte-swapped architecture of the VMS operating system to be successfully adapted. It is still uncertain how general a package can be developed to deal with the unspecified variety of formats produced by spacecraft engineers. Nevertheless the approach now taken in the OPUS system, an approach which is even more table-driven than before, is expected to be easily modified for future missions.

FUSE requires building a time-ordered database of the engineering telemetry, which was not a requirement of the HST mission; this new capability was built into the FUSE pipeline at this stage. It is, in parallel with the other specifications for this task, fully table-driven: all mnemonics, relation names and field names are easily configurable in the database.

5.5 Raw Science Conversion

The Raw Science Conversion step in the OPUS pipeline occurs after Data Validation, Support Schedule processing, and the Telemetry Decoding are complete. Each of those prior steps produce simple text output files which the conversion processes use in developing the final FITS products.

The conversion tasks are the workhorses of the entire pipeline, containing instrument- and mode-specific information imbedded in algorithms. From unscrambling the science stream to decoding photon events to calculating special keyword values from the science information, these tasks adapt much less from the existing instrument-specific code for HST. However the lower-level packages are still relevant: the FITS++ package, the keyword handling, and low-level utilities such as string handling and message reporting are still shared and are available to all applications.

5.5.1 FES Images

The Fine Error Sensors, which are used to aid in pointing and guiding the FUSE spacecraft, are simple imaging devices that allow the scientist to verify the location of the target in the field of view. Prior to each FUV exposure the ground system can receive a number of FES images delineating the field of view. OPUS is required to access the package of FES telemetry and produce a single FITS file for each FUV exposure, with each FES image stored as a separate FITS extension in the file.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
GCF_fes	7	471	198	96%

Four of the seven modules developed for this task are reused in the other FUSE Science Conversion processes. Only three modules are unique to the FES image conversion. Regardless of instrument mode, all components of raw science conversion make considerable use of the existing OPUS code libraries. The keyword package is central to managing the construction of standard FITS keyword entries, and the FITS++ package is used throughout to construct the final files themselves. That package, developed originally for the AIPS++ development effort, is being maintained at the STScI.

 

5.5.2 Photon Event Tables

The FUSE instrument can store the time, the location, and the pulse height of each photon hitting each of its four detector segments. Even for a faint source this can be a prodigious quantity of data. The information for each photon event is packaged in a single word in the data stream. It is the task of OPUS to unpack the telemetry and construct a separate standard FITS binary table containing the photon events from each detector segment.

In any environment in which meaning is attached to individual bits of a telemetry stream, there can be problems of missing or corrupted data. For example, time stamps are intermingled with the photon data; determining when the event occurred depends solely on this imbedded time stamp. The integrity of this occasional bit of information is crucial to the scientific interpretation of the result. Therefore, in addition to the relatively simple task of unpacking the telemetry stream, OPUS is required to verify the integrity of the time stamp, and to take reasonable action when anomalies are discovered.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
GCF_addr	13	686	205	94%

The OPUS team has a good deal of experience with photon event data streams. The Space Telescope Imaging Spectrograph (STIS) has an equivalent event stream and has encountered similar problems with its imbedded time stamps. While none of the STIS-specific code was used directly for the FUSE mission, the algorithms are largely similar. Reuse of all the keyword software packages already found in OPUS, as well as the FITS library, was a critical factor for the timely delivery of this code.

5.5.3 Histogram Images

When sources are too bright for the photon mode to be used, there is an alternative for FUSE: the histogram mode. This mode is effectively an imaging spectrograph that produces a spectral image of the detected photons. Unlike the STIS which has an effective imaging size of 2048 x 2048, the four FUSE detectors each have an imaging size of 16384 x 1024 for a total of 64M pixels.

There is insufficient memory in the onboard computers to save the entire set of histograms, so the standard mode of operation is to save specific regions within each detector and to downlink the information in each of these regions along with a table describing the location of these spectral regions in detector coordinates. The challenge for OPUS is to reconstruct an image from these data and save the images as standard FITS file extensions.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
GCF_hist	21	773	208	90%

As in the other Science Conversion tasks there was major reuse of the blackboard, keyword, and FITS software that is common to many of the OPUS applications. This particular mode required new algorithms and new code in order to assemble spectral images from each detector segment.

5.5.4 Engineering Snapshots

While the Telemetry Decoding step in the pipeline does most of the work in extracting and converting the engineering parameters, an additional step is required to format this information into FITS files to ensure there is complete documentation that accompanies the science exposure. Each engineering snapshot produces thousands of bits of data regarding the state of the spacecraft and the instrument. Not all of this information is useful, or even meaningful, to a scientist with only passing familiarity with the spacecraft. Consequently the FITS files produced at this step in the pipeline contain a very limited subset of these data.

OPUS is required to handle the situation in which there are a number of engineering snapshots associated with a particular exposure. For each snapshot OPUS constructs the FITS keywords and a binary table of engineering data, and packs these as individual extensions in a single FITS file for that exposure.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
GCF_eng	8	392	211	96%

 

Again, reuse of the OPUS blackboard (OPUSBB), OSTDB, keyword, and FITS++ packages was central to the development of this application. FITS files containing multiple extensions of binary tables are nothing new to the OPUS team. Not only are they used extensively in presenting the time-tagged mode science data, but also the HST Observatory Monitoring System uses these tables extensively in observation logs.

 

5.6 Calibration Shells

As mentioned earlier, the OPUS component of the FUSE pipeline does not include the calibration code itself; scientists intimately familiar with the character of the instruments develop that code. Each step in the calibration process is developed independently, and is invoked from a simple shell script requiring a single argument, the name of the dataset, which is provided by the OPUS pipeline. All further information about that dataset, and about the processing which is required, is obtained by the calibration task from keywords in the FITS headers of that dataset.

The calibration steps are linked together by standard OPUS pipeline resource files that specify environment parameters unique to each step. The calibration code for each step communicates back to the OPUS pipeline environment through its exit status code; each exit code is interpreted through the resource file for that step as a trigger for subsequent pipeline actions.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
CAL_fuse	2	88	157	98%

While the calibration tasks themselves involve a substantial amount of new code, the OPUS system code which controls each task start-up is usually a simple script. That shell script is invoked by the standard OPUS "external poller", which determines what to do and when to do it from the pipeline's observation status files (OSF). The external poller is part of the standard OPUS platform distribution and requires a script file, but no code development.

5.7 Assessment & Archiving

After processing is successfully completed for an exposure, it is critical to evaluate whether there were serious problems encountered on-board, or during the OPUS pipeline processing. For example the engineering telemetry has information about whether the guide-star acquisition was successful, and certain keywords contain information about how the calibration routines assessed the quality of the data.

The first job of this task is to determine if processing was successful or not. Culling status information from the various FITS file headers as well as the trailer (logfile) of the exposure, this task will perform a simple evaluation and post the results in the database. If all is well with the exposure, the second job of this task is to notify the archive system that the datasets are ready for archiving.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
ARC_fuse	8	412	183	95%

Built upon simple routines that read FITS headers and ASCII files, the largest challenge in developing this task was determining how to make the evaluation table-driven. Rebuilding the software is clearly a more lengthy and error prone process than just updating a table, and during operations the ability of on-site changes to the system are preferred.

5.8 Post Archive Processing

This task is triggered by messages from the archive system. Those messages indicate either that the data has been successfully archived, or that a problem occurred that the archive system could not correct. In the former case, copies of the datasets that have been archived are no longer needed and can be deleted from the pipeline directories. This process accomplishes other cleanup tasks so that the pipeline can run unattended.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
CLN_fuse	4	216	166	97%

The number of modules required for this process is still an estimate, but since this is an automated OPUS process, the amount of code reuse is expected to be substantial.

6. SUMMARY

The details of the steps in the pipeline, the challenges presented at each stage, and an explicit count of the code reuse have been presented. The summary in the table below is clear: basing the FUSE telemetry pipeline on the existing OPUS system has resulted in clear savings. The extensive reuse of the OPUS system libraries leads to an inexpensive but complete pipeline that has proven operational capabilities.

Task name	FUSE modules	Lines of code	OPUS modules	Percent module reuse
FUSE project	105	5098	268	71%

A total of 105 new modules were developed for the FUSE pipeline. The foundation of over 250 modules in the OPUS libraries allowed development to proceed quickly and with considerable confidence that the underlying functionality is reliable and robust. Each task represented roughly 90% reuse, and the project as a whole shows over 70% reuse of the existing OPUS system.

Reusable software has been the siren song of software engineering and object-oriented design for a decade or more. The development of inexpensive software systems by adapting existing code to new applications is as attractive as it has been elusive. The OPUS telemetry pipeline for the FUSE mission has proven to be a significant exception to that trend.

Taking an existing system that is operational, and will be maintained for many years to come, was a key decision for the FUSE mission. Adding the extensive experience of the OPUS team to the task resulted in the development of a complete telemetry pipeline system within a matter of months.

7. APPENDIX A. EXAMPLE CALL TREE

The reuse of software is a key factor in modern systems engineering. The objective is not only controlling development costs and development time, but also adapting software that has been proven in other environments. Through reuse, we gain a heritage of robustness and reliability.

The statistics presented in the body of this article are derived principally from the DEC Source Code Analyzer (SCA). The output of SCA was further processed to remove duplicate calls, so the count of OPUS modules includes only unique modules, regardless of how many times they were invoked.

The call tree below is a partial demonstration of the extent to which we have been able to reuse the OPUS software. The

FUSE photon address mode conversion task is selected as an example, but any of the tasks would present the same picture. The highest level modules on the left call the modules at the next level of indentation, which call modules at the third level of indentation, and so forth. Thus GCF_addr_main calls OPUSBB_init which, in turn, calls PRSC_init which calls MPARSE_init_c. Thirteen of the modules are specific to the FUSE project, and everything below that level (50 modules in this listing) is derived from existing OPUS and other packages. Presented in this fashion (where duplicates appear), this listing would take four pages (229 calls) if the complete call tree was detailed.

Module name			System	Comment
----------------------------  -------------------------------------------------
GCF_addr_main			FUSE	Driver for address mode conversion
  OPUSBB_init			OPUS	Initialize Process Resources
    STR_lower_c			OPUS	String utility package
    MSG_append_c		OPUS	Message reporting facility
    PRSC_init			OPUS	Initialize Process Resource File (PRSC)
      STR_concat_c		OPUS  	String utility package
      MPARSE_init_c		OPUS	Initialize File parser
      MPARSE_alloc_c		OPUS	Allocate a parser object
      MPARSE_get_next_rec_c	OPUS	Load the next record
      PRSC_substitute_value	OPUS	Modify the PRSC structure
      MPARSE_end		OPUS	Conclude File parsing
    PRSC_get_vali4		OPUS	Get values from the PRSC
    PRSC_get_values		OPUS	Get multiple values from PRSC
    SYS_get_file_extension	OPUS	System utility package
    OSFILE_init			OPUS	Initialize Observation Status File
    EVNT_init			OPUS	Initialize event processing
    OSF_alloc			OPUS	Allocate an OSF structure
  MSGRPT_printf			OPUS	Message reporting facility
  FITS_set_errhandler		XTRN	FITS package error handler
  GCF_init			FUSE	Initialize FUSE Resources
    KW_free			OPUS	Free the keyword structures
    KW_load_cgg1		OPUS	Load the keyword definitions
      OSTDB_query_setup		XTRN	Database query setup
      STDB_bind_string		XTRN	Bind a string for a database query
      STDB_string_param		XTRN	Define a string parameter for query
      STDB_query		XTRN	Define the database query
      STR_concat_c		OPUS	String utility package
      OSTDB_get_exec		XTRN	Execute the database query
      STDB_close_query		XTRN	Close the database query
      MSG_append_c		OPUS	Message reporting facility
    KW_load_cgg4		OPUS	Load the header definitions
    KR_load_rules		OPUS	Load the keyword rules tables
  OPUSBB_poll			OPUS	Poll for work to do
  OPUSBB_get_directory		OPUS	Get path names
  STR_concat_c			OPUS	String utility package
  MSGRPT_open_trailer_c		OPUS	Open the observation trailer (log)
  GCF_init_obs			FUSE	Initialize exposure-specific structs
    KW_reset_values		OPUS	Reset the keyword values
    GCF_load_kw			FUSE	Load the FUSE keyword files
      KW_load_keywords 		OPUS	Load the keyword values
    KR_set_keywords		OPUS	Implement the keyword rules
  GCF_build_table		FUSE	Process the photon data file
    GCF_addr_hdr		FUSE	Process the SCC header information
      GCF_addr_fill		FUSE	Substitute fill for bad data
    GCF_rootname		FUSE	Construct the rootname of the FITS file
    FITS_open_file		XTRN	Open the new FITS file
    GCF_addr_primary		FUSE	Construct the primary headers
      KW_write_keywords 	OPUS	Write the keywords
    GCF_addr_ext		FUSE	Construct the ttag extension headers
      FITS_open_new_table	XTRN	Open a new FITS binary table
      KW_write_keywords 	OPUS	Write the keywords
    GCF_parse_ttag 		FUSE	Read the photon data
      GCF_new_goodtime		FUSE	Construct a new goodtime interval rec
      FITS_write_table		XTRN	Write a FITS binary table record
    GCF_goodtimes		FUSE	Write the goodtime interval table
      FITS_open_new_table	XTRN	Open a new FITS binary table
      KW_write_keywords 	OPUS	Write the keywords
      FITS_write_table		XTRN	Write a FITS binary table record
    FITS_close_file		XTRN	Close the new FITS file
  OPUSBB_update_status		OPUS	Update the Observation Status file
  MSGRPT_close_trailer		OPUS	Close the observation trailer
  MSGRPT_report_all		OPUS	Message reporting system
  OPUSBB_end			OPUS	Wrap up the OPUS interface

8. REFERENCES