The APT MSA Planning Tool Rewrite ProjectR. Hawkins (rhawkins[at]stsci.edu), G. Curtis (curtis[at]stsci.edu), A. Henry (ahenry[at]stsci.edu), D. Karakla (dkarakla[at]stsci.edu), A. Myers (amyers[at]stsci.edu), and E. Wislowski (ewislowski[at]stsci.edu)
Planning multi-object spectroscopic (MOS) mode observations with the James Webb Space Telescopes (JWST) Near-Infrared Spectrograph (NIRSpec) requires configuring the Micro-Shutter Assembly (MSA) to optimize observing efficiency. Generating these configurations had been supported in the prototype MSA Planning Tool (MPT), which was developed by STScI's Astronomer's Proposal Tools (APT) and NIRSpec instrument teams. Due to the protype nature of the tool, users had to deal with performance and usability issues. The JWST launch delay was recognized as a unique opportunity to rewrite the tool, to streamline the code and lay out an architecture that would support future development, while simultaneously addressing user feedback on performance and usability. We discuss how teams joined forces to transform the prototype into a tool which is easier to maintain and meets the functionality of the prototype, while surpassing it in power and performance.
History of MPT development
The NIRSpec Instrument for JWST will feature astronomy's first space-based MOS capability. Central to MOS observing mode is the MSA, with nearly a quarter-million micro-shutters that can be individually opened or closed to create highly flexible slit configurations, capturing spectra of many sources in a single observation. The MPT was first prototyped in APT in late 2008, based upon an IDL prototype coded by Klaus Pontoppidan with the help of other NIRSpec team members. This prototype and its subsequent iterations eschewed the standard tool framework of APT, and suffered from performance, usability, and maintainability issues. Expanded development of this prototype began in 2010. Early versions of the tool were available in APT by late 2012, and a feature-complete version of the tool was unveiled at the 2014 AAS Winter Meeting.
The planning of MOS observations requires a complex process of aligning the fixed MSA grid to maximize the number of high-priority observed sources, while ensuring that the spectra from those sources will not overlap on the detector. These constraints become even more difficult for dithered observations that seek to observe the same sources at each dither point, making the manual planning of optimal observations infeasible in all but the simplest cases.
The prototype tool was located on a tab in the Observation Folders in APT, and contained three sub-tabs (Catalogs, Planner, and Plans, as seen in Figure 1), as opposed to being implemented as a proper tool in APT's toolbar, like the Visit Planner, Aladin, etc. The user would import a catalog, configure and run the MPT Planner to produce Plans with optimal pointings, a set of observed sources at each pointing, and their associated MSA configurations. In support of the MPT, APT has a MSA Configuration editor that allowed users to manually modify configurations, or less typically, to create them from scratch.
Motivation for rewrite
Initial MPT development used a design which allowed for very fast prototyping, but required that the user interface and data model be tightly coupled. This was easiest to achieve in the APT architecture by implementing MPT within the data model layer directly and into the existing APT UI, rather than implementing it as a discrete APT Tool, as shown above. This tied the MPT more tightly to the JWST Data Model than desirable, resulted in a tool that was needlessly complex to maintain, and that constrained user workflows in ways that made new, useful workflows difficult to support.
Because the MPT was originally developed as a prototype, there was a significant amount of rework in the codebase that needed to be resolved. Additionally, performance and memory usage in the tool tended to be poor, and performance of the manual configuration editing and visualization tool frequently made it unusable.
Collecting and prioritizing work
In March of 2018, the JWST Launch Delay was announced shortly before the deadline for the initial Call for Proposals. STScI staff quickly initiated a "Feedback Salon" in the STScI lobbies to collect feedback from local users who had been using our tools. This feedback highlighted several issues with MPT (and other systems). Around the same time, community feedback was also collected in a user survey distributed by the JWST Mission Office. The NIRSpec team collected this information, together with a body of existing known issues, giving us a catalog of 90 tickets in our tracking system related to performance, cleanup, and improvements for the MPT.
It became clear that we could have an opportunity to do a rewrite project, so three position papers on requirements, motivations, and priorities for rewriting the MPT were written by the APT, AURA and ESA NIRSpec teams. These papers highlighted motivations and supported prioritization for the workload for the rewrite. Additionally, the APT Team paper outlined a process plan for the rewrite that the joint APT and NIRSpec teams would use for breaking up the work. The MPT rewrite would partially be a large refactoring (improving the implementation architecture of the existing functionality without changing its behavior) and partially a redesign to improve the interface and add new functionality.
Planning the rewrite
After the JWST Mission Office made the decision to approve the project in early 2019, an initial design for the project was presented. This design included a project plan, in which we broke the project into four phases:
Phase 0: Identify new or improved MPT workflows
For this phase, our goal was to identify architecturally significant new features by studying the three position papers to tease out new workflows, as well as improvements to existing workflows. We developed 31 use cases which helped guide this effort.
Phase 1: Rewrite planner internals
The goal of this phase was to rewrite the MPT planner internals to cleanly support existing features. Some components of this phase were:
- Replace existing planner algorithm and configuration handling code with better implementation
- Design new implementation with input from Phase 0 in mind
- Migrate code that can be reused into relevant new packages
Phase 2: Move the MPT into an APT Tool
The goal of this phase was to have all the old functionality in an APT tool with a new storage format for MPT output products.
- Design and implement Tool interface between Observations and the core MPT, migrate to new package.
- Design and implement new storage format, implement converter using old file storage code if possible. Migrate code to new package.
Phase 3: New User Interface
The goal of this phase was to implement an improved UI for the new Tool.
- Identify interface elements that should be in a tool and elements that should be available on the Observation (may overlap).
Bringing the work together
The APT Team had transitioned to scrum-based development methodology shortly prior to beginning the MPT rewrite project. The team had settled on three-week sprints of focused development, and estimated that we were likely be able to complete the development of the MPT Rewrite in four focused sprints.
As the MPT rewrite project unfolded, Phases 0 and 1 ended up intertwined: the MPT core development team began work on Phase 1 coding while, in parallel, the developers and members of the NIRSpec Instrument Team held a series of long and intensive meetings in February and March of 2019 to complete review of workflows. These use-case reviews, though in parallel, were able to focus the coding effort. The effort to separate the internals of the existing MPT (libraries, algorithms, models, etc.) from the existing tool and observation template concluded shortly after the use-case review process finished.
The work involved in Phases 2 and 3 was then split into four core projects:
- Making MPT a proper APT Tool;
- Redefining catalogs as targets;
- Rewriting the configuration editor/viewer for performance; and
- Manual planning in the MOS Observation template.
These projects for rewriting the tool and the interfaces were implemented over the summer and fall of 2019.
As the pieces came together, bringing the project towards closure, we partitioned the remaining work into three primary bins: First, a group of interface/usability items we wanted in place in time to prepare for the Master Class training classes in November; Second, the remaining items that were critical to have in the official release of the rewritten MPT; and finally, the remaining items that were lower priority. We focused on getting the first category done in a development push for an early November beta release for the Master Class trainers. Then, in our final phase, we closed out the second category, augmented with several items motivated by feedback from the Master Class participants. The final version of the project was released with APT 2020.1.1 in mid-January of 2020, to support the re-issued Call for Proposals.
Feedback on rewrite
From the beginning, the NIRSpec team has consistently worked with the APT team to provide prototypes, requirements and usability testing. It was no different during the rewrite, but the feedback loop was much tighter, with the NIRSpec team often using nightly builds to give the development team feedback on progress. Sprint reviews were held in the early fall 2019 to demonstrate and receive feedback on the progress of the new tool. A beta version of the new tool was released for the teams running the Master Class which both allowed the Master Class trainees to get up to speed with the new MPT that they would be using in their own workshops, and gave the developers an excellent opportunity to get hands-on feedback from novice users trying out the tool for the first time. There remain many ideas for additional usability improvements. Feedback of the Cycle 1 proposers will help inform the prioritization of remaining work efforts.
Results and Lessons Learned
Memory utilization, planning speed and load times have all been significantly improved. We had documented significantly long times in one of the tickets that motivated the rewrite. During testing of that ticket, we documented improved load times. Documented improvements depend on the proposal and machine performance, but comparison testing showed from 5–100 times faster loading times. Planning, using the rewritten core planner is significantly faster in most cases, as well.
Memory utilization, particularly in the configuration editor, is dramatically reduced—allowing for multiple editors to be opened without the dramatic performance and memory problems of the old MPT—which had been a significant issue facing nearly all users. Many workflows have been streamlined, and are more intuitive and discoverable.
Master Class trainees provided very positive feedback on the "new" tool, which was very welcome. They are now "carrying the torch" to train other MOS proposers across the globe.
As previously mentioned, we thought we would be able to complete the project in as little as three months, but the timespan of the project itself was over 10 months. There were a variety of reasons for this:
- Focus: While the development team was able to devote a large chunk of our resources on the rewrite, we were, at all times, required to split our force to maintain other efforts: HST support, critical JWST support, etc. In addition, a major project to update the APT Timing model to reflect what was discovered in testing on the hardware unexpectedly coincided with the tool development timeframe, reducing resources by about 20% from initial planning.
- Estimation: Due to the complexity of the work entailed in this project, we underestimated the resources required of the APT team. We realized this as the project unfolded, and were able to fine tune our estimation capabilities.
- Cross Training: One of the most positive aspects of this undertaking is the training it afforded to the entire team working in concert on different aspects of a such a large project. The developers, testers, and documentation specialists, all gained a far greater understanding of the MPT, and demonstrated that we could work efficiently in parallel. Over the span of development of the original MPT, there had never been more than cursory understanding of the tool by the APT developers, outside of one to two team members. The training they gained was a valuable result, but came at a cost (in time) for the development of the tool.
- Swarming: As we described previously, in Phase 2/3, we split the project into lcore sub-projects. We parceled those chunks out to individual developers, and each developer worked in relative isolation on their respective chunk of the project. In hindsight, we think that it might have been more valuable—and efficient—to "swarm" together as a team on each chunk.
The NIRSpec and APT teams joined together to collaboratively improve the NIRSpec MSA Planning Tool for MOS spectroscopy on JWST. We successfully evolved the tool from a working prototype that had significant performance issues into a proper APT Tool which has improved performance across the board, and made the tool significantly easier for proposers to understand and use. Given the success we achieved with this approach, we believe that we should consider using this model for future projects. A demonstration of the MPT is included in the training video "NIRSpec Multi-object Spectroscopy Overview and Tool Demo."
Ferruit, P., & Rawle, T., "Inputs for the upgrade of the NIRSpec MOS observation preparation tool (MPT)"
JWST User Documentation: NIRSpec – MPT Planner
Karakla, D. et al. 2014, Proc. SPIE 9149, "The NIRSpec MSA Planning Tool for multi-object spectroscopy with JWST"
Karakla, D. et al. 2014, "Introducing the NIRSpec Planning Tool for Multi-Object Spectroscopy"
Karakla, D., Henry, A., & Beck, T., "NIRSpec team goals for an MPT re-write"
Myers, A., Curtis, G., & Hawkins, R., "Rewriting the MSA Planning Tool"