CalCOS v2.17.3 Release Notes

This describes changes from CALCOS version 2.15.6 to 2.17.3.

Relevant Trac Tickets and PRs

Trac	PR	Updates
575	65353	Calcos should have an option to search for the target spectrum.
645	67204	Add a TIMELINE extension to calcos corrtag files.
719	68545	Calcos needs an option to use the default location for the target spectrum.
720	68550	Calcos should accept data taken with different cenwaves.
761	69143	Calcos should flag gain-sagged regions in FUV data.
763	69206	Calcos should check for no rows in G140L lampflash table.
764	69207	Calcos needs a utility function for segment-specific keywords.
786	69401	Calcos should extrapolate the TDS correction.
789	69437	Calcos should expand wildcards for the input file names.
791	69679	Calcos should use the shift file if it's relevant.
793	69730	In calcos, modify some pyfits.open calls.
795	69737	Change the way calcos fixes APERTURE = "RelMvReq".
805	69937	Account for the width of the OI 1304 line in calcos.
806	69938	Inconsistent return value from calcos wavecal function.
818	69977	Calcos should not look for wavecal offsets if wavecorr is not complete.
829	70033	Calcos should check whether a keyword exists before deleting it.
830	70036	Calcos should read the HVTAB and populate keywords.

Overview

• The TIMELINE table in the corrtag file has a new column, SUN_ZD, the angle between HST and the Sun, as seen from the center of the earth. This is approximately equal to the zenith distance of the Sun as seen from HST, except that the parallax of the Sun is not included.
• A new reference table GSAGTAB may be specified. This "gain sag" table is a bad pixel table that flags regions of the FUV detector that have very low gain.
• A new high voltage history table (hvtab) can be read and used to populate keywords HVLEVEL[AB]. The gain sag table now uses these keywords.
• Previously, calcos would accept multiple input file names, but there was a limit due to the line length. The input file or files may now be specified (in quotes) using wildcard characters.
• The syntax for the command-line option --find has been changed; a value should also be specified. This option has to do with finding the target spectrum in the cross-dispersion direction.  The value can be "yes", "no", or a numerical value, a cutoff.  ("true" and "false" are aliases for "yes" and "no" respectively.)  If a cutoff value is given, this means that the spectrum will be extracted at the found location if the error estimate for the location is less than or equal to the cutoff value.
• When finding the cross-dispersion location of a spectrum, the quadratic fit was being done to too few pixels (at least for FUV), resulting in a poor value for the standard deviation of the fit. The number of pixels will now be based on the FWHM, or a minimum of five pixels.
• The command-line option --shift  did not work for the case that the raw file was not tagflash and there was no auto/GO wavecal. The code expected tagflash data, and the shift file would specify which lamp flash was to be overridden. This has been modified so that if a shift file was specified for non-tagflash data, the given shift will be applied.
• The extrapolation of the time-dependent sensitivity factors has been modified. Previously, if the time of observation were before the first or after the last time in the TDS table, the factor would be extrapolated with zero slope.  Now the extrapolation will be done using the slope at the first or last time, as appropriate. Note, however, that the TDS table should cover all observations without extrapolation.
• Changes were made to some pyfits.open statements to allow using memory mapping.
• When computing the count rate for an airglow line for the TIMELINE table, a wider range of wavelengths is now used for the OI 1304 triplet.
• A function used for wavecal processing could return the wrong number of items if the input data contained no counts at all. This could happen for tagflash data, e.g. if the duration of a flash were extremely short. This has been fixed.

Details:

Some of the calls to pyfits.open were modified to change the mode from "readonly" to "copyonwrite" and/or change the memmap flag. The modified files are:  accum.py, average.py, calcos.py, cosutil.py, extract.py, fpavg.py, splittag.py, timetag.py, wavecal.py, x1d.py.
Trac #793, PR 69730

Changes to accum.py:

In the call to cosutil.updateDQArray in doDqicorr, bpixtab was dropped and reffiles was added.
Trac #761, PR 69143

Changes to calcos.py:

shift_file was added to the calling sequences for Observation and its subclasses. This was done to cover the case that a shift file was specified but the exposure has tagflash = NONE. The tagflash flag will be reset to True if the current root name and fpoffset are listed in the shift file (and exptype is not WAVECAL).
Trac #791, PR 69679

Function uniqueInput was modified to expand environment variables and wildcards in the input string.
Trac #789, PR 69437

Command-line options --find and --nofind (which did not have an associated value) were replaced with --find value, where value could be "yes", "no", or a numerical value, a cutoff.  ("true" and "false" are aliases for "yes" and "no" respectively.)  The find_target argument to calcos used to be a boolean flag, and now it's a dictionary containing a boolean flag and the cutoff (if specified).
Trac #719, PR 68545

Function missingRefFiles now checks for a GSAGTAB, if dqicorr is perform. Function mergeKeywords now calls extract.updateGsagComment to concatenate the comments for keyword GSAGTAB for the segment A and segment B files.
Trac #761, PR 69143

Changes to concurrent.py:

Method outFlashSetup was renamed to setUpOutFlash for consistency with other functions with verb/object names.  This was noticed while checking out the problem that led to PR 69938.
Trac #806, PR 69938

processConcurrentWavecal was modified to set numflash to 1 (and to assign dummy values for lamp_on, etc.) if no flash was detected but there is a shift file.  Function dummyFlash was added to do this.
Trac #791, PR 69679

In copySegAtoSegB, the test on whether ofd[1].data is None was moved to an earlier point, and if it is None, the function prints a warning message, sets numflash to zero, and returns without trying to copy the data.
Trac #763, PR 69206

In the call to cosutil.updateDQArray in findShifts, bpixtab was dropped and self.reffiles was added.
Trac #761, PR 69143

Changes to cosutil.py:

Function exptimeKeyword() was deleted, replaced by a new function segmentSpecificKeyword().  This is now also called in doImageStat.
Trac #764, PR 69207

Functions getDQArrays, concatArrays, and findGSagExtn were added. bpixtab was dropped from the calling sequence of updateDQArray, and reffiles was added. Function getTable was modified to allow a two-element tuple as a value in the filter dictionary. In this case the first element is a numpy function (e.g. np.less_equal) and the second is the numerical value with which values in the table will be compared using the specified function. This capability was added in order to select rows of a table based on a column of MJD dates; if the date in a row of the table is less than or equal to the specified value, the row will be selected.
Trac #761, PR 69143

Changes to extract.py:

Function fixApertureKeyword was changed to set APERTURE to just the aperture name (assumed to be "PSA"), excluding the "-FUV" or "-NUV".
Trac #795, PR 69737

Function getTdsFactors was modified to change the extrapolation of the TDS factors. Instead of extrapolating with zero slope if the time of observation is before the first or after the last time in the TDS table, the extrapolation will now be done using the slope at the first or last time.
Trac #786, PR 69401

The find_target argument was boolean flag, and now it's a dictionary containing a boolean flag and the cutoff (if specified), based on the "--find value" command-line argument to calcos. Function checkLocation no longer returns find_target; instead, it updates the "flag" entry in-place. Function extractSegment checks both the flag and cutoff to determine whether or not to use the found location.
Trac #719, PR 68545

The checkLocation function now only returns location and extrsize; the "flag" member in the find_location argument may be modified in-place.

Function segmentSpecificKeyword() is now called instead of exptimeKeyword() to get the exposure time keyword.
Trac #764, PR 69207

In the call to cosutil.updateDQArray in doExtract, bpixtab was dropped and reffiles was added.  Function updateGsagComment was added; this concatenates the comments for keyword GSAGTAB from two input files, one for segment A and the other for segment B.
Trac #761, PR 69143

Changes to fpavg.py:

Function getInputInfo saves cenwave as a list, including all distinct values of cenwave in the input files.  Function createOutput checks the length of that list, an if it's greater than one, keyword cenwave will be deleted from the primary header that's used as a template for creating the output file.
Trac #720, PR 68550

Changes to getinfo.py:

Function segmentSpecificKeyword() is now called instead of exptimeKeyword() to get the exposure time keyword.
Trac #764, PR 69207

Function getRefFileNames was modified to get the name of the gain sag table. Function getGeneralInfo was modified to get the FUV detector high voltage, keyword DETHVLA or DETHVLB.
Trac #761, PR 69143

Changes to splittag.py:

Function segmentSpecificKeyword() is now called instead of exptimeKeyword() to get the exposure time keyword.
Trac #764, PR 69207

Changes to timeline.py:

The AIRGLOW_WAVELENGTHS dictionary was modified to specify a tuple of wavelengths for an airglow line, instead of just a single value. This was needed primarily for OI 1304, which is a triplet with a range corresponding to about 380 pixels for an FUV M grating.  Function findPixelRegion uses the minimum and maximum wavelengths of an airglow line or lines to find the range of pixels on the detector.
Trac #805, PR 69937

Function timelineHDU now defines a new column, SUN_ZD, the angle between HST and the Sun, as seen from the center of the earth. This is approximately equal to the zenith distance of the Sun, as seen from HST, except that the parallax of the Sun is not included. Function computeZD was created; this returns the angle from HST to an object (in particular, the Sun), as seen from the center of the Earth. This is called in createTimeline to populate the new SUN_ZD column.
Trac #645, PR 67204

Changes to timetag.py:

Function getWavecalOffsets was modified; wavecorr was added to the calling sequence, and a test on wavecorr was added to the test on obstype. In doDqicorr, a check on minmax_doppler was added before calling psaWcaBoundary.
Trac #818, PR 69977

Function segmentSpecificKeyword() is now called instead of exptimeKeyword() to get the exposure time keyword.
Trac #764, PR 69207

In doDqicorr, printRef is also called for the GSAGTAB.  cosutil.getDQArrays is now called to get the data quality arrays (combining bpixtab and gsagtab), and the call to cosutil.updateDQArray was modified to include reffiles and remove bpixtab.  The comment for keyword GSAGTAB is replaced by a string that gives the number of the extension in the gain sag table that matches the segment and detector high voltage.
Trac #761, PR 69143

Changes to wavecal.py:

Function ttFindWavecalSpectrum returns a tuple of four items (as of calcos version 2.8.5), but if the input array xi has zero length, ttFindWavecalSpectrum returned only three:  (0., {}, {}).  Now it returns four items, and the two dictionaries (xd_shift and xd_locns) are populated with None and 0 respectively for the current segment or for each stripe. The dictionaries were populated because concurrent.findShifts expects an entry for these dictionaries for each segment or stripe. Function ttFindSpec now checks whether cosutil.centerOfQuadratic returns None for y_temp.
Trac #806, PR 69938

Changes to x1d.py:

In function makeFltCounts, switches["wavecorr"] was added to the argument list for getWavecalOffsets.
Trac #818, PR 69977

The command-line option --find was replaced with --find value, where value could be "yes", "no", or a numerical value, a cutoff.  ("true" and "false" are aliases for "yes" and "no" respectively.)  The find_target argument is now a dictionary containing a boolean flag and the cutoff (if specified).
Trac #719, PR 68545

Function segmentSpecificKeyword() is now called instead of exptimeKeyword() to get the exposure time keyword.
Trac #764, PR 69207

Changes to xd_search.py:

The findPeak function was modified to (sometimes) increase the range over which the quadratic is fit.  For FUV data, the previous value of five pixels was unreasonably small.  The FWHM is now initially determined by starting with the location of the brightest pixel (instead of waiting until after an accurate location had been found), and a quadratic is then fit to either FWHM or 5 pixels, whichever is larger.  The code in findPeak to find the limits of the slice to use for the fit was flawed; it could have resulted in a negative lower limit for the slice.
Trac #575, PR 65353