To mitigate the effects of gain sag on the COS/FUV detector (see COS ISR 2011-05) the location of the COS FUV spectra was moved to Lifetime Position 3 (LP3) on Feb 9, 2015 (except for G130M/1055 and G130M/1096 that remain at LP2). The changes in performance expected from this move, including a possible 10% decrease in spectral resolution below values obtained at LP2, were documented in the COS Cycle 22 and 23 Instrument Handbooks and the Cycle 22 HST Primer, and also discussed in STAN articles published in March and June 2014. Due to their very wide cross dispersion profiles, observations using the G130M 1055 and G130M 1096 CENWAVE settings will continue to be done at LP2. All NUV observations remain at the original position on the NUV detector.
Initial test and calibration observations done immediately following the move to LP3 show that the instrument is performing as expected. Note that observers do not need to make any alterations to their Phase 2 proposals to accommodate the lifetime position change as this is handled transparently by the HST ground system and COS flight software.
Figure 1: A comparison of monitoring observations taken at LP2 (red) and LP3 (black), show very similar instrument performance. Profiles of absorption lines here, interstellar C II at 1334/1335Å) remain consistent before and after the move.
With the move to LP3, the new spectral location is offset by -2.5" from the original lifetime position (LP1). This is close enough to LP1 that for many modes the standard boxcar pipeline extraction will reject and discard large wavelength regions due to overlap with previously sagged LP1 regions (i.e. regions where 5% or more of the sensitivity was lost due to the low gain). The current "BOXCAR" algorithm will reject the entire wavelength bin if one of these pixels is included, even if only at the edge of the extraction region. To avoid unnecessarily discarding columns affected by such pixels, a new algorithm, referred to as the "TWOZONE" algorithm, has been implemented that avoids discarding wavelength bins when a gain-sagged region is far enough out in the wings of the profile that it does not have a significant effect on the extracted flux. The new "TWOZONE" algorithm will only be used to calibrate data obtained at LP3. Apart from a few new informational columns in the x1d data files, the formats for all data products are as before. Data obtained at other lifetime positions, including all NUV spectral observations and any new observations at the G130M 1055 and 1096 settings, will still be calibrated by default with the "BOXCAR" algorithm. The new algorithm is however not optimized to calibrate data of extended targets or data obtained with the BOA aperture. The use of this new extraction algorithm is transparent for all users. Below, we provide more detailed information about the new algorithm; an updated version of the COS Data Handbook will be released later in the year.
The publicly released distribution of STScI Science Support Branch Software (SSBREL) is typically only updated twice per year and does not yet include CalCOS 3.0. To obtain the new CalCOS prior to its inclusion in SSBREL it is necessary to install the testing (SSBX) or development version (SSBDEV) of the SSB software. See http://ssb.stsci.edu/ssb_software.shtml for installation instructions.
II. The New Extraction Algorithm
The steps in this new algorithm are as follows:
- The 2D spectral image is straightened to remove residual localized geometric distortion (module: TRCECORR)
- The straightened image is aligned with a reference cross-dispersion profile, applicable to the grating and central wavelength of the exposure (module: ALGNCORR)
- A full extraction region is defined, with a height over which the counts will be summed that varies as a function of wavelength. This region is defined so that it contains 99% (default) of the enclosed energy as measured using a reference profile for that setting. A smaller inner extraction region is also defined, that contains 80% (default) of the enclosed energy. Wavelength bins will be flagged as bad only when pixels contained in this inner zone are marked with a serious data quality flag. (For COS, bad pixels are determined by comparing the bits set in the data quality (DQ) array to the "Serious Data Quality" or SDQFLAG parameter. See section 2.7.2 of the COS DHB for further details).
The reference file containing the parameters controlling the new extraction algorithm is specified in the "TWOZXTAB" keyword. The reference file storing the trace correction vectors used for straightening the spectral image is listed in the TRACETAB keyword, and the file containing the 2D reference profile shapes for each CENWAVE is specified in the PROFTAB keyword.
II.1 Spectral Image Straightening (Trace correction)
Although CalCOS applies a 2D geometric correction to the raw event locations, noticeable distortion remains in the corrected images. In particular, there remain significant small scale distortions in the cross-dispersion (Y) direction that appear to be fixed on the physical detector, and which remain consistent over heights of several tens of pixels. Removing this residual Y distortion results in significantly smoother cross-dispersion profiles that are easier to align with observed data. Using high signal-to-noise observations of each mode, we define the "trace" center location at each wavelength to be the flux weighted center Yc = ∑i(niYi)/∑ini of the incident counts, where ni is the number of counts at each location Yi. Traces for a number of modes are shown in Figure 2. In Figure 3, we can see the effect of applying the trace correction. The gray-scale image in the lower panel of Figure 3 shows the spectral image without the application of the trace correction, while the upper panel of the same figure shows the straightened image.
Figure 2: Spectral trace locations for one CENWAVE setting of each of the COS FUV gratings are shown. Trace locations for the other CENWAVE settings are very similar. The difference between the measured YCORR center as a function of XCORR (blue lines) and the median center (orange line) is tabulated for each CENWAVE, and subtracted from the event locations to straighten the science images. The upper panel also shows the effect of applying the TRCECORR correction to the data.
II.2 Spectral Alignment
To align the trace straightened image with the tabulated reference profile for that setting, both the profile and the observed image are collapsed along the dispersion direction to yield 1D cross dispersion profiles. A simple background subtraction is done, and the flux weighted centers of both the observed image and the reference profile are calculated. Columns contaminated by airglow lines or serious data quality flags are excluded from this collapsed profile. The YFULL positions of the individual events are then shifted by the difference between the centers calculated for the reference profile and for the data to bring the measured centers of the observation and the reference profile into alignment.
II.3 Spectral Extraction
Once the observation and the reference profile are aligned, the reference profile is used to define the regions expected to contain the central 99% and central 80% of the enclosed energy in each wavelength column. The boundaries of these regions are illustrated and labeled in the upper panel of Figure 3. The gross count rate in each column is then just the sum over the region between 0.5% and 99.5% of the enclosed energy. However, a wavelength bin is only rejected when a bad pixel occurs in the inner zone between 10% and 90% of the enclosed energy. Any wavelength bins that are marked as bad in an individual exposure are excluded from the X1DSUM products which combine data from multiple exposures and FP-POS offsets.
Figure 3: The plots above show the spectral images and extraction regions used for the old BOXCAR (bottom panel) and new TWOZONE (top panel) extractions for a G130M 1291 observation of the standard DA star WD 0308-565 using segment FUVB. The grey scale images show the gross count rates in each pixel, with the darker regions corresponding to areas of higher counts. Regions that show significant gain sag (> 5% loss) are marked in red; green boxes mark other serious data quality flags such as dead spots. The dotted lines in the bottom panel show the extraction region used by the BOXCAR algorithm. Any columns for which gain sagged regions (red) or other bad pixels (green) fall between the dotted lines are marked as bad and excluded from the final X1DSUM files, possibly leading to substantial holes in the spectrum even when the bad regions have only a very small effect on the net counts. For the TWOZONE algorithm (upper panel), the counts are included from the region containing 99% of the enclosed (outer blue lines), while gain sagged regions and other bad pixels are only considered when they fall within the inner blue lines marking 80% of the enclosed energy. As illustrated in Fig. 4, this results in far fewer wavelength bins being rejected in the extracted spectrum.
The resulting difference between this new TWOZONE algorithm and the original BOXCAR algorithm is illustrated in Figure 4, which compares the wavelength bins that are rejected by each of the algorithms.
Figure 4: Same as Fig. 3, with yellow regions marking the wavelength bins excluded from the extraction when using the new TWOZONE algorithm (top) and the older BOXCAR algorithm (bottom). The normal recommendation for COS FUV observations is to take a set of four such exposures, each at a different FP-POS offset. This shifts the detector X location at which each wavelength falls, allowing many of the holes to be filled in the combined exposure at slightly reduced signal-to-noise. However, when the bad columns are caused by previous gain sag at LP1 the spacing of the bad columns will often match the FP-POS offset and some gaps in wavelength coverage will remain in the combined product. For this exposure, this would occur when using the BOXCAR extraction, (especially in the region centered near XFULL=5000), but can be avoided with the TWOZONE algorithm.
III. Changes to Output Products
II.1 New columns in the X1D files
Table 1 lists the columns in the x1d output files which are produced for each segment of each individual exposure. The columns listed in red are the new columns produced by CalCOS 3.0 that were not included in earlier versions. These new columns are included to make it easier to understand how the extraction was performed, but they are not used by any subsequent calibration steps. Note that these new columns are included only in the x1d files for individual exposures and are not in the X1DSUM files which combine the results for multiple exposures.
The Y_LOWER_OUTER and Y_UPPER_OUTER vectors show the edges of the extraction regions. All pixels from Y_LOWER_OUTER through Y_UPPER_OUTER are included in the extraction. The Y_LOWER_INNER and Y_UPPER_INNER boundaries similarly mark the area used for the data quality (DQ) flagging, i.e., they correspond to 80% enclosed energy. Overplotting these vectors on the flat-fielded spectral images (FLT files), provides a useful way of verifying the success of the alignment algorithm and illustrating which detector pixels were included in the extraction. Since the sum is done over integer pixels, the actual fraction of the enclosed energy of the profile that is enclosed in the extraction box differs slightly from the nominal 99%. The actual fraction of the profile included in the sum is tabulated in the ACTUAL_EE vector. This enclosed energy fraction is used to correct the final net count rate for differences in the actual enclosed energy fraction.
Because the extraction height will vary from column to column with the TWOZONE algorithm, the total BACKGROUND that is subtracted from the GROSS to derive the NET will also fluctuate with the height of the extraction region and so we also provide the BACKGROUND_PER_ROW that gives the mean detector background that is measured in the defined background regions.
Some of the new information columns are incorrectly populated in the CalCOS 3.0 release; they will be corrected in the next CalCOS release. The calculations for the WAVELENGTH, FLUX, ERROR, GROSS, GCOUNTS, NET, or BACKGROUND columns are not affected. The NUM_EXTRACT_ROWS column should be set to Y_LOWER_OUTER - Y_LOWER_INNER + 1, but in the current version of the code the value is instead set to Y_LOWER_OUTER - Y_LOWER_INNER. These new columns are also included when the BOXCAR extraction algorithm is performed, but currently for data using the BOXCAR algorithm the Y values for the upper and lower boundaries are set to the scalar value at column 0 and do not include the tilt of the region used for the BOXCAR extraction. Also, for the BOXCAR extraction, the BACKGROUND_PER_ROW vector is not correctly populated.
|Column Name||Data Type||Units||Description|
|NELEM||I||-||Number of elements in array|
|NET||E||counts/s||Normalized sum of counts over extraction region|
|BACKGROUND||E||counts/s||Total background subtracted from each column|
|DQ||I||-||DQ combined over inner region|
|DQ_OUTER||I||-||DQ combined over both regions|
|BACKGROUND_PER_ROW||E||counts/s||Background per row|
|NUM_EXTRACT_ROWS||I||pixels||Extracted height in each column|
|ACTUAL_EE||E||counts/s||Normalization factor in each column|
|Y_LOWER_OUTER||E||pixels||Pixel location of boundary|
|Y_UPPER_OUTER||E||pixels||Pixel location of boundary|
|Y_LOWER_INNER||E||pixels||Pixel location of boundary|
|Y_UPPER_INNER||E||pixels||Pixel location of boundary|
IV. Controlling the New Algorithm
IV.1 Keyword Switches
Several keywords are now used to control the extraction algorithm:
- X1DCORR – PERFORM or OMIT
- TRCECORR – PERFORM or OMIT
- ALGNCORR – PERFORM or OMIT
- XTRCTALG – BOXCAR or TWOZONE
When using the old BOXCAR algorithm, the MAST pipeline will set the header keywords TRCECORR and ALGNCORR to "OMIT", X1DCORR to "PERFORM" and XTRCTALG to "BOXCAR". When using the new "TWOZONE" algorithm, X1DCORR, TRCECORR, and ALGNCORR are set to "PERFORM" and XTRCTALG to "TWOZONE". While it is possible to manually set other combinations of these keywords, this is not normally recommended as it may result in an inaccurate calibration with the current reference files. Currently the MAST pipeline will only use the new algorithm for FUV spectra taken at LP3 using the PSA and BOA apertures. Data taken at LP1 or LP2 will continue to use the BOXCAR algorithm as they have in the past.
The offsets calculated by the ALGNCORR step for each detector segment are stored in the SP_OFF_A and SP_OFF_B keywords in the extension header of the x1d file. These keywords give the value subtracted from the Y location of each event during the ALGNCORR step. When manually recalibrating data with CalCOS, a user can force the ALGNCORR step to apply a different shift by setting the keywords SP_SET_A and/or SP_SET_B in the extension header of the input rawtag or corrtag file to the desired offset value.
IV.2 The TWOZXTAB Reference File
The TWOZXTAB reference file specifies a number of the parameters used in the new algorithm, including the area of the detector to search for the source spectrum, and the location and size of the background regions to use. These quantities are similar to values used in the XTRACTAB reference file that controls the BOXCAR extraction. However, in the TWOZXTAB the HEIGHT specifies the size of the initial region to search for centering the spectrum before the zone boundaries described above are specified, while B_SPEC gives the location at which to center the initial search region.
When the ALGNCORR step calculates the flux weighted center of the observed spectrum, it also calculates an estimate of the Poisson uncertainty in that location. When this error estimate becomes large, the calculated centroid is no longer a good estimate of the actual spectral location, possibly because no spectrum is visible in the image. So, when the Poisson error for the centroid location exceeds the quantity YERRMAX, the measured location is not trusted and no shift is applied by the ALGNCORR step. This causes the extraction to be done at a location centered on the unshifted reference profile. Tests using a wide variety of COS FUV data sets suggest that a YERRMAX value of 0.8 pixels does a good job of distinguishing detections from blank images for most cases, and this value is currently used for all modes in the TWOZXTAB reference file.
The TWOZXTAB also specifies how to calculate the boundaries illustrated in the upper panel of Figure 3 that are used for the extraction and data quality flagging by defining the enclosed energy fraction that defines each of these boundaries. These quantities are listed in Table 2. Currently for all modes included in this table, LOWER_OUTER = 0.005 and UPPER_OUTER = 0.995, which includes 99% of the enclosed energy as defined by each wavelength column of the reference profile in the extraction region, while LOWER_INNER = 0.1 and UPPER_INNER = 0.9 so that only bad pixel flags that fall within the central 80% of the enclosed energy are included in the DQ vector. However, in principle different values could be chosen for each GRATING/CENWAVE/SEGMENT combination.
|Column Name||Data Type||Units||Description|
|SEGMENT||CH*4||-||Segment Name FUVA or FUVB|
|APERTURE||CH*4||-||PSA or BOA|
|B_SPEC||D||Pixel||YFULL location of the middle of extraction region|
|HEIGHT||I||Pixel||Full height of extraction region|
|B_BKG1||D||Pixel||Center in YFULL of 1st background region|
|B_BKG2||D||Pixel||Center in YFULL of 2nd background region|
|BHEIGHT||D||Pixel||Full height of the background extraction regions|
|BWIDTH||I||Pixel||Width of boxcar smoothing for background|
|LOWER_OUTER||D||-||enclosed energy fraction defining lower edge of the total extraction zone|
|UPPER_OUTER||D||-||enclosed energy fraction defining upper edge of the DQ combination zone|
|LOWER_INNTER||D||-||enclosed energy fraction defining upper edge of the DQ combination zone|
|UPPER_INNTER||D||-||enclosed energy fraction defining upper edge of the DQ combination zone|
|YERRMAX||D||Pixel||Maximum acceptable error for flux weighted centroid position|
V. Known Bugs and Other Remaining Issues
In addition to the incorrect population of some of the new informational columns described above, a few remaining issues with the improved extraction algorithm in CalCOS 3.0 have been identified. None of these issues compromises the calibration of data at LP3. Fixes for these issues are currently being tested, and we anticipate releasing an updated version of CalCOS later in 2015.
The current CalCOS version does not properly shift the flagging of the bad pixel regions for the shifts calculated by the TRCECORR and ALGNCORR steps. To ensure that this bug does not cause bad wavelength columns in the extracted spectrum to be erroneously flagged as good, the bad pixel map in the LP3 region has been adjusted to be more conservative in the flagging of bad regions by expanding the vertical extent of these bad regions by 6 pixels in the region of the detector used for LP3.
Other minor issues include incorrect format specifications for some columns in the output fits files, and a bug that prevents successfully running CalCOS if any of the enclosed energy fractions specified in the TWOZXTAB are set to either 0 or 1.