FOS FLATS FROM SUPER SPECTRA D. Lindler, R. Bohlin, G. Hartig, and C. Keyes FOS Instrument Science Report CAL/FOS-088 March 29, 1993 ABSTRACT: A new algorithm for computing the FOS detector non-uniformities (flat field response) uses spectra of stars taken in 9 different positions in the 4.3 arcsec aperture. Despite some "ringing" at frequencies corresponding to the constant offset of the star in the dispersion direction, the new blue flat fields achieve our accuracy goal of about 1% and are appropriate for FOS BLUE side spectra taken in any aperture in 1992. Ringing at a higher level of 1-2 percent in the RED data is caused by the loss of two-thirds of the G191B2B data due to observing problems. These RED side flats should be used only for blemish location or for visual verification of potential features in observed spectra. Future flat field data should be collected with a newly recommended set of locations of the star in the FOS 4.3 arcsec aperture. INTRODUCTION: In order to derive the FOS flat field corrections for small scale instrumental artifacts, these pixel-to-pixel variations in the photocathode and diode responses must be distinguished from spectral features in the calibration source. The strengths of the spectral lines in the stars are not known at FOS resolutions of 1-10 Angstroms. Better FOS flat field calibrations require determinations of small scale variations in the intrinsic stellar flux distribution. A new technique uses high S/N observations of stars to simultaneously compute both the flat field and a precise stellar "super spectrum." Flat fields at other epochs can be derived simply from the ratio of the observed count rate to the super spectrum of the same star. The new technique that was executed in proposal 2821 requires stellar observations at nine locations in the FOS 4.3 arcsec aperture (Figure 1). Moving the star in the X (dispersion) direction moves spectral features to a different photocathode and diode position in the observed data. Detector granularity features will not move, as illustrated by the spectra in Figure 2, which shows a portion of the G191B2B (WD0501+527) data observed at aperture positions 1 (solid), 2 (short dashes), and 3 (long dashes). The slew step size in the X-direction (1.79 arcsec) results in a shift of 4.75 diodes (19 quarter stepped data points with an uncertainty of 1 point) in the spectral features between data points 1800 and 1870. Most of the structure in the data does not move and can be attributed to the Page 2 detector non-uniformities. To separate the spectral flux distribution from the FOS flat field granularity we create a non-linear system of equations: Let: C(i) be the unknown count rate spectrum. F(i) be the unknown flat field vector. We observe the star at three different X locations in the 4.3 arcsec aperture to obtain observations O1(i), O2(i) and O3(i). The spectral features are shifted by k data points between observations. These observations lead to the system of equations: O1(i) = F(i) * C(i) i=1,n O2(i) = F(i) * C(i+k) i=1,n O3(i) = F(i) * C(i+2k) i=1,n In our case, n is 2064 and k is 19. This leads to a system of 3n (6192) equations with only 2n + 2k (4166) unknowns. By moving the star in the vertical direction in the 4.3 arcsec aperture and observing at the photocathode location of the upper and lower position of the FOS paired entrance apertures, the number of unknowns increases by 4128 (two additional granularity vectors). The stellar flux distribution remains the same and increases our system to 9n (18,576) equations with only 8294 unknowns. If a second star is observed the number of equations is doubled and the number of unknowns is only increased by n+2k (2102). The second star will have a new flux vector but will share the same flat field granularity vectors. An iterative method for solving this large system of equations is described in a later section. OBSERVED DATA: The FOS was used to observe two stars, BD+28D4211 and WD0501+527 (G191B2B) at the nine locations in the 4.3 arcsec aperture shown in Figure 1. Complete sets of data were obtained for the BLUE detector in disperser modes H13, H19, H27, H40, L15 and the prism during the period of March through July 1992 (Proposal 2821). For the RED detector, a complete set of data was collected only for BD+28D4211 (Proposal 2821, June 1992). Data were collected for only the 3 UPPER aperture locations with WD0501+527 (February 1992). Observed RED detector disperser modes included: H19, H27, H40, H57, L15, L65, and the prism. The RED detector data for dispersers H19, H27, and L15 were supplemented by seven sets of observations taken under the flat field monitoring proposal 3975 during the period of January through June 1992. Proposal 3975 observations contain data only for the central position in the 4.3 arcsec aperture. All observations analyzed were quarter stepped (XSTEPS=4) with OVERSCAN=5. Page 3 FLAT FIELD COMPUTATION: The following procedure was used to compute the flat field vector and average count rate spectrum for each detector/disperser mode. Initially, spectra taken at positions 1,3,4,6,7, and 9 (Figure 1) are assumed to have a spectral offset of 19 data points (4.75 diodes) from the center positions (2, 5, and 8). X offsets between the three center positions are set to 0. 1) Partially reduce all observed data using CALFOS (the standard FOS pipeline reduction routine). The reduction consists of conversion to count rates, correction for paired-pulse non-linearities, and correction for geomagnetic image motion. 2) Initialize the flat field vectors for the three aperture positions (UPPER, LOWER, and SINGLE) to 1.0. Repeat steps 3 through 8 for 15 iterations. 15 iterations are chosen, because the change in the flat fields from 10 to 15 iterations is on the order of 0.1%. In one test case, the average change from 15 to 100 iterations is less than 0.2% with most of that difference being low frequency variations. 3) Correct all observed count rate spectra using the present flat field vectors (initially set to all 1.0's in step 2 for the first iteration). 4) Shift to align wavelengths and average all flat fielded spectra for each star. 5) Compute a normalization vector for each spectrum by fitting a cubic polynomial (as a function of data point) to each observation divided by the average stellar spectrum computed in step 4. This is required because of the loss of light resulting from YBASE errors and the loss of light in the wings of the point spread function when the target is not centered in the 4.3 arcsec aperture. 6) Divide each of the raw observed count rates (not flat fielded) by the corresponding normalization vector from step 5. 7) Divide each normalized observed count rate spectrum from step 6 by the appropriately shifted average stellar spectrum from step 4. The results are the computed flat field for each observation. 8) Average the results from step 7 at each of the three aperture positions (UPPER, LOWER, and SINGLE) to give three new flat field vectors. Repeat the iteration begining at step 3 with the new estimate of the flat field response. 9) Remove low frequency variations in the computed flat field vectors by dividing by a least squares cubic spline fit (typically 13 nodes for 2064 data points) through the data. Page 4 Once this process has been completed, we have computed flat field vectors for each of the three aperture positions (UPPER, LOWER, and SINGLE) and an average flux rate distribution for the stars. New spectral shifts (initially assumed to be 19) are computed by cross correlation of the spectrum at the center of the aperture (position 5 in Figure 1) with the other eight positions in the 4.3 arcsec aperture after correcting by the new estimate of the flat field vectors. The shifts are rounded to the nearest integer. With these new, more precise offsets between the spectra, the entire process (beginning at step 2) is repeated. This process is repeated until no changes in the shifts are required. We have included in the above algorithm the ability to use three types of data quality masks. The first type is used to flag data as unusable (eg. telemetry errors). The second type of mask is used to flag data that should be used in the computation of the flat field vector but not in the computation of the flux vector. This mask is used to flag large blemishes that change significantly from one observation to another even though the data was taken with the star in approximately the same Y-position in the aperture. Flat field variations can result from small changes in the position of the spectra due to target acquisition errors, jitter, and temporal or thermal motion. The third type of mask is used to flag data that should be used in the computation of the flux vector but not in the computation of the granularity vector. This type is used to flag sharp changes in the flux distribution that may cause flat field artifacts when alignment is not perfect (spectra are aligned only to the nearest data point or 1/4 diode). Mask type 3 is also used to flag spectral features that have a large contribution from the sky spectra and can vary from one observation to the next. RESULTS: Figures 3 to 28 show the average stellar count rate spectra for each detector/disperser mode observed. The estimated one sigma uncertainty (percent) in the mean is plotted below each spectrum. This uncertainty is computed from the scatter of the computed count rate vectors C(i) of the individual observations from the average. Figures 29 to 67 show the average flat field response for each aperture position (lower, single, upper) along with the one sigma scatter about the average. The scatter about the mean includes both counting statistic errors and actual changes in the granularity among the observations. The counting statistics is the lesser of the two causes of scatter. Regions with abnormally large sigmas indicate blemishes which change significantly from one observation to the next. A 1-2 percent "ringing" is seen in some of the flux distributions and is particularly evident in the region below 2400 Angstroms in the RED H27 BD+28D4211 spectrum (Figure 17). Compare this region to the BLUE H27 data in Figure 8. The major contribution to the ringing occurs at frequencies of approximately 19.0 and 9.5 pixels and can be explained by the sampling method used to observe the data. Data were collected with the star shifted in the aperture such that the spectral Page 5 features move 19 pixels to the left or right of the positions observed in the X-center of the aperture. Any component of the granularity vector and flux vector at the frequency of 19 cannot be properly retrieved and can be visualized by the following two cases. 1) The flat field is uniformly 1.0 except at every 19th pixel which has a value of 0.9. The stellar flux rate is uniformly 1.0 at all wavelengths. 2) The flat field is uniformly 1.0 at all locations. The stellar flux rate is uniformly 1.0 at all wavelengths except at every 19th pixel which contains a value of 0.9. In both cases, the observed data will be identical. It is not possible to determine if the 10% features are in the flat field or the flux array. A better method for collecting data is to vary the step (slew) sizes in the 4.3 arcsec aperture. Figure 68 shows the computed spectrum from a simulation using the 19 pixel pattern. The ringing is clearly evident at the 1 to 3 percent level. Figure 69 shows the simulation in which the spectra were taken with shifts of (17,0,-12) in the upper aperture position, (18,0,-14) in the single aperture position, and (19,0,-8) in the lower aperture position. No ringing is evident in these results. We recommend that these varying steps sizes be used for future data collection. The actual results for the FOS show that the level of ringing varies with detector/disperser modes. Some modes have better results because of small 1 to 2 pixel positioning errors due to target acquisition centering errors and geomagnetic image motion. These 1 to 2 pixel errors result in the spectral offsets varying from 17 to 21 pixels. When larger variations in the offsets are present, the ringing is decreased. The ringing on the BLUE detector modes (which had a full set of data for both stars) appear to be less then 0.5% except for H40 and isolated spectral regions where it approaches 1%. The ringing varies from 1 to 2% for the RED results except for the RED L65 where it is under 0.5%. CONCLUSIONS/RECOMMENDATIONS: We recommend that the results for the RED detector not be implemented in the routine reduction of FOS data because of the level of 19-pixel ringing present within the data. The plots of the RED flat fields included in this report are still valuable to visually determine if an observed feature is at a blemish location in the flat field vector. The BLUE detector flat fields are appropriate for current data observed in the A-1 aperture. The spatial extent of the spherically aberrated telescope point spread function causes the flat field response to vary with aperture size. As the aperture size increases, the spectral profile perpendicular to the dispersion becomes broader. As the profile becomes broader, Page 6 smoothing of the granularity occurs, as illustrated in Figure 70. Data taken in the B-3 (1.0 arcsec circular) aperture and C-2 (0.25 x 2 arcsec) slit show a more pronounced photocathode blemish than the same blemish illuminated by a star in the A-1 (4.3 arcsec) aperture. Figures 71 to 83 show the comparison of the new A-1 flat field vectors with the B-3 aperture flat fields computed using Science Verification (SV) data (Anderson 1992). Ratios of the two sets of flat fields show variations by as much as 5 percent. Some of the variation may be the result of the different aperture sizes. Other possible causes include actual changes in photocathode/diode response and changes in the y-position of the observed data. Changes in the y-position can result from target acquisition errors, filter grating wheel non-repeatability, and temporal or thermal motion. As a check on the stability of the flats over a shorter time interval, the separate BLUE flats are computed for the independent observations of G191B2B in 1992 March and of BD+28D4211 in 1992 July. The typical differences between the flats for these two epochs are less than 1 percent, in agreement with the uncertainties in the average BLUE flats shown in Figures 29-46. However, the differences are occasionally as big as 2 percent. Therefore, any feature in a BLUE side FOS spectrum that is weaker than about 2 percent could be spurious, especially at wavelengths where the flat field correction itself exceeds 5 percent. In order to determine the applicability of the new flats over the time period of January 1991 through October 1992, 70 BLUE spectra are flat fielded with both the new A-1 flats and the old SV B-3 flats. The results are: 1) Calibrations with the new A-1 BLUE flat field vectors are better for both the B-3 and A-1 apertures for data taken in 1992. Even though the A-1 aperture flat field vectors are not perfect for the smaller apertures, applying the A-1 Flat is better than leaving the data uncorrected. 2) Calibrations with the old SV B-3 BLUE flat field vectors are better for 1991 data taken in the B-3 apertures. 3) Calibrations of 1991 A-1 BLUE data show mixed results with the old SV flats giving better results the majority of the time. A few of the observations showed regions where the best results were obtained when no flat field correction was applied. Our recommendation for routine calibration of FOS data is to use the SV B-3 flats for all BLUE side data taken in a SINGLE aperture prior to January 1, 1992. Beginning on January 1, 1992, the new A-1 flats should be used to calibrate all BLUE side data. BLUE side data taken in a paired aperture during 1991 should be corrected with the new A-1 flats computed for the UPPER and LOWER aperture positions or should be left uncorrected. In no circumstances should the SV B-3 flats be used to correct data taken in a paired aperture. A review of Page 7 Figures 29 to 67 clearly shows significant differences in the flat fields between the three aperture locations. The FOS/BLUE A-1 flats described here have been delivered to the Calibration Data Base System (CDBS) and installed in the Post Observation Data Processing System (PODPS) pipeline. Observers wishing to use CALFOS to re-calibrate data may obtain these new flats via anonymous ftp from node stsci.edu in subdirectory /cdbs/yref. Observers with access to the ST ScI VAX Science Cluster can also find the files in the directory, DISK$REFERENCE:[CDBSDATA.REFER.YREF]. Table 1 contains the reference file names for particular instrument configurations. FUTURE CALIBRATION: We recommend that the failed RED side observations of WD0501+527 be repeated as soon as possible. The repeat should be with the slew sizes in the X (dispersion) direction that are specified above. After the deployment of COSTAR, the computed flux vectors for the two targets will no longer be applicable. A complete set of new calibration data should be taken at the nine recommended positions in the 4.3 arcsec aperture for both stars. The flat field response vectors computed for post-COSTAR A-1 data should be applicable to all FOS entrance apertures. The significant suppression of the flux in the wings of the point spread function will make the profile of the spectra perpendicular to the dispersion direction nearly identical for all apertures. This assumption should be verified by comparing the spectra taken with the target in different apertures for at least one grating mode. The time variability of the flat field vectors can then be monitored by observing the star in a single location in the aperture (or 3 locations if monitoring of the upper/lower apertures is required). The flux distributions computed using the 9 position data can be used to reduce all subsequent single position data. In fact, the new flux distributions might be smoothed to match pre-COSTAR data to obtain better pre-COSTAR flats for archival research. REFERENCES: Anderson, S. F. 1992, FOS Spectral Flat Field Calibration (Science Verification Phase Data), Instrument Science Report CAL/FOS-075 Page 8 TABLE 1 FOS/BLUE Flat Field Reference Files Derived from Superflat Technique APERPOS GRATING HEADERFILE DATAFILE single h13 d2f1332py.r1h d2f1332py.r1d single h19 d2f1332sy.r1h d2f1332sy.r1d single h27 d2f1332ty.r1h d2f1332ty.r1d single h40 d2f13331y.r1h d2f13331y.r1d single l15 d2f1332ry.r1h d2f1332ry.r1d single pri d2f13332y.r1h d2f13332y.r1d lower h13 d2f1333dy.r1h d2f1333dy.r1d lower h19 d2f1333gy.r1h d2f1333gy.r1d lower h27 d2f1333iy.r1h d2f1333iy.r1d lower h40 d2f1333jy.r1h d2f1333jy.r1d lower l15 d2f1333fy.r1h d2f1333fy.r1d lower pri d2f1333ly.r1h d2f1333ly.r1d upper h13 d2f13334y.r1h d2f13334y.r1d upper h19 d2f13336y.r1h d2f13336y.r1d upper h27 d2f13338y.r1h d2f13338y.r1d upper h40 d2f1333ay.r1h d2f1333ay.r1d upper l15 d2f13335y.r1h d2f13335y.r1d upper pri d2f1333cy.r1h d2f1333cy.r1d NOTE: APERPOS = single implies apertures A-1, B-1, B-2, B-3, and C-2. APERPOS = lower implies apertures A-2, A-3, A-4, and C-1. APERPOS = upper implies apertures A-2, A-3, A-4, and C-1. Page 9 FIGURE CAPTIONS 1) The nine locations where the stars were observed in the FOS 4.3 arcsec acquisition aperture. The location where the C-1 (1.0 arcsec square) paired aperture would fall is shown by the dashed squares. Locations 1, 2, and 3 correspond to the y-position of the lower aperture of the paired aperture, locations 3, 4, and 5 correspond to the center location of the single apertures, and locations 7, 8, and 9 correspond to the upper aperture Y-position. 2) A portion of three spectra of WD0501+527 (G191B2B) taken with the star at locations 1 (solid line), 2 (dotted line), and 3 (dashed line) in Figure 1. Spectral features move approximately 19 data points between positions. Detector non-uniformities remain at the same location. 3 to 28) The average count rates for the BLUE and RED (=AMBER) spectra of WD0501+527 and BD+28D4211 computed using the method described in this paper. Below each plot is the estimated one sigma uncertainty in the super spectrum. The wavelength range of all plots is limited to the region where the typical flat field correction exceeds its typical statistical uncertainty. 29 to 67) The average detector flat-field granularity vector computed for each disperser mode and each of the three aperture positions (LOWER, SINGLE, and UPPER). The one sigma scatter from the average is plotted beneath each granularity vector. The flat field is set to 1.0 in regions where low counting statistics or sharp changes in the stellar flux distribution give poor flat field calibrations. 68) Results of a simulation of the algorithm described in this paper using the star pattern shown in Figure 1. The simulated spectrum is smooth except for a sharp feature at data point 420. The "ringing" is an artifact of the observation technique. 69) The same simulation as Figure 68 except that the slew sizes between the position of the star in the aperture were changed to (17, 0, -12) data points in the upper aperture position, (18, 0, -14) in the single aperture position, and (19, 0, -8) in the lower aperture position. 70) A flat field feature shown in the spectrum of BD+75D325 observed in three different apertures; the A-1 (4.3 arcsec square) is shown as the thick solid line. The B-3 (1.0 arcsec circular) and the C-2 (0.25 x 2 arcsec slit) spectra are shown as the thin solid line and the dashed line for spectra obtained on June 5, 1991. 71-83) A comparison of the flat field vectors computed with the 1992 A-1 data and algorithm described in this report (NEW, top plots) and the ones computed from SV aperture B-3 data Page 10 (OLD, center plot). The bottom plot shows the ratio of the two vectors. The simularity of the Old and New flats demonstrates the validity of both results, since the derivations are entirely independent. Gaps in the ratio are where either flat field vector is set to 1.0, which is where there is line contamination in the Old flats. The New flats are set to 1.0 in regions where sigma is generally larger than the actual flat field deviation from unity.