An instrument science report is being written concerning the changes to the dark reference files. In the meantime, this note summarizes the changes and offers examples of header HISTORY comments, one from a dark before and one from a dark after the Aug. 1, 1996 change.
New dark frames are obtained regularly each week. Generally, it takes a couple weeks to obtain and process a set of these (5 frames are taken each week) into a dark reference file which is then delivered into the pipeline. Hence, many observations may not have the optimal dark and you may want to consider recalibration. For example, data taken in the first week or two after a decontamination will likely have a dark correction applied with a reference file that was generated from dark frames taken before the decontamination; however, many of the hot pixels will have been "repaired" or annealed during the decontamination and won't be present in the science data. On the other hand, in many cases this will not be a limiting factor for your science and you may decide not to recalibrate.
This article was taken from the August 1996 STAN.
Change in Calibration Dark Files:
by S. Baggett, S. Casertano, M. S. Wiggs, M. Mutchler
Starting August 1, we have modified the procedure by which the dark reference files used in the default calibration pipeline are obtained. Previously, such dark files were generated by combining the dark frames - usually 10 - obtained in the week of the observation and in the previous week. The two week time frame was a compromise between noise in the dark frame (which would have been improved by a longer baseline) and timeliness of the information about hot pixels, which change on a daily scale. In the new system, the dark is obtained by combining a superdark, composed of 120 individual darks obtained over the last year, and a weekly dark, usually comprising 5 dark images. If the dark current in the weekly dark differs from the superdark value by more than a certain threshold, the value is taken from the weekly dark: the implication is that the pixel is probably hot, and thus the more recent value is preferred. If the dark current differs by less than the threshold, the more precise value from the superdark is preferred. With this scheme, the noise is minimized for the majority of the pixels where there is no indication of change, while at the same time new hot pixels are tracked properly. The threshold is set at 5 times the 3-sigma-clipped rms dispersion of the pixels in the weekly dark, measured separately in the four chips; the threshold is set at 5 times the 3-sigma-clipped rms dispersion of the pixels in the weekly dark, measured separately in the four chips; the value actually used is reported in a HISTORY comment in the header.
The hot pixels (those whose value comes from the weekly dark) are identified in the associated Data Quality file by setting either the 10th or the 11th least significant bit (values 512 or 1024). The value 1024 is used if the pixel value is close (within 0.003 e/s) to the value from the previous week; such pixels can be considered "fixable", since their dark current has remained stable. The value 512 is used if the dark current differs from the previous week's value by more than 0.003 e/s; such pixels are probably "new" hot pixels, and dark current subtraction is inherently more uncertain, since its value changed some time within the previous week. This use of the values 512 and 1024 is consistent with that of the task for warm pixel correction, 'warmpix', recently made available as part of the latest STSDAS release. In addition, the second least significant bit is set (value of 2) if the pixel value is flagged in the dark frame from which it is taken; this happens, for example, if an insufficient number of the weekly dark frames contained a valid value (multiple CR hits, image residuals).
New method of dark generation, header HISTORY comments from g8l1149cu.r3h.
This is a DARK created from an average of 4 sets of 30 input
darks, plus 1 set of 5 darks which are taken on a weekly basis
as part of the ongoing WFPC2 Calibration plan.
The "superdarks" used were:
g1h1207ku, g1h1207nu, g1h1207qu, g1h1207tu.
The 4 sets of "superdarks" were created by the
Hubble Deep Field team in the following fashion:
the 30 dark frames were calibrated using CALWP2.1307.
They were combined with the following algorithm to eliminate
cosmic rays: The dark calibration image was initially
estimated as the median of the 30 input images at
each pixel. An iterative technique was then used to
improve this estimate. Each iteration proceeded by
computing the noise at each pixel using the initial
combined image (assuming total readout, a-to-d con-
version, etc. noise of 12 electrons). Then masks were
generated for each input image with value 1 if the
input image was within 4 sigma of the median image,
and 0 otherwise. Zeros in each mask were then expanded
to a 3 x 3 square pattern, so as to also
exclude pixels immediately adjacent to cosmic rays.
The masks were then multiplied into their respective
images, the resulting images were summed, and finally
the result was normalized using the sum of the masks.
The few pixels left without data were filled with the
median of the 30 input images at those pixels. This
procedure was then iterated using the new combined
image as the initial estimate with 4 sigma rejection,
and then additional iterations were made with 3 sigma
and 2 sigma rejection.
The dark calibration image was normalized to a darktime of
1.0 second, using a darktime value of 1843.60 seconds for
each CCD. This value is somewhat different from the DARKTIME
value in the current calibration dark headers (1800 seconds).
Pixels with fewer than 15 input image pixels were marked
in the DQF along with all hot pixels over 0.02 DN/sec.
The mean of this resultant weekly dark, excluding values deviating
by more than 3*sigma, was saved to a file, with one mean value for
each chip. This was done for 3 iterations and the final means and
sigmas were used in conjunction with the weekly dark and the
superdark, to generate the pipeline dark: each pixel was compared
to the chip statistics, to determine whether it exceeded a threshold
based on average and sigma.
That is, if abs value(weekly dark-superdark) > 5*sigma
then pixel value of weekly is used in the pipeline dark.
If the pixel value of the weekly dark did not exceed the
threshold, the pixel value of superdark is used for the pipeline
dark.
The values of 5*sigma used to make this dark are (for the PC,
WF2, WF3, and WF4, respectively):
0.001411
0.001226
0.001317
0.001289
ATODGAIN 15 darks were created by multiplying the
corresponding ATODGAIN 7 dark by 0.5.
The 1 set of 5 weekly darks was created in the same fashion,
with the following on-orbit darks used:
u2ry3j01t
u2ry3k01t
u2ry3l01t
u2ry3m01t
u2ry3n01t
(Files created using the STSDAS CRREJ task by
Max Mutchler, 19 August 1996)
The values in the pipeline dark DQF (*.b3h) reflect whether a pixel
value comes from the weekly dark or from the superdark, and whether it
has changed recently. That is, the following values are used:
2 if pixel came from weekly dark and was generated from 3 or less
of the 5 input frames for that week
512 if the pixel value comes from the weekly dark, and did vary
substantially with respect to the previous week's pipeline dark
(an "unfixable" pixel in stsdas task warmpix).
1024 if the pixel value comes from the weekly dark, and did not vary
substantially with respect to the value in the previous week
(considered a "fixable" pixel in warmpix).
Calwp2 will bit-wise OR any pipeline DARK DQF values with the values
from any of the other DQF files used in the calibration processing and
place results in c1h file.
Previous method of dark generation, header history comments from g7g1427eu.r3h
This is a DARK created from:
u2ry8d01t u2ry8e01t u2ry8f01t u2ry8g01t u2ry8h01t
u2ry2701t u2ry2801t u2ry2901t u2ry2a01t u2ry2b01t
The ten dark frames above were calibrated using CALWP2.1308.
They were combined with the following algorithm to eliminate
cosmic rays: The dark calibration image was initially
estimated as the minimum of the 10 input images at
each pixel. An iterative technique was then used to
improve this estimate. Each iteration proceeded by
computing the noise at each pixel using the initial
combined image (assuming total readout, a-to-d con-
version, etc. noise of 12 electrons). Then masks were
generated for each input image with value 1 if the
input image was within 4 sigma of the minimum image,
and 0 otherwise. Zeros in each mask were then expanded
to a 3 x 3 square pattern, so as to also
exclude pixels immediately adjacent to cosmic rays.
The masks were then multiplied into their respective
images, the resulting images were summed, and finally
the result was normalized using the sum of the masks.
The few pixels left without data were filled with the
minimum of the 10 input images at those pixels. This
procedure was then iterated using the new combined
image as the initial estimate with 4 sigma rejection,
and then additional iterations were made with 3 sigma
and 2 sigma rejection.
The dark calibration image was
normalized to a darktime of 1.0 second, using darktime
values given in the table below. These values are
somewhat different from the "darktime" value in the
current image headers (1800 seconds).
CCD Darktime (sec.)
--- ---------------
PC1 1843.60
WF2 1843.60
WF3 1843.60
WF4 1843.60
Pixels with fewer than 7 input image pixels were marked
in the DQF along with all hot pixels over 36 DN. This
corresponds to a dark current of more than 0.02 DN/sec.
====================================
For use with images taken at -88 C.
Files created using the STSDAS CRREJ task by
Michael S. Wiggs, 15 July, 1996
PEDIGREE= 'INFLIGHT 01/07/1996 - 10/07/1996'
DESCRIP = 'For data taken after 22/06/1996 decon (-88 C),
10 input darks'
ATODGAIN 15 darks were created by multiplying the
corresponding ATODGAIN 7 dark by 0.5.