STScI Logo

Hubble Space Telescope
WFPC2 Darks - Summary

An instrument science report is being written concerning the changes to the dark reference files. In the meantime, this note summarizes the changes and offers examples of header HISTORY comments, one from a dark before and one from a dark after the Aug. 1, 1996 change.

New dark frames are obtained regularly each week. Generally, it takes a couple weeks to obtain and process a set of these (5 frames are taken each week) into a dark reference file which is then delivered into the pipeline. Hence, many observations may not have the optimal dark and you may want to consider recalibration. For example, data taken in the first week or two after a decontamination will likely have a dark correction applied with a reference file that was generated from dark frames taken before the decontamination; however, many of the hot pixels will have been "repaired" or annealed during the decontamination and won't be present in the science data. On the other hand, in many cases this will not be a limiting factor for your science and you may decide not to recalibrate.

This article was taken from the August 1996 STAN.

Change in Calibration Dark Files:

by S. Baggett, S. Casertano, M. S. Wiggs, M. Mutchler

Starting August 1, we have modified the procedure by which the dark reference files used in the default calibration pipeline are obtained. Previously, such dark files were generated by combining the dark frames - usually 10 - obtained in the week of the observation and in the previous week. The two week time frame was a compromise between noise in the dark frame (which would have been improved by a longer baseline) and timeliness of the information about hot pixels, which change on a daily scale. In the new system, the dark is obtained by combining a superdark, composed of 120 individual darks obtained over the last year, and a weekly dark, usually comprising 5 dark images. If the dark current in the weekly dark differs from the superdark value by more than a certain threshold, the value is taken from the weekly dark: the implication is that the pixel is probably hot, and thus the more recent value is preferred. If the dark current differs by less than the threshold, the more precise value from the superdark is preferred. With this scheme, the noise is minimized for the majority of the pixels where there is no indication of change, while at the same time new hot pixels are tracked properly. The threshold is set at 5 times the 3-sigma-clipped rms dispersion of the pixels in the weekly dark, measured separately in the four chips; the threshold is set at 5 times the 3-sigma-clipped rms dispersion of the pixels in the weekly dark, measured separately in the four chips; the value actually used is reported in a HISTORY comment in the header.

The hot pixels (those whose value comes from the weekly dark) are identified in the associated Data Quality file by setting either the 10th or the 11th least significant bit (values 512 or 1024). The value 1024 is used if the pixel value is close (within 0.003 e/s) to the value from the previous week; such pixels can be considered "fixable", since their dark current has remained stable. The value 512 is used if the dark current differs from the previous week's value by more than 0.003 e/s; such pixels are probably "new" hot pixels, and dark current subtraction is inherently more uncertain, since its value changed some time within the previous week. This use of the values 512 and 1024 is consistent with that of the task for warm pixel correction, 'warmpix', recently made available as part of the latest STSDAS release. In addition, the second least significant bit is set (value of 2) if the pixel value is flagged in the dark frame from which it is taken; this happens, for example, if an insufficient number of the weekly dark frames contained a valid value (multiple CR hits, image residuals).

New method of dark generation, header HISTORY comments from g8l1149cu.r3h.

                                                                    
 This is a DARK created from an average of 4 sets of 30 input          
 darks, plus 1 set of 5 darks which are taken on a weekly basis        
 as part of the ongoing WFPC2 Calibration plan.                        
                                                                       
 The "superdarks" used were:                                           
                                                                       
 g1h1207ku, g1h1207nu, g1h1207qu, g1h1207tu.                           
                                                                       
 The 4 sets of "superdarks" were created by the                        
 Hubble Deep Field team in the following fashion:                      
 the 30 dark frames were calibrated using CALWP2.1307.                 
 They were combined with the following algorithm to eliminate          
 cosmic rays:  The dark calibration image was initially                
 estimated as the median of the 30 input images at                     
 each pixel. An iterative technique was then used to                   
 improve this estimate. Each iteration proceeded by                    
 computing the noise at each pixel using the initial                   
 combined image (assuming total readout, a-to-d con-                   
 version, etc. noise of 12 electrons). Then masks were                 
 generated for each input image with value 1 if the                    
 input image was within 4 sigma of the median image,                   
 and 0 otherwise. Zeros in each mask were then expanded                
 to a 3 x 3 square pattern, so as to also                              
 exclude pixels immediately adjacent to cosmic rays.                   
 The masks were then multiplied into their respective                  
 images, the resulting images were summed, and finally                 
 the result was normalized using the sum of the masks.                 
 The few pixels left without data were filled with the                 
 median of the 30 input images at those pixels. This                   
 procedure was then iterated using the new combined                    
 image as the initial estimate with 4 sigma rejection,                 
 and then additional iterations were made with 3 sigma                 
 and 2 sigma rejection.                                                
                                                                       
 The dark calibration image was normalized to a darktime of            
 1.0 second, using  a darktime value of 1843.60 seconds for            
 each CCD. This value is somewhat different from the DARKTIME          
 value in the current calibration dark headers (1800 seconds).         
                                                                       
 Pixels with fewer than 15 input image pixels were marked              
 in the DQF along with all hot pixels over 0.02 DN/sec.                
 The mean of this resultant weekly dark, excluding values deviating    
 by more than 3*sigma, was saved to a file, with one mean value for    
 each chip. This was done for 3 iterations and the final means and     
 sigmas were used in conjunction with the weekly dark and the          
 superdark, to generate the pipeline dark: each pixel was compared     
 to the chip statistics, to determine whether it exceeded a threshold  
 based on average and sigma.                                           
                                                                       
 That is, if abs value(weekly dark-superdark) > 5*sigma                
 then pixel value of weekly is used in the pipeline dark.              
 If the pixel value of the weekly dark did not exceed the              
 threshold, the pixel value of superdark is used for the pipeline      
 dark.                                                                 
                                                                       
 The values of 5*sigma used to make this dark are (for the PC,         
 WF2, WF3, and WF4, respectively):                                     
                                                                       
 0.001411                                                              
 0.001226                                                              
 0.001317                                                              
 0.001289                                                              
                                                                       
 ATODGAIN 15 darks were created by multiplying the                     
 corresponding ATODGAIN 7 dark by 0.5.                                 
                                                                       
 The 1 set of 5 weekly darks was created in the same fashion,          
 with the following on-orbit darks used:                               
                                                                       
 u2ry3j01t                                                             
 u2ry3k01t                                                             
 u2ry3l01t                                                             
 u2ry3m01t                                                             
 u2ry3n01t                                                             
                                                                       
 (Files created using the STSDAS CRREJ task by                         
 Max Mutchler, 19 August 1996)                                         
                                                                       
 The values in the pipeline dark DQF (*.b3h) reflect whether a pixel   
 value comes from the weekly dark or from the superdark, and whether it
 has changed recently. That is, the following values are used:         
                                                                       
    2 if pixel came from weekly dark and was generated from 3 or less  
      of the 5 input frames for that week                              
                                                                       
  512 if the pixel value comes from the weekly dark, and did vary      
      substantially with respect to the previous week's pipeline dark  
      (an "unfixable" pixel in stsdas task warmpix).                   
                                                                       
 1024 if the pixel value comes from the weekly dark, and did not vary  
      substantially with respect to the value in the previous week     
      (considered a "fixable" pixel in warmpix).                       
                                                                       
 Calwp2 will bit-wise OR any pipeline DARK DQF values with the values  
 from any of the other DQF files used in the calibration processing and
 place results in c1h file.                                            

Previous method of dark generation, header history comments from g7g1427eu.r3h

 This is a DARK created from:                                          

 u2ry8d01t u2ry8e01t u2ry8f01t u2ry8g01t u2ry8h01t                     
 u2ry2701t u2ry2801t u2ry2901t u2ry2a01t u2ry2b01t                     
                                                                       
 The ten dark frames above were calibrated using CALWP2.1308.          
 They were combined with the following algorithm to eliminate          
 cosmic rays:  The dark calibration image was initially                
 estimated as the minimum of the 10 input images at                    
 each pixel. An iterative technique was then used to                   
 improve this estimate. Each iteration proceeded by                    
 computing the noise at each pixel using the initial                   
 combined image (assuming total readout, a-to-d con-                   
 version, etc. noise of 12 electrons). Then masks were                 
 generated for each input image with value 1 if the                    
 input image was within 4 sigma of the minimum image,                  
 and 0 otherwise. Zeros in each mask were then expanded                
 to a 3 x 3 square pattern, so as to also                              
 exclude pixels immediately adjacent to cosmic rays.                   
 The masks were then multiplied into their respective                  
 images, the resulting images were summed, and finally                 
 the result was normalized using the sum of the masks.                 
 The few pixels left without data were filled with the                 
 minimum of the 10 input images at those pixels. This                  
 procedure was then iterated using the new combined                    
 image as the initial estimate with 4 sigma rejection,                 
 and then additional iterations were made with 3 sigma                 
 and 2 sigma rejection.                                                
                                                                       
 The dark calibration image was                                        
 normalized to a darktime of 1.0 second, using darktime                
 values given in the table below.  These values are                    
 somewhat different from the "darktime" value in the                   
 current image headers (1800 seconds).                                 
                                                                      
        CCD      Darktime (sec.)                                       
        ---      ---------------                                       
        PC1          1843.60                                           
        WF2          1843.60                                           
        WF3          1843.60                                           
        WF4          1843.60                                           
                                                                       
 Pixels with fewer than 7 input image pixels were marked               
 in the DQF along with all hot pixels over 36 DN.  This                
 corresponds to a dark current of more than 0.02 DN/sec.               
                                                                       
 ====================================                                  
 For use with images taken at -88 C.             
                                                                       
 Files created using the STSDAS CRREJ task by                          
 Michael S. Wiggs, 15 July, 1996                                       
                                                                       
PEDIGREE= 'INFLIGHT 01/07/1996 - 10/07/1996'

DESCRIP = 'For data taken after 22/06/1996 decon (-88 C),
           10 input darks'

 ATODGAIN 15 darks were created by multiplying the                     
 corresponding ATODGAIN 7 dark by 0.5.