4.1 NICMOS Dark Current and Bias
Some of the major challenges for achieving high quality NICMOS data reduction arise from difficulties in removing additive components of the instrumental signature that are present in a raw NICMOS image. For the purpose of discussion here, we will divide these additive components into two categories, bias and dark, according to whether or not the signal is noiseless and purely electronic in origin (bias), or noisy and arising from thermal or luminous sources (dark). In practice, the NICMOS bias and dark signals each consist of several different components which exhibit a range of different behaviors.
In the standard reference files used for processing NICMOS data, dark and bias components are combined together as a single DARK image and are handled in the same step (DARKCORR) of the calnica pipeline processing. NICMOS dark images (really dark + bias) are highly dependent on the readout history of the array since it was last reset, and therefore cannot be simply rescaled to the exposure time of the science data (as is done with most conventional CCD data). Each science file must be calibrated with a dark frame of equal exposure time and number of readouts.
Because there is such a large variety of NICMOS MULTIACCUM readout sequences, it was considered to be impractical to obtain and regularly monitor darks in every possible sequence. However, most components of the NICMOS bias and dark current are highly reproducible and can be reliably calibrated. On-orbit darks obtained during SMOV and throughout the lifetime of the instrument have been used to characterize the dependence of the major dark and bias components on pixel position, on time, and on temperature for each of the three NICMOS detectors. This information has been used to construct "synthetic" dark calibration reference files for all MULTIACCUM readout sequences, using as basic data the on-orbit darks obtained during the first part of the Cycle 7 calibration program. These are described in section 4.1.3 below.
Unfortunately, some components of the NICMOS bias have turned out to be unstable or unpredictable, making it difficult or impossible to remove them using the standard reference files. In order to do a good job removing additive dark and bias signatures, it is important to understand their origin and behavior. Here we describe the various components of NICMOS biases and darks in some detail, highlighting their stability or lack thereof, and describing (briefly) how they are incorporated into the standard STScI synthetic dark reference images. In section 4.1.5 below we describe methods and tools for measuring and removing residual dark and bias artifacts from NICMOS images.
4.1.1 Dark Current
Linear Dark Current
The true, thermal dark current is the detector current when no external signal is present. This component grows linearly with integration time:
where D(x,y,t) is the observed signal in a given readout, t is time since reset, and dc(x,y) is the dark current. At the operating temperatures used for NICMOS in Cycle 7, the mean dark current for all three cameras was of order 0.1 e-/sec. It is likely to be higher in Cycle 11 when operating at warmer temperatures with the NCS. The dark current has some two dimensional structure, and is roughly a factor of two higher in the corners than at the center.
Amplifier Glow
Each quadrant of a NICMOS detector has its own readout amplifier, which is situated close to an exterior corner of the detector. When a readout is made, the amplifier emits radiation which is recorded by the detector, an effect known as amplifier glow (figure 4.1). This signal is largest close to the corners of the detector where the amplifiers are situated, and falls off rapidly towards the center of the array. The signal is only present during a readout, but is repeated for each readout (e.g., a MULTIACCUM sequence or an ACCUM with multiple initial and final reads). Typically the extra signal is about 20-30 DN at the corners of the detector and 2-3 DN at the center, for each readout. The signal is highly repeatable, and exactly linearly dependent on the number of reads. The amplifier glow also depends on the length of time for which the amplifiers are switched on, which is slightly shorter for ACCUM mode.
The amplifier glow is a real signal and is subject to photon statistics, so it is a source of noise in NICMOS exposures. In the processing pipeline and calibration reference files, it is considered to be a component of the dark signal, although its physical origin and temporal dependence is quite different than that of the thermal dark current. Thanks to the repeatability of the signal, images calibrated with the appropriate dark frames (same MULTIACCUM sequence or same exposure time for ACCUM images) will have the amplifier glow removed. For the STScI synthetic dark reference files, the amplifier glow is modeled as
where A(x,y) is the cumulative signal due to the glow in a sequence, amp(x,y) is the amplifier glow signal per readout (a function of the pixel location (x,y) and the amp-on time), and NR is the total number of readouts of the array since the last reset. In the corners of a full, 26-readout MULTIACCUM response there will be of order 500-800 DN due to amplifier glow, as well as the associated Poisson noise from this signal. This nominal Poisson noise is propagated into the ERR array of the NICMOS calibrated images by calnica.
Figure 4.1: Amplifier glow images for NICMOS cameras 1, 2 and 3.
4.1.2 Bias, Shading, and "Pedestal"
There are three readily identifiable (but not necessarily physically distinct) components of the NICMOS bias: the detector reset level, shading, and variable quadrant bias or "pedestal."
Bias Reset Level
First, a net DC bias with a large, negative value (of order -25000 ADU) is introduced when the detector is reset. This bias is different in each readout quadrant, but is essentially constant within each quadrant. In standard MULTIACCUM processing, this is removed by subtracting the so-called "zeroth readout'' of the image from all subsequent readouts, e.g. in the ZOFFCORR step of calnica. It is therefore not a component of any calibration reference file, but is removed using the zeroth readout of the science image itself.
Shading
Shading is a noiseless signal gradient, a pixel-dependent bias, which changes in the direction of the pixel clocking during a readout. This bias change is caused by a temperature dependence of the readout amplifier bias. The amplifier temperature changes rapidly during the readout of the array. The result is a bias which changes considerably between the time the first and last pixels of a quadrant are read. Visually, this appears as a ripple and a signal gradient across a given quadrant of an uncorrected image (figure 4.2). The amplitude of the shading can be as large as several hundred electrons across a quadrant in NIC2, with smaller amplitudes in NIC1 and NIC3. The shading exhibits all the characteristics of a bias change, including lack of noise.
The shading signal is not the same for each readout, but depends primarily on the time interval since the last readout (not reset) of a pixel. For each readout in a NICMOS MULTIACCUM sequence, this time interval is recorded in the FITS header of each imset by the keyword DELTATIM. If the time 
t between reads remains constant, the bias level introduced by the shading remains constant, but if t varies (e.g. logarithmically, as in some MULTIACCUM sample sequences), then the bias level changes with each successive read, and thus the overall shading pattern evolves throughout the sequence.
Figure 4.2: Examples of shading for NICMOS cameras 1, 2 and 3 
In addition to the DELTATIM dependence, the shading amplitude and shape also depend on the mean temperature of the detectors, which slowly warmed as the cryogen sublimated over the lifetime of the instrument. Subtle temperature changes during a MULTIACCUM exposure can also lead to shading changes. A sequence with many long DELTATIMEs (such as a SPARS256) can cool between the first and last reads, resulting in a DELTATIME=256s shading that is different in the 25th read than it was in the 4th read. A similar situation occurs in the MIF sequences, when the DELTATIMEs switch over from the relatively long steps in the middle to the rapid reads at the end: the first short read of the set taken at the end of the observation has slightly different shading than the remaining short reads or the initial short reads.
Numerically, shading is of the form:
where the shading s is a function of the pixel location, DELTATIME dt and detector temperature T.
Images of the shading as a function of DELTATIME and temperature can be made by subtracting the amp-glow and linear dark components from DARK observations. These images can then be used to build dark reference files for any sequence given a set of DELTATIMEs and a temperature.
The shading patterns of the NICMOS cameras were measured using data taken early in the Cycle 7 calibration program, and the synthetic dark reference files used in the standard STScI pipeline processing use these shading images (taken with the detectors at ~61.4 K in the fall of 1997). However, as the instrument was monitored over the course of many months, the temperature dependence was discovered. Because the temperature varied during the lifetime of the instrument, so did the shading. This variation is not yet taken into account in the standard pipeline processing done for the HST Archive, and the DARK reference files that have been available from STScI do not currently include temperature dependent shading. Therefore, if a particular observation was taken when the instrument was at a different temperature than that for which the standard DARK reference files were calibrated, then residual errors in shading subtraction may result. The effect is most noticeable for NICMOS Camera 2 because the amplitude of the shading is much larger for that camera. Shading variations are much smaller, and usually negligible, for Cameras 1 and 3. The STScI NICMOS group has developed a fully temperature dependent model for shading in all three NICMOS cameras, and has implemented a WWW-based tool for generating temperature dependent dark reference frames (see "cures," section 4.1.5). As of the time of this writing, the STScI NICMOS group is also working to implement automatic temperature-dependent dark correction in the calnica pipeline. It is quite likely that this will be available by early 2002, and readers should check the STScI NICMOS WWW pages and NICMOS STANs for further updates.
Variable Quadrant Bias or "Pedestal"
In addition to the net quadrant bias introduced at array reset, there is some additional offset which is time-variable and, to some degree, stochastic. This variable quadrant bias has been described as the "pedestal effect" in many discussions of NICMOS data, although we note here that the term "pedestal" has also been applied to other aspects of NICMOS array behavior. The variable quadrant bias is usually constant over a given array quadrant, but different from one quadrant to another. Its amplitude varies from readout to readout, sometimes drifting gradually, but occasionally with sharp changes from one readout to another (not always seen in all quadrants simultaneously).
On 22 August 1997, a modification was made to the NICMOS flight software which reduced but did not eliminate the pedestal effect. Data taken before that date is, in general, severely affected by variable bias levels, and requires careful handling in order to achieve high quality data reductions. However, essentially all NICMOS data, even after the flight software change, are impacted by pedestal to one degree or another.
The variable quadrant bias has two major effects on NICMOS MULTIACCUM data. The first (and generally less important) effect is that the signal in a given pixel, which should normally accumulate linearly with time over the course of an integration (after other sources of bias and dark current are removed, and when intrinsic array non-linearity is corrected), can instead vary irregularly as the bias level in a quadrant changes "underneath" the astronomical signal from source + background. The CRIDCALC step of the calnica pipeline fits a linear ramp (counts vs. time) to the accumulating signal in the MULTIACCUM to derive the source + background count rate, with a rejection procedure designed to eliminate transient cosmic ray events (see section 3.3 and section 4.7). A varying bias level can improperly trigger the CRIDCALC cosmic ray rejection or reduce its sensitivity to real cosmic ray events.
Secondly, the net bias change over the course of the exposure results in an additive offset (different in each quadrant) when the MULTIACCUM sequence is reduced to a single count rate image (the *_cal.fits file) by CRIDCALC. When the image is flatfielded, this undesired, additive offset is then modulated by the flatfield, and appears as an inverse flatfield pattern in the final, reduced data. For illustration, consider an image where the incident astronomical flux (sources plus sky background) is given by S(x,y). This is modulated by the spatially dependent quantum efficiency, or flatfield, Q(x,y). To this is added a quadrant bias offset Bq, which may be different in each quadrant. Here we neglect all other sources of bias and dark current, assuming that they can be adequately removed by standard processing. The recorded raw image is I(x,y):
I(x,y) = S(x,y) × Q(x,y) + Bq
If this image were then divided by the flatfield (or, to follow the STScI pipeline convention, multiplied by the inverse flatfield), the result would be:
I(x,y) * Q-1(x,y) = S(x,y) + Bq × Q-1(x,y).
Thus, the desired image S(x,y) is recovered, but an additive, inverse flatfield pattern is also present, with an amplitude that may be different for each quadrant. These inverse flat patterns, along with discontinuities between quadrants, are the typical hallmarks of a pedestal problem in processed NICMOS data (see example in figure 4.3).
Figure 4.3: Data affected by variable quadrant bias. Left: image processed normally with calnica; note the quadrant intensity offsets, and also the residual flat field pattern imprinted on the data, due to the unremoved bias being multiplied by the inverse flat. Right: image after processing through pedsky.
It is important to note here that a residual flatfielding pattern may also arise from reasons completely unrelated to pedestal. In particular, the NICMOS flat fields have a strong color dependence, and the spectrum of the background (especially at longer wavelengths where thermal emission dominates) does not necessarily match that of the lamps used to create the flat fields. Residual patterns may therefore sometimes result from division by standard internal lamp flats, again especially at longer wavelengths in the medium and broad band filters. We return to this point in section 4.1.5 in the discussion of the pedsky software routine and again in section 4.4.2. Unremoved shading also introduces a bias offset (but a positionally dependent one) which, when multiplied through by the inverse flatfield, will create a pedestal-like effect.
The unpredictable nature of this variable quadrant bias means that it is not possible to remove it with standard reference frames. (In passing, we note that it also considerably complicates the task of generating "clean" calibration reference files of any sort in the first place.) The user must attempt to determine the bias level from the data themselves and subtract it before flatfielding the data. The difficulty, then, is determining the bias level independent of the sky + source signal present in the data. No one method has been developed which does this equally well for all types of NICMOS data. The methods which have been tried depend on the nature of the target being observed, e.g. sparse fields consisting mostly of blank sky are treated differently from images containing large, extended objects or crowded fields. We discuss pedestal removal techniques in section 4.1.5 below.
Bias Jumps or Bands
Occasionally, spatial bias jumps (sometimes called bands) are seen in NICMOS images (figure 4.4). These are apparently caused by a bias change when the amplifiers of one NICMOS camera are being used at the same time as another is reading out. They are very commonly seen in the last readout of a MULTIACCUM sequence, but may occasionally occur in intermediate readouts as well.
Figure 4.4: Bias jumps or "bands" in a NICMOS image. 
4.1.3 Dark Reference Files
Synthetic Darks
The NICMOS dark calibration reference files which are used in the HST Archive pipeline processing are so-called "synthetic" dark images. These include the linear dark current, amplifier glow, and bias shading terms described above. The synthetic darks were constructed using on-orbit measurements of the linear dark current and amplifier glow, plus an empirical shading model based on a fit to on-orbit data.
Random Uncertainties (i.e., noise) in the Synthetic Darks
In the center of the NICMOS arrays, where the effects of shading and amplifier glow are smallest, the uncertainties in the dark reference files are dominated by the readout noise. The older STScI synthetic darks were typically based on an average of about 15 measurements per readout sample per pixel. Therefore the estimated pixel-to-pixel uncertainties in the DARK reference files are of the order of 1 DN (about 5 electrons). In the corners of the arrays the amplifier glow is the largest source of noise, increasing as a function of the number of readouts. For the largest number of readouts (26) the estimated uncertainty is of the order of 5 DN (about 27 electrons). It is important to note that the effect of these "random uncertainties" in the calibration files on science data is not actually "random," however. The pixel-to-pixel noise pattern in the DARK reference files is systematically imprinted on all science images from which they are subtracted. This can introduce a sort of "pattern noise" in the images, which is apparently random but actually affects the pixel-to-pixel statistics of reduced data in a systematic way. In general, this is not a limiting source of noise in NICMOS data, but it can set a limit to the pixel-to-pixel noise achievable with images reduced by calnica using the standard reference files. In the newer, temperature-dependent synthetic dark reference files (now available via a WWW-based tool - see section 4.1.5 below), a much larger number of dark exposures has been averaged to produce the final product, thus reducing this pixel-to-pixel component of the dark frame "noise" to a negligible level.
Systematic Uncertainties in the Synthetic Darks
The dark current pedestal adds some uncertainty to the synthetic darks, since on-orbit dark frames are used to generate the calibration reference files. In essence, the pedestal makes it difficult to establish the absolute DC level of the dark current. However, every effort was made to minimize the effects of the pedestal when making the reference files currently in the database.
Also, during the lifetime of the instrument, no temperature dependence was included in the dark files used in the calibration data base. As discussed elsewhere (see section 4.1.2 and section 4.1.5), this can lead to systematic bias subtraction errors, primarily due to problems with the shading correction.
|
New dark reference files for all cameras have been generated at STScI which implement a temperature-dependent shading correction. These new files also average a larger number of individual frames for the linear dark current measurements, providing better signal-to-noise for the DARK reference files. The new darks are not yet used in standard Archive OTFR, but can be obtained using the new WWW-based temperature-dependent synthetic dark generation tool (see section 4.1.5 below). Users must then reprocess the data locally using these darks. Initial tests show that using these new darks can significantly reduce pixel-to-pixel noise in processed data and improve the shading correction.
|
On Orbit Darks
All dark images taken with NICMOS are available through the HST Archive. Certain MULTIACCUM sequences have extensive collections of on-orbit dark data, particularly those used for the dark monitoring program. Other NICMOS MULTIACCUM sequences, however, have very little on-orbit data. Users wishing to use on-orbit dark data instead of synthetic darks may retrieve darks for the camera of interest with the appropriate MULTIACCUM sequence and the correct number of readouts (NSAMP), process them by subtracting the zeroth readout (ZOFFCORR), and combine them on a readout-by-readout basis with some suitable rejection scheme to eliminate cosmic rays (e.g. median or sigma clipping).
In practice there are several difficulties when doing this. First, on-orbit darks are affected by pedestal effects. Care must be taken when averaging frames, particularly with sigma rejection schemes, since the DC bias level of a given quadrant in a given readout may vary considerably from image to image. The average dark image will still have some mean pedestal value in it. Second, one should be careful to examine all dark frames used for the average, and to discard images which are adversely affected by bright object and SAA-induced persistent signal (see section 4.6.2 below). Finally, because shading is temperature dependent, care should be taken to combine darks taken at the same detector temperature (within approximately 0.1 degrees K) as the observations for which they will be used.
|
Detector temperatures are stored in the NDWTMP11 and NDWTMP13 keywords in the *_spt.fits files. For Cameras 1 and 2, use NDWTMP11; for Camera 3, use NDWTMP13. (These keywords are incorrectly labelled in some NICMOS *_spt.fits headers.)
|
ACCUM and BRIGHTOBJ Mode Darks
In principle, ACCUM mode allows the user to specify any of a large number (173, to be precise) of possible exposure times, ranging from 0.57 to 3600 seconds, and either 1 or 9 initial and final readouts (NREAD). As was discussed above, the various components of the DARK reference files (e.g., bias shading, linear dark current, and amplifier glow) depend not only on the integration time, but on the number of readouts and the readout delta time intervals. Therefore each and every combination of ACCUM exposure time and NREAD requires a unique dark image for calibration, and it was not practical to calibrate all of these on orbit. In addition, as has also been noted, the shading (particularly for NIC2) also depends on the instrument temperature.
At the present time, there are no "standard" DARK calibration reference files available from the HST Calibration Database for use with ACCUM mode data. The DARK reference files used for processing ACCUM mode images in the OPUS pipeline were dummies. In principle it should be possible to create synthetic ACCUM dark reference files using a procedure similar to that which has been used for MULTIACCUM data. In the future, STScI may make a tool available for generating synthetic ACCUM mode darks, but the analysis needed to do this has not yet been carried out. Many individual, on-orbit ACCUM mode dark exposures are available from the Archive, and it is not unlikely that for any given ACCUM mode science exposure there will be darks available with the right exposure time (if not necessarily the right temperature). If you need to calibrate ACCUM mode science images, you should search the Archive to see if suitable darks are available, or discuss the matter with your Contact Scientist.
In general, BRIGHTOBJ mode exposures are so short that true linear dark current is negligible. Moreover, by design they are generally used only for very bright targets, and most sources of dark and bias are probably relatively unimportant compared to the object signal. The behavior of the detector bias in BRIGHTOBJ mode has not been systematically characterized at STScI, and no DARK reference frames are available for this mode.
4.1.4 What is Removed by Standard Pipeline Processing
The DC bias or reset level is removed from NICMOS images by subtracting the zeroth readout from subsequent readouts. For MULTIACCUM data, this is accomplished by calnica with the ZOFFCORR step. For ACCUM data, the zero read subtraction is performed automatically on board the telescope.
The DARKCORR step of calnica subtracts a dark reference image, specified by the DARKFILE header keyword, from the science exposure. For MULTIACCUM data, this is done for each sample readout (i.e., imset), matching the appropriate imsets of the DARK reference file to those of the science image. The DARKFILE includes all three additive components described above, i.e., true dark current, amplifier glow, and bias shading.
In general, the DARKCORR step of calnica, using standard STScI reference files, should do an adequate job of removing linear dark current, amplifier glow, and most shading.
4.1.5 Cures: How To Get Rid of What's Left
Several components of the bias may not be adequately removed by the standard pipeline dark correction. In particular, residual shading in NIC2 data, bias drifts and jumps, and the net pedestal may still be present after standard processing. Here we describe ways of handling each of these.
Residual Shading
As described above, the bias shading function is temperature dependent, and may not be completely removed from data using the standard synthetic dark reference files. This is particularly true for Camera 2 data, where the shading amplitude is largest. The STScI NICMOS group has now numerically modeled the temperature dependence of the shading and has made a temperature-dependent dark generating tool available via the WWW. This program that can generate synthetic darks for any NICMOS MULTIACCUM sequence and at any temperature within the NICMOS operating range.
|
When reprocessing NIC2 images, the use of temperature-dependent darks generated with the STScI WWW tool will often significantly improve the quality of the reduced data, minimizing both shading residuals and pixel-to-pixel noise for non-background limited data (i.e., almost all NIC2 images taken through filters at wavelengths shorter than 1.8 microns). You may extract the appropriate detector temperature information from the NDWTMP11 and NDWTMP13 keywords in the *_spt.fits files. Use NDWTMP11 for NIC1 or NIC2, and NDWTMP13 for NIC3.
|
Variable Quadrant Bias or "Pedestal"
Variable quadrant bias or "pedestal" is not removed by standard processing. STScI has distributed several STSDAS tools for removing this variable bias level. Here we briefly describe the tools that are currently available in the stsdas.hst_calib.nicmos package.
biaseq
The biaseq task in the stsdas.hst_calib.nicmos package is designed to remove the changes in quadrant bias level from readout to readout during the course of a MULTIACCUM exposure. It adjusts the bias levels in each NICMOS quadrant so that the net counts in that quadrant increase linearly with time. This "bias equalization" procedure does not remove the net bias offset that produces the pedestal effect. Essentially, it removes any temporally non-linear components of the bias drift, i.e., the second and higher order time derivatives of the bias, but leaves an unknown linear term in the bias drift (i.e., the first time derivative). The program cannot distinguish between this linear bias drift and an actual, linearly accumulating astronomical signal, and thus leaves the linear bias term in the data so that it must be removed by some other method (see "pedsky" and "pedsub" below). In principle, biaseq will work on any NICMOS MULTIACCUM image, regardless of the nature of the astronomical sources present, provided that there are enough MULTIACCUM samples available. As a by-product, biaseq can also attempt to identify and remove bias jumps or bands (see section 4.1.2 above) from individual readouts.
The biaseq task must be run on an intermediate image file (*_ima.fits) which is produced by partially processing a raw NICMOS data set through only the first few processing steps of calnica. The pipeline processing steps BIASCORR, ZOFFCORR, ZSIGCORR, MASKCORR, NOISCALC, NLINCORR, DARKCORR and BARSCORR should be performed before running biaseq, but not FLATCORR, UNITCORR or CRIDCALC, i.e., the image should not be flatfielded and should be in units of counts (not counts per second). The nicpipe task in the stsdas.hst_calib.nicmos package provides a convenient way to carry out the partial calnica processing needed as preparation for biaseq (see example below, and also section 5.1).
The biaseq task assumes that the astronomical signal (sky plus sources) accumulates linearly with time, and that any non-linear behavior is due to changing bias levels that are constant within each array quadrant, except perhaps for bias jumps. If these assumptions are not correct, then the task may not work properly. E.g. if the sky background is changing with time, either because it is dominated by variable thermal emission or because of scattered earthlight (see section 4.8), then the routine may not function correctly. The nicmos task pstats may be used to compute and graph data statistics versus time or readout number, which can help to identify time-varying background levels. Objects which saturate the NICMOS array will also no longer accumulate signal linearly with time. However, in this case, unless a large fraction of the pixels in a quadrant are saturated, there should be no noticeable effect on biaseq. Also, residual shading may result in a bias which changes from readout to readout but is not constant across the quadrant, and this may also cause problems for biaseq. In particular, residual shading can improperly trigger the bias jump finding algorithm. If there is residual shading in the images, it should be removed with a temperature-dependent shading correction (see Residual Shading) before running biaseq. If biaseq is run without doing this, the user should at least disable the bias jump finding option. The biaseq help pages give further information about this task and its parameters.
pedsky
For NICMOS images of relatively blank fields, free of very bright or large sources which fill a substantial portion of the field of view, the pedsky task may be used to measure and remove an estimate of the sky background and quadrant-dependent residual bias (or "pedestal"). The task depends on having a large fraction of the image filled by "blank" sky, and thus may not work well for images of large, extended objects or very crowded fields.
Pedsky runs on a single science image (i.e. not on all the separate readouts of a NICMOS MULTIACCUM file). It operates only on the [SCI,1] extension, which is appropriate when the task is run on, e.g., the *_cal.fits images that are the final product of the calnica calibration pipeline. The task's internal algorithms operate on an unflatfielded image, but a fully-calibrated (including flatfielding) image may be used as input, as the task will check the status of the flatfielding (via the value of the FLATDONE keyword in the input image header) and will temporarily remove and, at the end of processing, reapply the flatfield if necessary.1 Note, however, that the FLATFILE used for the processing must be available locally in order to run pedsky. Therefore if you wish to use this task on reduced data taken from the HST Archive, be sure to retrieve the appropriate flatfield reference file as well.
Following the discussion in section 4.1.2 above, let us say that a NICMOS image I(x,y) may be described as
I(x,y) = S(x,y) × Q(x,y) + Bq
where S(x,y) is the incident astronomical flux (sources plus sky background), Q(x,y) is the flatfield, and Bq is the quadrant-dependent bias offset. The pedsky task works by minimizing the quantity
X2 = 
xy (Ixy - S × Qxy - Bq)2
which is a measure of the total image variance, as a function of the sky level S and four quadrant bias levels Bq. Here we have made the simplifying assumption that the "true" incident flux S(x,y) in a relatively blank-field image can be approximated as a constant sky background S, i.e. that there are no sources present. In real data where there are real sources, the quantity X2 includes a contribution due to the presence of actual objects in the image, above the assumed uniform, constant sky level S. The impact of these sources is minimized, however, by using sigma-clipped statistics which exclude pixels with strongly deviant values, e.g. those due to actual astronomical sources, bad pixels, etc. Additionally, the user may apply a ring median filter to the image when computing X2. This can effectively remove compact or point-like sources, and may help the task perform better for moderately crowded fields, but considerably slows the computation speed. The user may wish to experiment by trying pedsky both with and without the ring median option. In practice, a certain amount of "source noise" contribution to X2 is tolerable to the pedsky algorithm. It acts as an offset to the amplitude of X2, but generally has no effect on the location of the minimum value for X2 relative to S or Bq.
Pedsky works in both interactive and non interactive modes. Alternatively, the user can supply a sky value to be subtracted, in which case the remaining quadrant-dependent pedestal is estimated and subtracted. After pedsky processing, the remaining standard calibration steps, including flatfielding, can then be easily applied using another call to the script nicpipe.
Note that, in principle, any image may be used to represent the spatial structure of the sky, i.e., you do not need to use a standard NICMOS flatfield reference file. In particular, for some NICMOS images the two-dimensional structure of the sky may not exactly resemble that of the flatfield. This may happen for at least two reasons. First, as will be discussed below (section 4.4.2), the NICMOS flatfields are strongly color dependent, and the spectrum of the internal flatfield lamps does not necessarily match that of the sky background, especially at longer wavelengths (> 1.8 µm) where thermal emission begins to dominate over the zodiacal sky. The result is that the spatial structure of the sky may not be quite the same as that of the flatfield, and that the sky multiplied by the inverse flat reference file may have some residual structure which correlates with the flatfield pattern and contributes to the total image variance X2 measured by pedsky. This can confuse pedsky, resulting in improper sky and pedestal measurement. This problem is most important for images dominated by thermal background, but may also affect shorter wavelength data, especially for Camera 3 data where the ratio of background counts to quadrant bias offset amplitude is larger than for the other two cameras. Second, at longer wavelengths, the thermal background generated within the telescope may illuminate the detector differently than does the zodiacal sky, and thus the overall background may not match the QE response pattern measured in the flatfield.
In cases like these, you may want to provide your own "sky" images for use by pedsky, rather than rely on the flatfield reference files to represent the shape of the sky. One possibility would be to use sky frames constructed from the median of many dithered science or background exposures. This can sometimes improve the quality of the sky + pedestal fitting even for data taken at shorter wavelengths. This was the approach taken for STScI reductions of the HDF-South NICMOS data, for example. Another possibility, especially for long wavelength data, might be to use a specially-constructed color-dependent flat field (see section 4.4.2). The pedsky help pages give further information about this task and its parameters, including guidance for how to use images other than the flatfield as the sky model.
pedsub
As described above, the pedsky task requires lots of "blank" sky to be effective, and will only work on relatively sparse NICMOS images. The pedsub task provides an alternative method for images which contain larger objects that fill the field of view. The basic methodology for pedsub is essentially the same as that of pedsky, modeling the image as the sum of a constant (per quadrant) pedestal offset plus an astronomical signal (sky + objects) that is modulated by the flatfield, and then loops over a range of trial values for the pedestal signal, searching for the amplitude which minimizes pixel-to-pixel variations. Unlike pedsky, however, pedsub can optionally apply a spatial filtering function to each trial image in order to remove unwanted features or spatial frequencies (i.e., the signal from objects) that might bias the calculation of the pixel value spread. The filtering options are "median" and "mask," which essentially carry out low-pass and high-pass filtering of spatial frequencies. The "mask" option removes all large-scale structure from the trial image, leaving the RMS minimization process to operate only on the small-scale, pixel-to-pixel component of the flatfield signal. This option can be effective when trying to measure and remove pedestal from, e.g., images of large galaxies.
Pedsub has many other parameters and options which are fully described in the STSDAS help pages for the task. These should be consulted carefully before using the task, and the user may wish to experiment with combinations of these parameters in order to achieve the best results.
- Using nicpipe, biaseq and pedsky: An Example.
Here we illustrate the use of the nicpipe, biaseq and pedsky tasks with one example. A similar sequence could be used with pedsub substituted for pedsky. The raw data frame is n4yx23x0q_raw.fits. This is a NIC3 image taken with the SPARS64 readout sequence and NREAD=24. We might start by using the sampinfo task (see section 5.1) to look at the details of the readout sequence.
ni> sampinfo n4yx23x0q_raw.fits
|
This will produce a table of output information that looks like this:
We see that the readouts with linearly spaced DELTATIME values (SAMPNUMs 4 through 24) are in imsets 1 to 21. The last readout (imset 1) often has bias jumps (see Bias Jumps or Bands), so we may want to exclude it when feeding the desired range of sky samples to biaseq for use in constructing the "clean" sky image. So:
ni> nicpipe n4yx23x0q_raw.fits "" stage=biaseq
ni> biaseq n4yx23x0q_ima.fits n4yx23x0q_beq.fits \
>>> skysamps=2-21 fitbias+ fitjump+
|
The output image n4yx23x0q_beq.fits has been corrected for non-linear bias drifts and for spatial bias jumps as well.
Next, we complete the pipeline processing using nicpipe, doing the FLATCORR, UNITCORR, CRIDCALC steps to prepare for pedsky:
ni> nicpipe n4yx23x0q_beq.fits "" stage=final
|
We now run pedsky non-interactively, letting it fit for the sky level and four quadrant biases on its own.
ni> pedsky n4yx23x0q_beq_ima.fits n4yx23x0q_ped.fits \
>>> salgorithm=auto interactive- rmedian-
|
The end product, n4yx23x0q_ped.fits, is the fully processed and pedestal corrected image.
Other Pedestal Removal Software
The STScI NICMOS group will continue to explore methods for measuring and removing bias offsets from NICMOS data. In particular, as of the time of this writing, we are investigating the possibility of automatically implementing the biaseq and pedsub procedures in the calnica pipeline. Users should check the NICMOS web pages and STANs for updates on this effort.
In addition, there are other, "freelance" packages for NICMOS data reduction. One example is Brian McLeod's NICRED package (McLeod 1997, in the proceedings for the 1997 HST Calibration Workshop, ed. S. Casertano et al., p. 281), which offers a general suite of NICMOS data reduction tools, including routines which estimate and subtract pedestal. Ultimately, there are similarities in the pedestal removal methods used by pedsky, pedsub, and the NICRED algorithms, and therefore it should be possible in principle to unify them in a single task.
1 This is a change from earlier versions of the pedsky task and earlier editions of this Handbook. Previously, pedsky required a partially processed image, and would not work on flatfielded data.
