STScI logoThe Hubble Deep Field South

NICMOS Data Reduction / Technical Issues

Check the page with warnings and advisories before making any use of the data.

Information is available about the following topics:

NICMOS imaging

Description of data products

1. General

The final reprocessed data (version 1.0) can be retrieved from the directory nicmos/ on this anonymous ftp site. This directory represents our best present combination of the NICMOS images and the STIS images of the NICMOS field as of 23-NOV-1998.

The field was observed with the F110W, F160W and F222M filters of NICMOS Camera 3 (pixel size = 0."2) and in 50CCD (open filter) mode with STIS. The images were processed using a variant of the CALNIC and DIMSUM procedures to remove the electronic pedestal, dark and sky, were registered using new versions of tasks in the STSDAS "dither" and ditherII STSDAS packages (Fruchter, Hook, Busko and Mutchler 1997, Fruchter and Hook 1998), and combined using Drizzle (Fruchter & Hook 1998). Some images which had noise levels substantially larger than predicted (nearly always as a result of persistent cosmic rays in images taken shortly after an SAA passage) were excluded from the combination. The total exposure times in seconds are:

F110W	108539
F160W	128441
F222M	103163
50CCD	 25900
The F222M images were taken during "bright" time, i.e. when the telescope was pointed near the bright limb of the earth. The F110W and F160W were taken in dark time. The 50CCD images, unlike the NICMOS images, were taken outside of CVZ, and thus the telescope was pointed away from the earth limb.

The images have all been combined with North up (the orientation, based on the HST guide stars, has an estimated error of ~0.1 degree) and astrometric zeropoint was set using the position of a star visible in the NICMOS image whose position was determined by the Naval Observatory (with an estimated total error of ~50 mas).

Follow this link for a legend of the file names for the final reprocessed data. In short, the available images are:

f110w_v1.fits			The F110W NICMOS image
f110w_w_v1.fits			"               " weight image

f160w_v1.fits			The F160W NICMOS image
f160w_w_v1.fits			"               " weight image

f222m_v1.fits                   The F222m NICMOS image
f222m_w_v1.fits			"               " weight image

stis_nic_v1.fits		The STIS on NIMCOS 50CCD image
stis_nic_w_v1.fits		"                      " weight image

stis_nic_psf_v1.fits		The convolved STIS on NICMOS image
stis_nic_psf_w_v1.fits		"                          " weight image
In the subdirectory pf1p0/, another version of the F110W, F160W and F222M images can be found. These images were created with a "pixfrac = 1.0" and are primarily of use to those wishing to study sources at the very edges of the images, where the above images begin to "break-up" due to a lack of sufficient input images to cover the output plane at the finer pixfrac scale.

The subdirectory individual/ contains all individual calibrated (un-combined, un-mosaiced) NICMOS exposures as gzipped tar files.

2. Understanding the images

The NICMOS images in this directory were created using the Drizzle parameter values, pixfrac = 0.45, scale = 0.36974. The scale was set to create an output pixel size of 0."075, or exactly three times the scale of the STIS images of 0."025 of both the NICMOS field and the primary quasar field. The pixfrac is a compromise between providing uniform coverage as much of the field as possible and obtaining the highest possible resolution. These goals were, however, fairly well met, as nearly all of the available field has a ratio of r.m.s. to median of the weight image of less than 0.3, and the FWHM of the stars in the J and H band images are about the width of an input (0."2) NIC-3 pixel.

The noise in the images is estimated by the weight map images. The weight maps of individual input images were set equal to the square of the exposure time of the image divided by the expected r.m.s. in counts of the image. This is the inverse of the expected variance of the image. Drizzle divides the input weight up among the output pixels so that the total sum of the pixels in the input image is preserved.

If one measures the r.m.s. in the sky of the drizzled image one will find it does not equal the square root of the inverse of the weight map. There are two factors that must be applied to make these numbers equal:

1) The inverse variance map estimates the noise in a single INPUT pixel, but drizzle divides the flux among scale**2 output pixels. One must therefore divide the estimated r.m.s. by scale**2 (or multiply the weight map by scale**4). This has already been done for the WFPC2 images (as part of creating the PC+WFPC mosaic), but was not done for the STIS or NICMOS images.

2) Drizzling causes adjacent pixels to be correlated. The pixel-to-pixel noise therefore underestimates the true noise of a larger area. This correlation factor is a function of the ratio of (pixfrac/scale). For the NICMOS images this factor is ~1.9, for the STIS image it is ~1.8.

In the case of the NICMOS images then the noise directly estimated from the weight map is about a factor of 14 = 1.9/(0.36*0.36) larger than that measured on small-scales from the images.

In order to obtain the highest signal-to-noise from photometry on a drizzled image one should perform a weighted sum of pixels in the region of interest. The program, weighted block average or wba, found in
will perform a weighted AVERAGE (note not sum) of a rectangular region of a drizzled image and its weight image. Notes on using the program can also be found on this page.

3. PSF Convolution of the STIS Image

In order to allow a more accurate photometric comparison of the STIS and NICMOS images, the STIS image was convolved to the resolution of the NICMOS image and the 0."025 pixels of the STIS image were block-summed to the 0."075 resolution of the NIMCOS image.

The convolution kernel was created using Tiny Tim and the IRAF task "psfmatch". First a NICMOS F160W PSF was created on the STIS 0."025 drizzled STIS pixel scale. This PSF was then divided in fourier space by a gaussian comparable to the core of the STIS PSF. This step was accomplished using psfmatch. The resulting kernel was then rotated to the average orientation of the NICMOS image, and was applied to the STIS image using the STSDAS task fconvolve. The resulting convolved image was then block-summed 3x3 to the NICMOS pixel scale.

Data Reduction

1. Introduction

The NICMOS HDF-South images were processed using a combination of STScI pipeline procedures and customized tasks designed to remove the basic instrumental signatures from the data. Further processing to remove bad pixels and residual artifacts, and final image combination, were carried out using the Drizzle procedure (Fruchter & Hook 1998) and associated software. A detailed description of the methods used will be presented at a later date both on this web page and in various publications, but here we give a brief outline of the steps taken.

The observations were taken in MULTIACCUM mode using the SPARS64 sample sequence. In this operating mode, the array is first reset and then read out nondestructively at various regular time intervals to sample the accumulating signal throughout the integration. All exposures were obtained using NSAMP = 5, 10, 15, 20 or 25 readouts.

2. Basic Image Processing

1) The images were partially processed through the STSDAS 'calnica' pipeline applying the steps ZOFFCORR, ZSIGCORR, MASKCORR, BIASCORR, NOISCALC, DARKCORR, and NLINCORR. Standard STScI reference files from the calibration database were applied during these steps. This processing accomplishes the subtraction of the nominal bias and dark current from the readouts of the MULTIACCUM image. At this stage, then, no cosmic-ray correction, flat-fielding or conversion to counts per second have been performed. However, some components of NICMOS biases and darks are not temporally stable, and are thus not fully removed by the pipeline processing.

2) In particular, NICMOS data are subject to floating bias levels in each readout quadrant which can change during the course of a MULTIACCUM sequence (sometimes known as "pedestal" ). These variable bias levels were dealt with in two stages using a routine we call NBIASFIX. Bias drifts during the course of each individual MULTIACCUM exposure were measured and removed with a customized procedure which also identifies and eliminates the occasional bias jumps or bands (which are particularly common in the last readout of a MULTIACCUM sequence, a.k.a. the "last read anomaly"). The effect of these processing steps is to remove changes in bias levels from readout to readout, leaving data which accumulate in a simple linear fashion with time. However a net (unknown) DC bias level is still present in the data (see step 4 below).

3) The data were then processed through the UNITCORR and CRIDCALC stages of 'calnica.' A slope is fit to the accumulating counts versus time in each pixel. CRIDCALC detects sharp discontinuities in the accumulating counts versus time ramp and identifies these as cosmic ray events. When these occur, the offending readout is eliminated for the pixel in question and the slope is recalculated without it, thus effectively removing the cosmic ray from the image. The output of this processing step is an image representing count rates for each pixel. Note that at this stage in our processing procedure the data have not yet been flatfielded.

4) Step 2 above removes drifts in bias during a readout sequence but not the absolute DC bias level. Sky and bias subtraction were accomplished using the data frames themselves through an iterative procedure. Simple median "sky+bias" images were created from the data taken in each filter grouped by NSAMP (i.e. number of readouts). These were then fit to each image allowing one multiplicative degree of freedom (the DC sky level) and four additive degrees of freedom (the bias levels in each quadrant). The sky and bias subtracted images were flat fielded, coaligned and assembled into mosaics for each bandpass. Objects were identified in the mosaics and masks were created marking their positions in each individual exposure. New sky+bias images were then created in a second iteration, using the scaling information determined in the first pass, and with objects masked during the combination. These second pass sky+bias frames were then scaled again to the individual images following the same procedure as before and subtracted to produce the new sky and bias subtracted exposures, which were then flatfielded. This iterative procedure effectively removes all the additive components (sky plus residual dark and bias) left over after the initial calnica pipeline processing.

5) Error models were created for each image using information about detector characteristics, sky levels, number of readouts and total integration time used for each pixel. The noise level in each image was compared to this model, and the frames were given careful visual inspection. In particular, signs cosmic ray persistence, "monster" cosmic rays, and moving targets (satellite trails, etc.) were noted. A significant number of frames were badly affected by cosmic ray persistence, which occurs in exposures taken after passages through the South Atlantic Anomaly. Frames which were irredeemably ruined by CR persistence were eliminated from the final data set. In general, measured image noise agreed well with the error models. Cases where there was a significant disagreement were tabulated and the relative variance scaling was used to re-weight the images in the final summation (section 12 below).

6) The F222M (K-band) data required special processing. The large background count rate in these data allowed us to identify previously unrecognized shortcomings in the nonlinearity corrections made by the standard pipeline. We constructed a new linearity correction procedure using in-flight calibration data and applied it to the F222M images which significantly improved the quality of the subsequent sky subtraction. Note that this new correction was not applied to the version 1 reductions of the F110W and F160W images; this may be done for a future version 2 release of the HDF-South NICMOS data; however, we expect such a corretion to only affect the brighter stars in these images. Raw F222M images showed a number of ring-like features which we attributed to out-of-focus thermal emission from dust particles lodged on one of the optical surfaces, probably the Field Offset Mirror (FOM). These were found to move several times during the course of the HDF-South observations, evidently at times corresponding to resets of the FOM position. The complete ten-day data stream in F222M was broken into four subsets corresponding to the four intervals where the dust features were found to be stable, and sky subtraction was carried out separately for each of these four intervals.

7) The final sky subtracted images were flatfielded using standard reference files from the calibration pipeline.

3. Image Combination

8) The images were spatially co-registered using cross correlation procedures allowing for rotations between images (necessary because the telescope roll angle changed from exposure to exposure as part of the dithering procedure designed to keep the HDF-S QSO in the STIS spectroscopic apertures); however, the rotation angle used was that provided by the spacecraft telemetry and was not determined independently.

9) Residual bad pixels (including the worst of the cosmic ray persistence features) were identified and masked via an iterative procedure described in the 'dither' and 'ditherII' package tutorial (Fruchter and Mutchler 1998). The resulting bad pixel masks for each image were combined with a custom static mask designed to exclude known and suspected bad pixels in the array, as well as the vignetted region at the bottom edge of the NIC3 field of view and a small unstable region at the upper right.

10) Pixels in the NICMOS detector arrays are not uniformly sensitive across their areas. Therefore undersampled images are subject to intrapixel response effects, wherein the measured counts depend sensitively on the positioning of the source within the pixel. This can have a particularly large effect on point sources observed with Camera 3, causing the measured counts in the peak pixel to vary considerably more than simple shot noise statistics would predict. We found that our procedure designed to identify residual bad pixels and cosmic rays was frequently "zapping" the peaks of bright stars for this reason. The peaks were therefore manually "unzapped" to correct for this effect. The final summed HDF-S NICMOS images should be largely free of photometric uncertainty due to these intrapixel response effects because they are averages over more than one hundred individual exposures with randomly placed sub-pixel centerings.

11) NICMOS images containing relatively bright sources are subject to electronic ghosting artifacts (known as the Mr. Staypuft effect. This phenomenon manifests itself in two ways. First, bright objects can produce "streaks" of elevated signal along image columns. These appear both in the columns where the object lies and also in the corresponding columns in other detector quadrants. Second, faint ghost images of a bright source in one detector quadrant also appear in the other three quadrants at the corresponding pixel positions. Both effects were visible in the HDF-S F110W and F160W images, arising from the small number of brighter stars present in the field. The column ghosts were quite prominent and were handled by fitting and subtracting medians from each column in each exposure. Pixels corresponding to the positions of all known astronomical sources were masked and excluded from the median fitting procedure to ensure that they would not bias the measurement. The fainter ghost images of the four brightest stars were invisible except in the final drizzled image combinations, where they appeared as faint, false "objects." These were eliminated by masking small areas in each exposure around their expected positions in each detector quadrant (i.e. positions 128 pixels away in each axis from the actual location of the responsible stars). These masked regions were set to zero in the weighting maps used to combine the data (see section 10 below). Thanks to the fact that the HDF-S dither pattern involved rotations, the regions which were masked and excluded at one telescope rotation were filled by "good" data from another, ensuring that the ghosts could be eliminated without producing gaps in the final images. However the effective exposure time underneath the regions affected by the Mr. Staypuft ghosts is significantly reduced from the total, and thus the resulting image noise is higher. The output pixel-wise weighting maps from the drizzling process, which are available together with the HDF-S images themselves, show distinctive "holes" which can be used to identify the locations where the ghost masks most severely affect the data.

12) The images were drizzled together using the individual bad pixel masks generated for each exposure combined with the overall static mask (see section 9 above), using spatially dependent weighting maps generated using the known flatfield pattern and noise characteristics of the data (see step 5 above). This procedure ensures optimal weighting by inverse variance during image combination. A new calibration for NICMOS Camera 3 geometric distortion (updating that in Cox et al. 1998) was used during drizzling. NICMOS pixels are slightly rectangular when projected onto the sky, an effect which is also removed during drizzling using the geometric distortion correction. The pixel scale for the final drizzles had a linear extent of 0.36974 original NIC3 pixels, a scale chosen to be exactly 3 times that of the final drizzling scale used for the STIS 50CCD images. The final drizzled pixel scale is therefore 0.075 arcseconds per pixel in the NICMOS images, and 0.025 arcseconds per pixel in the STIS-on-NICMOS image. The "drop size" or "pixfrac" for the NICMOS drizzles was set to 0.45 input pixels. The images in all bandpasses (including the STIS-on-NICMOS data) were drizzled so that they were co-aligned and so that North is up (i.e. in the positive y pixel direction) and East is right (i.e. negative x) to the best of our knowledge of the orientations of the science instrument apertures.

4. STIS image on the Main NICMOS Field

The STIS on NIC image includes 9 orbits of unfiltered STIS images of the NICMOS HDF-S field. Pairs of cosmic-ray split images were obtained at 9 dither positions during two visits. These frames were processed using the same procedures as for the main STIS field. Since the CCD was annealed between the first and second visits, an updated dark frame was created for the version 2 processing. The version 1 processing masked hot pixels from both darks in the post-annealing data. These masks were multiplied by an edge mask to create the final static masks.

The mode of the sky pixels was determined using an iterative rejection algorithm and subtracted from each cosmic-ray split frame. Image shifts were also determined individually via aniterative process of cosmic-ray masking and cross-correlation. The rotation angle was essentially identical for all these frames. The amplifier ringing correction described by Gardner et al. was applied, but the improvement to the sky was negligible for this field. The images were drizzled onto 0.025" pixels, scale = 0.492999, using a drop size of 0.6 pixels. The images were weighted by the variance of the sky, or wt_scl = t ** 2 / (N_sky + R^2 + N_dark). A small rotation of 5.55 degrees was included to rectify the image.

Photometric Zeroponts

There remains some significant uncertainty in the absolute photometric calibration of the NICMOS HDF-South images. The nominal photometric calibration information which is inserted into image headers by the STScI CALNICA data reduction pipeline use the most up-to-date calibration reference files taken from the STScI Data Archive. However, recent information based on revised ground-based photometry of the fundamental NICMOS photometric standards by Persson et al. 1998 (AJ 116, 2475) suggests that calibrations based on these reference files may be incorrect by 5 to 14% for some filters. Moreover some zeropoints also disagree with those used by the NICMOS IDT for their HDF-North observations (Thompson et al. 1998 in press). These discrepancies are presently being investigated

For the purposes of the version 1 data release and catalogs, we have adopted preliminary revised values of the photometric zeropoints for the F110W, F160W and F222M filters kindly provided by D. Calzetti of the STScI NICMOS group. We stress that these are PRELIMINARY values only, but we feel that there is sufficient reason to doubt the nominal pipeline photometric calibration that the use of new values, even preliminary ones, is justified. Here we present these new zeropoint values along with those from the calibration pipeline and those used by Thompson et al. for the HDF-North. In all cases, these are zeropoints in the AB magnitude system, such that

	AB = zeropoint - 2.5 * log(n)
where n is the count rate (ADU per second) measured from the images. All HDF-S images, including those of the NICMOS field have been normalized, to counts per second, i.e. to an effective exposure time of 1 second. Therefore the zeropoints given above may be applied directly to measurements from the images without further rescaling for the exposure times. Actual exposure time varies with position across the image due to the dithering pattern, bad pixel masking, etc. The sum total of the integration times of all images used in the final drizzled mosaics is recorded in the TEXPTIME header keyword in each image.
	Bandpass	Preliminary	Pipeline	Thompson et al.
			Revised		Zeropoint	HDF-North
			Zeropoint			Zeropoint

	F110W		22.75		22.89		22.68
	F160W		22.77		22.85		22.80
	F222M		21.80		21.89		-----
For the STIS-on-NICMOS image in the 50CCD (clear) bandpass, we adopt the nominal zeropoint from the calibration pipeline, identical to the value used for the primary HDF-South STIS field. We have no reason to question this zeropoint value at this time.
        50CCD (STIS)	26.386
Note that the photometry keywords recorded in the headers of the Version 1 HDF-South NICMOS images (e.g. the PHOTFNU keyword) are those provided by the standard STScI calibration pipeline, and thus correspond to the zeropoints in the middle column above. I.e. they have *not* been revised to account for the new, preliminary calibrations described above.

WE EMPHASIZE THAT ABSOLUTE PHOTOMETRY FROM THE HDF-SOUTH NICMOS IMAGES MAY BE UNCERTAIN AT THE 10% LEVEL DUE TO THESE ZEROPOINT UNCERTAINTIES. We will provide updated information on photometric zeropoints on these web pages as it becomes available.

Send us your feedback or questions.
Visit the Hubble Deep Field South main page.
Visit the Space Telescope Science Institute home page.

This page was last updated on March 11, 1999.
Copyright Notice.