Release Notes for STScI_Python 2.2 PyFITS ====== These are the changes that have been implemented in PyFITS since v0.9.6: - PyFITS version updated to 1.0 (from 0.9.6 delivered with the patch release) - PyFITS now uses the standard setup.py installation script - Added interactive convenience functions that allow getting data and headers from a FITS file in one step. This includes: getdata(), getheader(), getval(), writeto(), append(), and update(). See User's manual for more information - Now uses the Python boolean values True/False. The older TRUE/FALSE values continue to work. - Added support for the HEIRARCH convention - Improved formatting when printing cards or card lists. - Support for iteration and slicing for HDU lists - EXTNAMEs automatically converted to upper case - PCOUNT and GCOUNT keywords removed from PrimaryHDUs - Added optional keyword argument "clobber" to all writeto() functions and methods - Header argument is now optional in writeto(), update() and append() functions. When not supplied, a minimal header is constructed that is consistent with the data - ImageHDUs, if supplied to writeto(), is converted to a PrimaryHDU. - The EXTEND keyword is added to the Primary header if not present and extensions are added. - New keywords are appended after the last non-commentary card instead the end of all keywords - Improved error message when indexing HDU lists past the end. - Fixed errors when creating empty headers using the Header() constructor PyRAF ===== These are the changes that have been implemented in PyRAF since v1.1.2: - Version updated to 1.2.0 (from 1.1.2 delivered with the patch release) - The addition of limited support for GOTO statements in CL scripts. GOTOs that jump forward and do not jump to labels within code blocks are now supported. (It should be possible to restructure CL code in almost all cases to use only forward GOTOs). - Multidimensional arrays now permitted as parameters and local variables in CL scripts. Other array-related bugs fixed (e.g., they can be used in epar now.) - Python 2.4 introduced changes that caused problems with the use of INDEF values in places where Python functions expected a float or int value. This has been fixed so such uses can continue. All the built-in CL functions have been modified to handle INDEF values correctly. Comparisons of INDEF values now behave the same as IRAF CL comparisons of INDEFs. - Fixed problem with certain tasks' terminal interactions that caused the task to hang (e.g., with phot) - Fixed bug that caused CL scripts with many consecutive hash characters to hang the Python session. - If Verbose is set, PyRAF now prints a traceback when syntax errors are caught in parameter file parsing. - int parameters now accept (and truncate) float values - Improved error message for CL syntax errors. - "set" has been added to the list of keywords that PyRAF treats differently when used in a way that is different from how the new (as of Python 2.4) Python "set" is used. - Performance of epar has been improved by caching the Tk default root window. This results in much faster startup times on platforms that have slow Tk window creation times. - Changes made to support the simultaneous use of multiple image displays using named pipes (FIFOs). This now works identically to IRAF. - Added changes to support installation on 64-bit Linux (for X11 linking) - Improved handling of Python boolean values. - help works on functions again (broken by Python's addition of function attributes) - Many of the above changes were made to support the Gemini data reduction packages. - Added spaces after the prompts in CL compatibility mode. - the access function (which tests for file existance) now returns True for the special file names "STDIN", "STDOUT", and "STDERR" matching IRAF's behavior. - Various improvements to the CL script cache handling, including the compileall script (which now rename the old cache directory to clcache.old). - Added some more useful scripts to tools: - checkcompileall.py: Read output of compileallcl.py and pull out info on just the scripts that had errors or warnings. - cachesearch.py: Search the CL cache for entries with Python code that matches a particular regular expression. - cachecompare.py: Compare the contents of two CL caches, listing the names of scripts where the Python code differs. numarray ========= These are the primary changes since the last patch release of STScI_Python: - Version updated to 1.4.1 - Speed improvement for numarray operators. The Python level hook mapping numarray operators onto universal functions has been moved down to C. This makes operator notation (a+b) as fast as ufuncs (add(a,b)) for small arrays. - Speed improvement for string-array comparisons, any(), all(). String correlation is ~10x faster. - Better operation with py2exe to help it automatically detect the core numarray extensions to include in an installer. - scipy newcore 'dtype' keyword added to many array creation functions. - scipy newcore .dtypechar attribute attribute added to NumArray class. - ~10 minor bugfixes Full release notes are here: http://sourceforge.net/project/shownotes.php?release_id=366528 iterfile ========= A new IterFitsFile class for each input was added to this module. This new class manages the file I/O internally for the input FITS image, only opening/closing upon each request for data. This insures that only 1 file handle remains open at any one time regardless of the number of objects in memory at any given time. This allows an arbitrarily large number of FITS images to be processed at any one time, subject to memory constraints, when used with the new FileIter iterator class. In addition, the IterFitsFile object can be treated like a PyFITS object, complete with all the same attributes and methods, but with the extra overhead of file open/close for each call. nimageiter ========== A new iterator, FileIter, was added to iterate over IterFitsFile objects. It manages a buffer which gets used to extract 'sections' from FITS objects managed by the IterFitsFile class. The default buffer size used by MultiDrizzle is 1Mb per input image. Numcombine =========== Support was added for computing a clipped minimum array. This is based upon the minimum function in numarray.images.combine that was implemented for numarray 1.3. In the case that numarray 1.3 is not installed, an alternative implementation that does not support clipping will compute a minimum array. Regardless of numarray version, any masks provided will be used. Imagestats =========== Modified imagestats to throw an exception if the npix returned by the computeMean function is ever less than or equal to zero. =========================================== ==== ==== Applications ==== =========================================== SAACLEAN ======== The new task, 'saaclean' measures and removes an estimate of the SAA persistence signal in a NICMOS image. It was developed as a stand-alone Python task and an IRAF interface has been provided to allow it to be run as an IRAF task under the HST_CALIB.NICMOS package in STSDAS. The input image to the task must be at least partially calibrated, including zero-read subtraction, dark subtraction, and linearization. The image may also be flatfielded. The task works by determining an appropriate scale factor by which to multiply the persistence image before subtracting it from the science image. It does this iteratively, measuring the noise in the image after each subtraction, then fits a parabola to the results to determine the minimum. Before performing any corrections, the task flags hot pixels, pixels at the edges, and pixels in the central row and column, to exclude them from the estimate and correction process. A summary of the results is written to standard output, and is encoded in the header of the output file. A new section of header keywords is inserted following the "POST-SAA DARK KEYWORDS" section, labelled "SAA_CLEAN output keywords". They include: SAAPERS SAA persistence image SCNPSCL scale factor used to construct persistence img SCNPMDN median used in flatfielding persistence image SCNTHRSH Threshold dividing high & low signal domains SCNHNPIX Number of pixels in high signal domain (HSD) SCNLNPIX Number of pixels in low signal domain (LSD) SCNGAIN gain used for effective noise calculations SCNHSCL HSD scale factor for min noise SCNHEFFN HSD effective noise at SCNGAIN SCNHNRED HSD noise reduction (percent) SCNLSCL LSD scale factor for min noise SCNLEFFN LSD effective noise at SCNGAIN SCNLNRED LSD noise reduction (percent) SCNAPPLD to which domains was SAA cleaning applied MultiDrizzle ============ MultiDrizzle has been updated to version 2.7.0. The primary changes include: - MultiDrizzle now runs on 64-bit Linux using the version of 'f2c' included in this release. - Documentation derived from epydoc was updated, and put into a new 'multidrizzle_api' directory. This update also included new PDF and PS versions. - New parameters 'driz_cr_grow' and 'driz_cr_ctegrow' for flagging cosmic ray CTE tails have been added to the interface. These new parameters only control the regions used around already identified cosmic-rays for more stringent tests to identify and mask CTE affected tails. - Changes were made to 'buildmask' to always start with a copy of the DQ array to support changes in numarray 1.2a and later. - Changes were made to the IRAF parameter interface to allow for the use of the new 'minimum' combination function in the CREATE MEDIAN step. - A new IterFitsFile class to support working with arbitrary numbers of input images at once while only have 1 file open at any given time. The ImageManager class was then modified to use this for creating the median image. The only limitation now remains the size of the buffer used for the slice from each input image, as only the slice gets processed in memory. - Output products from MultiDrizzle now contain HISTORY keywords reporting what versions of MultiDrizzle, PyDrizzle, PyFITS, and other pertinent libraries were used to generate the product. - Input images with EXPTIME of 0 (zero) seconds now get ignored. They are not included when running MultiDrizzle, and a warning message reports what files were ignored. - The 'tophat' kernel now can be successfully selected for use. - The creation of the median image will now more closesly replicate the IRAF IMCOMBINE behavior of nkeep = 1 and nhigh = 1. This prevents the over-rejection of pixels in non-overlapping science images. - The minmed algorithm was modified such that the first median image created would ignore the use of nhigh = 1 clipping if using it would result in no image data for that pixel location. - The 'wt_scl' values used for IVM and ERR weight type specifications now get computed as the exposure time scaled by the pixel scale rather than just the exptime. - Modified mdrizpars _handleMdriztab method to support the new version of PyFits (0.9.8.1). The new version of PYFITS returns FITS standard types for columen definitions. This is in contrast with previous versions that returned numarray types for columns. - Changes were made to when the output frame gets defined so that MultiDrizzle can be successfully restarted at the 'blot' step. - A bug in the driz_cr step that created invalid simple FITS format files for the *_cor.fits and *_crmask.fits products has been corrected. - Turned off use of memory mapping. PyDrizzle ========= PyDrizzle has been updated to Version 5.6.0. This new version includes the following set of changes/bug fixes/improvements. - PyDrizzle now runs on 64-bit Linux using the version of 'f2c' included in this release. - This new version implements the proper conversion of the output product to 'counts' based on 'units' parameter. This required adding the exposure time as an attribute of the Exposure class in order to allow for proper conversion of the product from 'counts' to 'cps' and back. - Images without distortion coefficients had a 1 pixel shift in both X and Y imposed on them by PyDrizzle. This correction now allows the predicted output size to match the actual drizzle product resulting in almost no dropped pixels for most cases, and for images with no distortion being applied to come out nearly unchanged. - Output of the translated coeffs file was moved to the Exposure class method 'writeCoeffs'. - The 'XYtoSky' function was moved from 'wcsutil' module to its own module, so that 'wcsutil' can be imported separately from rest of PyDrizzle. - The 'read_archive' method in 'wcsutil' needed to be modified to use any existing prefix found in the header for archived keywords, especially for computing the pscale. - The output FITS products from PyDrizzle no longer have extension keywords in the primary headers when 'build=no' to conform to the FITS standard. - Velocity aberration correction now gets turned off when no distortion coeffs have been specified. - A new method, 'runDriz', has been added to the Exposure class to allow a single chip to be drizzled to an output image as if cut as a section from the final product. - The description of the 'bits' parameter in the PyDrizzle docstring was updated to explain both 'bits_single' and 'bits_final'. - The shifts applied to 'blot' are now properly scaled to the 'input' frame to match the STSDAS interface to 'blot'. This corrects the problem of the blotted images not matching the original input chips when scale != 1, which was the case for all WFPC2 images. This allows WFPC2 images to finally be masked properly by MultiDrizzle. - Distortion correction images now have the correct subarray section applied to subarray input images. - Subarray images with the reference point specified outside the image area have the WCS values properly interpreted for computing shifts between images. - Arbitrary numbers of input images can be processed now, due to changes in file handling so that only 1 file is ever open at a time. - Combining images of different input sizes now works as expected. Previously, a memory buffer the size of the first image was used for all input images when 'drizzling' the images. This was modified to create a new buffer for each input and delete it when finished, so that only 1 buffer ever remained in memory at the same time. - Turning off usage of memory mapping by default.