11 February 2008
This release includes new versions of everything, in that anything that used to use numarray now uses numpy. Furthermore, there are changes to PyRAF, PyFITS, pytools, Multidrizzle and related tasks.
Last release to support Numarray
This release is the last release that supports numarray, even as a deprecated
package. Numarray is no longer used within STScI_Python. Support for
numarray will be eliminated entirely for the next release. Support for numarray
now is limited to critical problems or major platform support issues. The
threshold for 'critical' or 'major' is continually becoming harder to meet;
you are advised to migrate to numpy as soon as possible. Information on
making the switch to numpy from numarray may be found at this
Normally, platform support issues do not apply to Python tasks, as most Python
code will run on all platforms on which Python has been installed. This
distribution was tested to correctly support installation on Linux, Mac OS X,
and Solaris, while also being provided for installation under Windows. The
single exception is that the text-based epar functionality now available in
PyRAF (in addition to the already existing GUI-based epar) is not available
under Solaris, and likely will never be.
Documentation for these tasks has been consolidated into a single location
complete with an index viewable as a local web page. The documentation
includes any available user guides and API documentation generated using
'EPYDOC' for the modules or packages. This index can be accessed using:
--> import stscidocs
This will automatically bring up the default web browser application to display the index page. All the available documentation for software packaged with this distribution can be accessed through this interface, as this documentation resides locally upon installation.
Python 2.3 or later is required. We recommend using the latest version of
Python if there is a choice (2.5.x)
PyRAF 1.5Changes since the PyRAF 1.4 release:
The following enhancements were made:
- Some general refactoring of PyRAF's GKI-related classes was performed in preparation for the coming Matplotlib graphics kernel. The first look at this new capability is expected in PyRAF 1.6. (ticket #76)
- The PyRAF 'time' command now matches the format of the associated cl command. (ticket #67)
- Two graphics-related test modules were added to PyRAF for automated regression testing. (rev 802, no ticket)
Internal GkiKernel behavior was changed so that communication pipe
constructs are able to handle special redirection cases. This is
useful for the embedding of PyRAF within external applications.
The following bugs were fixed:
- There was a bug which allowed the wrong package to be assigned to a task in cases where the package script loaded a different package. (ticket #68)
- The algorithm used by PyRAF to search for task parameters was changed to more exactly match that used by the cl. (rev 796, no ticket)
- The sscanf module's buffer size in PyRAF was made to be consistent with the same buffer size in the cl. (ticket #65)
- The ability to append graphics (e.g., via "append+") was not working correctly due to a problem in the logic of when to initialize a new graphics engine. (ticket #70)
- As a performance improvement, element checks for IRAF arrays were simplified, otherwise the whole array was being checked every time a single element was assigned. For large arrays (thousands of elements), this resulted in a performance increase of orders of magnitude. (rev 803, no ticket)
- PyRAF can now handle plots appended to GKI metacode files (e.g., "prow dev$pix 250 append+ >>G test.gki") via the redirection operators. (ticket #72)
- On Linux (Ubuntu) a user was seeing a traceback with "cd .." which turned out to have to do with poor handling of Unicode in PyRAF commands. Such PyRAF commands are now being handled correctly. (ticket #73)
This release contains PyFITS Version 1.3 (24-Jan 2008). Updates described
in this release are only supported in the NUMPY version of pyfits.
The following enhancements were made:
- Provided support for a new extension to pyfits called stpyfits.
The stpyfits module is a wrapper around pyfits. It provides
all of the features and functions of pyfits along with some STScI specific
features. Currently, the only new feature supported by stpyfits is
the ability to read and write fits files that contain image data quality
extensions with constant data value arrays. See:
http://stsdas.stsci.edu/pytools/stpyfitsfor more details on stpyfits.
Added a new feature to allow trailing HDUs to be deleted from a fits file
without actually reading the data into the file.
- This supports a JWST requirement to delete a trailing HDU from a file whose primary Image HDU is too large to be read on a 32 bit machine.
Updated pyfits to use the warnings module to issue warnings. All warnings
will still be issued to stdout, exactly as they were before, however, you
may now suppress warnings with the -Wignore command line option. For
example, to run a script that will ignore warnings use the following command
python -Wignore yourscript.py
- Updated the open convenience function to allow the input of an already opened file object in place of a file name when opening a fits file.
Updated the writeto convenience function to allow it to accept the
- In this way, the user can use the argument output_verify='fix' to allow pyfits to correct any errors it encounters in the provided header before writing the data to the file.
- Updated the verification code to provide additional detail with a VerifyError exception.
- Added the capability to create a binary table HDU directly from a numpy.ndarray. This may be done using either the new_table convenience function or the BinTableHDU constructor.
The following performance improvements were made:
- Modified the import logic to dramatically decrease the time it takes to import pyfits.
- Modified the code to provide performance improvements when copying and examining header cards.
The following bugs were fixed:
- Corrected a bug that occurs when reading the data from a fits file that includes BZERO/BSCALE scaling. When the data is read in from the file, pyfits automatically scales the data using the BZERO/BSCALE values in the header. In the previous release, pyfits created a 32 bit floating point array to hold the scaled data. This could cause a problem when the value of BZERO is so large that the scaled value will not fit into the float 32. For this release, when the input data is 32 bit integer, a 64 bit floating point array is used for the scaled data.
- Corrected a bug that caused an exception to be raised when attempting to scale image data using the ImageHDU.scale method.
- Corrected a bug in the new_table convenience function that occurred when a binary table was created using a ColDefs object as input and supplying an nrows argument for a number of rows that is greater than the number of rows present in the input ColDefs object. The previous version of pyfits failed to allocate the necessary memory for the additional rows.
- Corrected a bug in the new_table convenience function that caused an exception to be thrown when creating an ASCII table.
- Corrected a bug in the new_table convenience function so as to allow the input of a ColDefs object that was read from a file as a binary table with a data value equal to None.
- Corrected a bug in the construction of ASCII tables from Column objects that are created with noncontinuous start columns.
- Corrected bugs in a number of areas that would sometimes cause a failure to raise an exception when an error occurred.
- Corrected a bug where attempting to open a non-existent fits file on a windows platform using a drive letter in the file specification caused a misleading IOError exception to be raised.
- Removed all support for numarray objects so it only works with numpy objects as input.
- Added a reset method to allow for better user control over parameters controlling how the array gets transformed for display.
A new method was added to allow the user to get the values related to the
properties of the display buffer.
Scaling of arrays where all values are much less than 1.0 now handled
- Modified to remove all compatibility with numarray, relying on native numpy C-api.
- Revised to reproduce numarray downcasting behaviors for input array types.
PytoolsThis set of modules has been used to create a formal package for use under Python. All code from this package now need to be accessed using:
--> from pytools import <module>
- A new module, stpyfits, was added. This module is an extension to the pyfits module which offers additional features specific to STScI; specifically, the handling of Constant Data Value Arrays.
- A new function 'isFits' has been added to the fileutil module to determine whether an input file is a FITS file or not, and if so, what format of FITS file; namely, 'waiver' FITS (primary header with data cube and binary table extension containing parameter values related to each slice of the data cube), multi-extension FITS (MEF), or 'simple' FITS (primary header only, with or without data).
Updates were made to several modules to support automatic conversion and use
of GEIS images through the 'openImage' function in fileutil. The
changes to 'readgeis.py' included allowing it to work under win32, and
populating the PyFITS object with the extname and extver so that the
'[sci,1]' syntax will work just as if it were read in from a FITS file.
Error messages are now reported when using 'readgeis' or 'openImage'
to read in a GEIS image suspected of having a byteorder problem.
- Docstring was also heavily updated for the 'openImage' function.
- Changed the 'irafglob' function to raise an IOError exception instead of an empty list when no valid input files are found.
The 'parseinput' function can now support IRAF style @files which may
include a second column of filenames.
The following changes were made:
- Automatic generation of IVM files for use with NICMOS data has been added.
- Error states now generate exceptions with error messages more directly related to the cause of the problem.
Now updates WFPC2 DQ image (c1h files as converted to FITS format) with
cosmic-ray mask identifications made by MultiDrizzle. If no c1h files are
present on input, the cosmic-ray identifications do not get reported to the
- Simplified the creation of a NICMOS dark image for use in IVM generation. Multidrizzle assumes a constant dark rate for the detectors.
The following bugs were fixed:
- Fixed a bug in how the output name gets determined for a single input image.
Fixed a bug in the handling of flat field images in the NICMOS IVM
generation routines. Multidrizzle now uses NICMOS flats that are
properly gain corrected.
- The algorithm for creating the IVM files automatically was corrected.
- Removed all remaining references to use of older cubic distortion coefficients for WFPC2 data, replacing them with references to IDCTAB reference file.
Fixed a problem with the error messages generated when the boxcar
convolution in the minmed algorithm fails.
The following changes were made to improve support for WFPC2 data:
- DQ information for WFPC2 data now correctly recognized for all input formats
- PyDrizzle creates an empty DQ array for use by PyDrizzle if none are found.
The following bugs were fixed:
- Processing an image which does not have any distortion coefficients now works as expected; specifically, no changes get imposed on the image.
Corrected a problem in the definition of the default DQ array
The COS pipeline calibration processing software, calcos, has been added to this
release. This represents the first public release of this code in support
of the upcoming installation of COS on HST during Servicing Mission 4 currently
scheduled for September 2007. This code performs all the standard
calibrations to COS data for use in pipeline processing or reprocessing by an
NICMOS Data Analysis
The tasks RNLINCOR, SAACLEAN, and PUFTCORR have been reorganized so that they
are now part of a larger set of tasks designed for NICMOS data analysis and
calibration, the NICTOOLS package. As before, they can each be imported
independently and can be used on NICMOS data directly from Python without any
STSDAS interface available to run them. This package also includes new
nic_rem_persist: This is the first version of a tool to remove
persistence from NICMOS data. It is being provided on an experimental
basis only, with extensive testing and development still being
- temp_from_bias: This task computes the detector temperature based on the measurements of the bias values in NICMOS data. It is being provided on an experimental basis only, with extensive testing and development still being required.
- Handling of the sky values corrected in rnlincor.
Version numbers no longer printed out when running saaclean.
STIS Data Analysis
The STIS data analysis tasks were reorganized into a single package,
stistools. This package still contains modules for supporting data
analysis tasks developed for post-pipeline processing of STIS data; namely,
wx2d, stisnoise, mktrace, sshift. They can each be imported independently and
used on STIS data directly from Python.
No changes were implemented to any of the tasks themselves in this package for this release.
WFPC2 Data analysisA new package, wfpc2tools, has been added to provide new tools for working specifically with WFPC2 data. This new package currently contains the following modules:
- wfpc2cte: computes estimates of the effects of CTE on an input WFPC2 image, and reports the change in magnitude the CTE would impose on point sources with 3 values of total flux. The algorithm used by this module is explained in the help for the module, and can be obtained using the built-in help() command:
--> import wfpc2tools;wfpc2tools.wfpc2cte.help()
- wfpc2destreak: An experimental module which attempts to identify when low bias readout of WFPC2 data causes uneven bias values across the chip, and attemps to correct the uneven background caused by this bias readout problem. This module is provided on an as-is basis as it still requires additional testing and development.