STScI Logo

stsci_python 2.11 Release Notes

STScI_Python Version 2.11 Release Notes

December 2010

This release includes new versions of PyRAF, PyFITS, STWCS, pytools,
Multidrizzle, and related tasks.  This release also includes
instrument-specific packages to support ACS, NICMOS, and COS.

The convolve, image, and ndimage packages have been included with
this release to insure compatibility with the rest of our code.

Platform Support
Normally, platform support issues do not apply to Python tasks,
as most Python code will run on all platforms on which Python has
been installed. This distribution was tested to correctly support
installation on Linux, Mac OS X 10.5 (Leopard), and Solaris, while
also being provided for installation under Windows. The single
exception is that the text-based epar functionality now available
in PyRAF (in addition to the already existing GUI-based epar) is
not available under Solaris, and likely will never be. As of the
June 2010 release, PyRAF no longer requires the installation of IRAF.
Note that the only IRAF distribution for Windows runs under Cygwin,
but no testing of PyRAF has been conducted under Cygwin.

Solaris Support Ending
This version will be the last version which will be tested against
Solaris.  We will also no longer be building Solaris binaries for STSDAS
after this release, although the source code can always be downloaded
and compiled locally as needed.
Documentation for these tasks has been consolidated into a
single location complete with an index viewable as a local web
page. The documentation includes any available user guides and
API documentation generated using 'EPYDOC' for the modules or
packages. This index can be accessed using:
 --> import stscidocs
 --> stscidocs.viewdocs()
This will automatically bring up the default web browser application
to display the index page. All the available documentation for
software packaged with this distribution can be accessed through this
interface, as this documentation resides locally upon installation.

Python Environment
This release has been tested using Python Versions 2.5.4 and 2.7. This
release also requires at least numpy 1.3.

This new package contains tasks to process ACS data after retrieval from
the HST archive.  These tasks include:
    - PixCteCorr: A stand-alone task to correct ACS _flt.fits files of
        the effects of CTE artifacts prior to further analysis or
        processing (such as cosmic-ray cleaning and combination by
    - acs_destripe: A stand-alone task to remove the effects of row-to-row
        bias variations caused by the new electronics used to repair
        ACS during Servicing Mission 4 (SM4).
Both of these tasks have been developed with TEAL interfaces, a GUI
interface similar to the EPAR GUI under PYRAF without the dependence
on IRAF or PyRAF. They also include fairly extensive Sphinx-based
documentation to describe how to run the tasks.

NOTE:  This version (2.13.6) of calcos contains a bug that was discovered
too late to be fixed for this release.  If the input is a corrtag file
for spectroscopic data, calcos will fail (KeyError: "Keyword 'ctype1'
not found.") when trying to write the world coordinate system (WCS)
keywords to the header of the flt or counts file.  This can be worked
around either by deleting keyword SPWCSTAB or by changing its value in
the input corrtag header(s) to "N/A".  The WCS keywords will then not
be written.

A bug which resulted in incorrect values in NUV x1dsum tables has been
fixed.  In some cases there was a gap in the data quality flags (the DQ
extension in the flt and counts files) between the near-edge flag and
the out-of-bounds flag.  The data values were zero there, but because
DQ was zero, the zero data values were included in the average in x1dsum.

A bug was fixed in the code for comparing version strings.  For example,
2.13.1 was not recognized as being later than 2.13.  This comparison is
done to check for compatibility between calcos and reference files.

If the FLUXTAB does not have a row matching the observing configuration,
calcos was quietly setting the values in the FLUX column to zero.
Calcos now raises an exception if there is no matching row in the FLUXTAB.

The stim count rate was not being computed correctly.  This value is
just printed for information pertaining to the deadtime correction.

A new command-line argument --find was added.  If specified, calcos will
search for the target spectrum in the Y direction rather than relying
completely on the offset from the wavecal spectrum.

Options were added to the 1-D extraction to specify the location at
which the spectrum should be extracted and/or the extraction height;
these options are available via the x1dcorr task but not in calcos itself.

For spectroscopic data, calcos now writes WCS keywords to the flt and
counts images and to the corrtag table.  This is disabled, however,
if the SPWCSTAB keyword is missing or if its value is "N/A".

There were changes in the names and interpretation of some of the data
quality flags.  In particular, one flag DQ_PHA_OUT_OF_BOUNDS is now used
to flag pulse height out of bounds, rather than having separate flags
for low pulse height and high pulse height.

New columns NET and BACKGROUND are now written to the lampflash table.

New segment-specific or stripe-specific keywords are used.  Keyword SP_HGT
has been changed to SP_HGT_A, SP_HGT_B, SP_HGT_C.  Keywords EXPTIMEA,
NEVENTS will be set to NEVENTSA + NEVENTSB in an x1d table.  New keywords
are now written when extracting a 1-D spectrum.  SP_NOM_A, SP_NOM_B,
SP_NOM_C are the nominal location (the expected value), and SP_OFF_A,
SP_OFF_B, SP_OFF_C give the offset from nominal to where the spectrum
was actually found.

For most calibration steps for TIME-TAG data that use the exposure time,
the value that will be used is the difference between the first and last
events.  The exposure time that has been corrected for time lost due to
bursts and bad time intervals is used for converting counts to count rate.

Added Sphinx-based documentation to describe the code in this package.

Development on MultiDrizzle has focused on getting a replacement,
currently named 'betadrizzle', tested and validated so that it can
be publicly released. Only bug fixes required for continued use of
MultiDrizzle in HST operations and the archive will be investigated and
fixed until the replacement has been installed for use in HST operations
and the archive.

Only a few bug fixes required for pipeline use were implemented in
MultiDrizzle since the last release; specifically,
    - the runfile now gets deleted when the parameter 'clean' is set to True
    - The changes to PyDrizzle's 'pydrizzle.distortion.mutil' module
        support the computation of the time-dependent distortion (TDD)
        terms based on arbitrary numbers of polynomial coefficients read
        in from the PRIMARY header of the IDCTAB. This allows separate
        solutions to be created for different epochs of data, including
        pre- and post-SM4 data. This change supports the use of updated
        versions of the IDCTAB reference files provided by the ACS team
        to support both pre- and post-SM4 data.
    - Logic for printing out warning message about using hard-coded TDD
        coefficients was corrected to allow the default values to be used 
        for pre-SM4 data. 

A couple of small bugs affecting stand-alone use of MultiDrizzle by
users were also addressed; namely,
    - the value of 'in_un' as written out to the runfile has both starting
        and ending quote marks to allow it to work as input to IRAF
    - Corrected a couple of typos in pydrizzle.process_input to address 
        problems when trying to process a STIS association without the 
        SPT files. 
This version of MultiDrizzle has been identified to contain a couple of
bugs which will not be fixed in this code, although these issues have
already been verified to work correctly in 'betadrizzle'. In particular,
    - WFC3/UVIS binned observations are not properly distortion corrected
        resulting in the chips being separated by over 2000 pixels instead
        of the expected ~10 pixels. 
    - an small offset in the sky coordinates may be introduced into the 
        output product when specifying a non-default output orientation.

A new version of 'saaclean', Version 1.4(2010-10-20), includes all the
changes necessary for creating an updated header that contains keywords
derived from the input post-SAA-dark reference files, including the
DATE keyword.

A simple README file was added to the package to provide basic
instructions for manually installing the numdisplay package.

A problem with the inclusion of the 'from future' import statement in
opuscoords was fixed to allow this code to be run in HST operations.

This release contains Version 2.4.0 of PyFITS.

- The following enhancements were added:

  - Checksum support now correctly conforms to the FITS standard.
    pyfits supports reading and writing both the old checksums and new
    standard-compliant checksums.  The `fitscheck` command-line
    utility is provided to verify and update checksums.

  - Added a new optional keyword argument `do_not_scale_image_data` to
    the convenience function.  When this argument is
    provided as True, and an ImageHDU is read that contains scaled
    data, the data is not automatically scaled when it is read.  This
    option may be used when opening a fits file for update, when you
    only want to update some header data.  Without the use of this
    argument, if the header updates required the size of the fits file
    to change, then when writing the updated information, the data
    would be read, scaled, and written back out in its scaled format
    (usually with a different data type) instead of in its non-scaled

  - Added a new optional keyword argument `disable_image_compression`
    to the function.  When True, any compressed image
    HDU's will be read in like they are binary table HDU's.

  - Added a 'verify' keyword argument to the pyfits.append function.
    When False, append will assume the existing FITS file is already
    valid and simply append new content to the end of the file,
    resulting in a large speed up appending to large files.

  - Added HDU methods `update_ext_name` and `update_ext_version` for
    updating the name and version of an HDU.

  - Added HDU method `filebytes` to calculate the number of bytes that
    will be written to the file associated with the HDU.

  - Enhanced the section class to allow reading non-contiguous image
    data.  Previously, the section class could only be used to read
    contiguous data.

  - Added method HDUList.fileinfo() that returns a dictionary with
    information about the location of header and data in the file
    associated with the HDU.

- The following bugs were fixed:

  - Reading in some malformed FITS headers would cause a `NameError`
    exception, rather than information about the cause of the error.

  - pyfits can now handle non-compliant CONTINUE cards produced by
    Java FITS.
  - BinTable columns with TSCALn are now byte-swapped correctly.

  - Ensure that floating-point card values are no longer than 20

  - Updated flush so that when the data has changed in an HDU for a
    file opened in update mode, the header will be updated to match
    the changed data before writing out the HDU.

  - Allow hierarch cards to contain a keyword and value whose total
    character length is 69 characters.  Previous length was limited at
    68 characters.

  - Calls to FITS_rec['columnName'] now return an ndarray exactly the
    same as a call to FITS_rec.field('columnName') or
    FITS_rec.columnName.  Previously, FITS_rec['columnName'] returned
    a much less useful fits_record object.

  - Corrected the append convenience function to eliminate the reading
    of the HDU data from the file that is being appended to.

  - Eliminated common symbols between the pyfitsComp module and the
    cfitsio and zlib libraries.  These can cause problems on systems
    that use both PyFITS and cfitsio or zlib.

PyRAF 1.10 has been released (bundled with stsci_python 2.11 as well
as stand-alone).  This release supports Python 2.5, 2.6, and 2.7.
Since the 1.9 bundled release, the following enhancements have been made:

    - PyRAF commands can now be run as '-c' arguments to the 'pyraf'
        command, i.e. without entering the command-line shell. 
        See also FAQ 2.9.
    - The 'unlearn' function is now supported for TEAL-enabled tasks.
    - The CL filename information has been added to any syntax errors 
        thrown when PyRAF parses a CL file.
    - A change was made to EPAR (for TEAL-enabled tasks) to allow it 
        to recognize sub-modules of previously imported task packages.
    - PyRAF can now be run when there is no widget library (i.e. Tkinter)
        installed on the node. As with no display, i.e. terminal
        interaction only, to run this way without error remember to
        setenv PYRAF_NO_DISPLAY.

and the following bugs have been fixed:

    - Tabbed filename completion had an issue where PyRAF would list 
        directory contents before they were requested by TAB.
    - The 'pyraf -h' command now correctly displays all the possible 
        command-line options .
    - PyRAF was not elegantly handling when a corruption in the 
        .pyraf_history file was found.
    - An unused feature was removed: PyRAF had been adding its bin 
        directory to sys.path.
    - EPAR: There was a bug in the parameter interface code causing EPAR 
        and TPAR to incorrectly save array-type parameters. A second bug 
        during Save was fixed for cases where the 'uparm' variable is 
        not correctly formed. Some EPAR GUI color fixes were made as well.
    - Plots were not being fully drawn after imexamine commands via ds9 
        when using the newly sped-up matplotlib graphics kernel.
    - A user found an issue with saveToFile() and with clear() when they 
        had no installed /usr/bin/tput.
    - PyRAF installation notes state that the readline module is optional, 
        but it was causing an exception when missing. 

In addition, code changes not affecting behavior are gradually being
made to PyRAF in an effort to support the eventual transition to Python
3. This release includes such changes.

The version of pysynphot distributed with this release is the version used
by ETC 19.1.

A number of data changes are included with this release
    - The Vega spectrum used to calculate vegamags has been updated
        to alpha_lyr_stis_005.fits
    - Wavecat entries have been added or modified to support new or 
        changed COS obsmodes

The following code changes were also made:
    - Support for new flux units microjy (mujy) and nanojy (njy) was added
    - The Box bandpass constructor now supports an optional waveunits 
        keyword, to define a box in wavelength units other than Angstroms

Sphinx documentation for many of the modules in PyTOOLS has been added
to the pytools package, including the addition of the TEAL User Guide
to describe how to add a TEAL interface to any Python task.

    - The 'osfn()' function in 'pytools.fileutil' has been updated to
        return None if None is provided as input and to work properly for
        *nix style variable names in filenames. Examples of filenames
        that can now be expanded correctly when the variables are
        defined include: $PYSYN_CDBS/mtab/t2605492m_tmg.fits and
    - A new function was added to 'pytools.fileutil', countExtn(),
        which returns the number of extensions with a given EXTNAME. This
        new function will be used in conjunction with the STWCS functions
        for restoring or archiving keyed-WCSs.

    - A new function, getHelpFileAsString(), was added to pytools.teal 
        to make it easier to implement TEAL GUI's for tasks.
    - The teal.teal() interface has an optional boolean arg named 
        canExecute, where default=True.
    - Updated the mouse-wheel action to be move faster by default.
    - A new function, teal.print_tasknames(), was added to print out the
         names of all TEAL enabled tasks within a package. This can be
         used in the __init__ module of a package to automatically report
         all TEAL-enabled tasks to the user upon importing the package.
    - The set of keywords understood by TEAL in the configobj file was 
        expanded to support more advanced behaviors.  In particular,
        the "trigger" kwd was changed to "triggers", the "is_set_by" kwd
        was added to allow control by another parameter.
    - Some GUI changes were made for basic usability: status widget 
        messages will be held and queued, browser-launching code is now
        more flexible on more platforms, and a bug with focusOut events
        (triggering rule actions) has been fixed.

    -  The error messages when wcsutil.WCSObject raises an exception on
        reading a WCS have been refined to report the keyword which
        actually caused the failure. The bug and fix was reported by
        F. Stoehr(ESO).

This release contains Version 0.8 of the STWCS package, a package which
provides suport for WCS based distortion models and coordinate 
transformations.  It relies on PyWCS (based on WCSLIB).

Bugs fixed:
    - The default value for 'skew_coeffs' in STWCS.distortion.mutil needed
        to be reset to None so that logic in STWCS.updatewcs.corrections
        can recognize that there are no TDD coefficients available
        from the IDCTAB for use. The default set of skew_coeffs in
        STWCS.updatewcs needed to be updated to include the TDDORDER
        key and value with a default value of 1 as appropriate for the
        hard-coded default coeffs. These changes were needed to allow
        betadrizzle/updatewcs to work correctly with IDCTAB files which
        do not yet have the TDD coeffs in the PRIMARY header.
    - Code which defines the reference chip based on instrument/detector
        combination was changed to work with FITS files with arbitrary 
        number and order of extensions.
    - Fixed a problem with STIS observations caused by DATE-OBS being in 
        each science extension instead of the primary header.
    - Fixed a bug related to different keywords in STIS MAMA and CCD
    - Added 'SAMP' and 'TIME' to the list of EXTNAME keywords to support 
        NICMOS observations.
    - A number of changes were made to keep in sync with changes in PyWCS.

New Features:
    - Support for alternate WCS's in headers as described in WCS Paper I
        was added to STWCS.WCSUTIL. This includes functions for archiving,
        restoring and deleting an alternate WCS and listing the available
        WCS's by name and key.
    - When UPDATEWCS is run with a new IDCTAB, all WCS's except the original
        OPUS WCS are deleted now and a new one is computed.

    - Added Sphinx based documentation.
    - An initial draft of a document defining the headerlet and how it will
        be used has been added to the STWCS Sphinx docs directory.
    - A draft of a FITS Conventions TSR was added to the documentation
    - A TSR on how the NPOL files were created and tested was finalized and 
        added to the documentation.