STScI_Python Version 2.11 Release Notes
December 2010
This release includes new versions of PyRAF, PyFITS, STWCS, pytools,
Multidrizzle, and related tasks. This release also includes
instrument-specific packages to support ACS, NICMOS, and COS.
The convolve, image, and ndimage packages have been included with
this release to insure compatibility with the rest of our code.
Platform Support
================
Normally, platform support issues do not apply to Python tasks,
as most Python code will run on all platforms on which Python has
been installed. This distribution was tested to correctly support
installation on Linux, Mac OS X 10.5 (Leopard), and Solaris, while
also being provided for installation under Windows. The single
exception is that the text-based epar functionality now available
in PyRAF (in addition to the already existing GUI-based epar) is
not available under Solaris, and likely will never be. As of the
June 2010 release, PyRAF no longer requires the installation of IRAF.
Note that the only IRAF distribution for Windows runs under Cygwin,
but no testing of PyRAF has been conducted under Cygwin.
Solaris Support Ending
----------------------
This version will be the last version which will be tested against
Solaris. We will also no longer be building Solaris binaries for STSDAS
after this release, although the source code can always be downloaded
and compiled locally as needed.
Documentation
=============
Documentation for these tasks has been consolidated into a
single location complete with an index viewable as a local web
page. The documentation includes any available user guides and
API documentation generated using 'EPYDOC' for the modules or
packages. This index can be accessed using:
--> import stscidocs
--> stscidocs.viewdocs()
--> stscidocs.help()
This will automatically bring up the default web browser application
to display the index page. All the available documentation for
software packaged with this distribution can be accessed through this
interface, as this documentation resides locally upon installation.
Python Environment
==================
This release has been tested using Python Versions 2.5.4 and 2.7. This
release also requires at least numpy 1.3.
acstools
=============
This new package contains tasks to process ACS data after retrieval from
the HST archive. These tasks include:
- PixCteCorr: A stand-alone task to correct ACS _flt.fits files of
the effects of CTE artifacts prior to further analysis or
processing (such as cosmic-ray cleaning and combination by
MultiDrizzle).
- acs_destripe: A stand-alone task to remove the effects of row-to-row
bias variations caused by the new electronics used to repair
ACS during Servicing Mission 4 (SM4).
Both of these tasks have been developed with TEAL interfaces, a GUI
interface similar to the EPAR GUI under PYRAF without the dependence
on IRAF or PyRAF. They also include fairly extensive Sphinx-based
documentation to describe how to run the tasks.
calcos
=============
NOTE: This version (2.13.6) of calcos contains a bug that was discovered
too late to be fixed for this release. If the input is a corrtag file
for spectroscopic data, calcos will fail (KeyError: "Keyword 'ctype1'
not found.") when trying to write the world coordinate system (WCS)
keywords to the header of the flt or counts file. This can be worked
around either by deleting keyword SPWCSTAB or by changing its value in
the input corrtag header(s) to "N/A". The WCS keywords will then not
be written.
A bug which resulted in incorrect values in NUV x1dsum tables has been
fixed. In some cases there was a gap in the data quality flags (the DQ
extension in the flt and counts files) between the near-edge flag and
the out-of-bounds flag. The data values were zero there, but because
DQ was zero, the zero data values were included in the average in x1dsum.
A bug was fixed in the code for comparing version strings. For example,
2.13.1 was not recognized as being later than 2.13. This comparison is
done to check for compatibility between calcos and reference files.
If the FLUXTAB does not have a row matching the observing configuration,
calcos was quietly setting the values in the FLUX column to zero.
Calcos now raises an exception if there is no matching row in the FLUXTAB.
The stim count rate was not being computed correctly. This value is
just printed for information pertaining to the deadtime correction.
A new command-line argument --find was added. If specified, calcos will
search for the target spectrum in the Y direction rather than relying
completely on the offset from the wavecal spectrum.
Options were added to the 1-D extraction to specify the location at
which the spectrum should be extracted and/or the extraction height;
these options are available via the x1dcorr task but not in calcos itself.
For spectroscopic data, calcos now writes WCS keywords to the flt and
counts images and to the corrtag table. This is disabled, however,
if the SPWCSTAB keyword is missing or if its value is "N/A".
There were changes in the names and interpretation of some of the data
quality flags. In particular, one flag DQ_PHA_OUT_OF_BOUNDS is now used
to flag pulse height out of bounds, rather than having separate flags
for low pulse height and high pulse height.
New columns NET and BACKGROUND are now written to the lampflash table.
New segment-specific or stripe-specific keywords are used. Keyword SP_HGT
has been changed to SP_HGT_A, SP_HGT_B, SP_HGT_C. Keywords EXPTIMEA,
EXPTIMEB, NBADEVTA, NBADEVTB NEVENTSA, and NEVENTSB are now used for FUV.
NEVENTS will be set to NEVENTSA + NEVENTSB in an x1d table. New keywords
are now written when extracting a 1-D spectrum. SP_NOM_A, SP_NOM_B,
SP_NOM_C are the nominal location (the expected value), and SP_OFF_A,
SP_OFF_B, SP_OFF_C give the offset from nominal to where the spectrum
was actually found.
For most calibration steps for TIME-TAG data that use the exposure time,
the value that will be used is the difference between the first and last
events. The exposure time that has been corrected for time lost due to
bursts and bad time intervals is used for converting counts to count rate.
imagestats
=============
Added Sphinx-based documentation to describe the code in this package.
multidrizzle
=============
Development on MultiDrizzle has focused on getting a replacement,
currently named 'betadrizzle', tested and validated so that it can
be publicly released. Only bug fixes required for continued use of
MultiDrizzle in HST operations and the archive will be investigated and
fixed until the replacement has been installed for use in HST operations
and the archive.
Only a few bug fixes required for pipeline use were implemented in
MultiDrizzle since the last release; specifically,
- the runfile now gets deleted when the parameter 'clean' is set to True
- The changes to PyDrizzle's 'pydrizzle.distortion.mutil' module
support the computation of the time-dependent distortion (TDD)
terms based on arbitrary numbers of polynomial coefficients read
in from the PRIMARY header of the IDCTAB. This allows separate
solutions to be created for different epochs of data, including
pre- and post-SM4 data. This change supports the use of updated
versions of the IDCTAB reference files provided by the ACS team
to support both pre- and post-SM4 data.
- Logic for printing out warning message about using hard-coded TDD
coefficients was corrected to allow the default values to be used
for pre-SM4 data.
A couple of small bugs affecting stand-alone use of MultiDrizzle by
users were also addressed; namely,
- the value of 'in_un' as written out to the runfile has both starting
and ending quote marks to allow it to work as input to IRAF
- Corrected a couple of typos in pydrizzle.process_input to address
problems when trying to process a STIS association without the
SPT files.
This version of MultiDrizzle has been identified to contain a couple of
bugs which will not be fixed in this code, although these issues have
already been verified to work correctly in 'betadrizzle'. In particular,
- WFC3/UVIS binned observations are not properly distortion corrected
resulting in the chips being separated by over 2000 pixels instead
of the expected ~10 pixels.
- an small offset in the sky coordinates may be introduced into the
output product when specifying a non-default output orientation.
nictools
=============
A new version of 'saaclean', Version 1.4(2010-10-20), includes all the
changes necessary for creating an updated header that contains keywords
derived from the input post-SAA-dark reference files, including the
DATE keyword.
numdisplay
=============
A simple README file was added to the package to provide basic
instructions for manually installing the numdisplay package.
opuscoords
=============
A problem with the inclusion of the 'from future' import statement in
opuscoords was fixed to allow this code to be run in HST operations.
PyFITS
=============
This release contains Version 2.4.0 of PyFITS.
- The following enhancements were added:
- Checksum support now correctly conforms to the FITS standard.
pyfits supports reading and writing both the old checksums and new
standard-compliant checksums. The `fitscheck` command-line
utility is provided to verify and update checksums.
- Added a new optional keyword argument `do_not_scale_image_data` to
the pyfits.open convenience function. When this argument is
provided as True, and an ImageHDU is read that contains scaled
data, the data is not automatically scaled when it is read. This
option may be used when opening a fits file for update, when you
only want to update some header data. Without the use of this
argument, if the header updates required the size of the fits file
to change, then when writing the updated information, the data
would be read, scaled, and written back out in its scaled format
(usually with a different data type) instead of in its non-scaled
format.
- Added a new optional keyword argument `disable_image_compression`
to the pyfits.open function. When True, any compressed image
HDU's will be read in like they are binary table HDU's.
- Added a 'verify' keyword argument to the pyfits.append function.
When False, append will assume the existing FITS file is already
valid and simply append new content to the end of the file,
resulting in a large speed up appending to large files.
- Added HDU methods `update_ext_name` and `update_ext_version` for
updating the name and version of an HDU.
- Added HDU method `filebytes` to calculate the number of bytes that
will be written to the file associated with the HDU.
- Enhanced the section class to allow reading non-contiguous image
data. Previously, the section class could only be used to read
contiguous data.
- Added method HDUList.fileinfo() that returns a dictionary with
information about the location of header and data in the file
associated with the HDU.
- The following bugs were fixed:
- Reading in some malformed FITS headers would cause a `NameError`
exception, rather than information about the cause of the error.
- pyfits can now handle non-compliant CONTINUE cards produced by
Java FITS.
- BinTable columns with TSCALn are now byte-swapped correctly.
- Ensure that floating-point card values are no longer than 20
characters.
- Updated flush so that when the data has changed in an HDU for a
file opened in update mode, the header will be updated to match
the changed data before writing out the HDU.
- Allow hierarch cards to contain a keyword and value whose total
character length is 69 characters. Previous length was limited at
68 characters.
- Calls to FITS_rec['columnName'] now return an ndarray exactly the
same as a call to FITS_rec.field('columnName') or
FITS_rec.columnName. Previously, FITS_rec['columnName'] returned
a much less useful fits_record object.
- Corrected the append convenience function to eliminate the reading
of the HDU data from the file that is being appended to.
- Eliminated common symbols between the pyfitsComp module and the
cfitsio and zlib libraries. These can cause problems on systems
that use both PyFITS and cfitsio or zlib.
PyRAF
=============
PyRAF 1.10 has been released (bundled with stsci_python 2.11 as well
as stand-alone). This release supports Python 2.5, 2.6, and 2.7.
Since the 1.9 bundled release, the following enhancements have been made:
- PyRAF commands can now be run as '-c' arguments to the 'pyraf'
command, i.e. without entering the command-line shell.
See also FAQ 2.9.
- The 'unlearn' function is now supported for TEAL-enabled tasks.
- The CL filename information has been added to any syntax errors
thrown when PyRAF parses a CL file.
- A change was made to EPAR (for TEAL-enabled tasks) to allow it
to recognize sub-modules of previously imported task packages.
- PyRAF can now be run when there is no widget library (i.e. Tkinter)
installed on the node. As with no display, i.e. terminal
interaction only, to run this way without error remember to
setenv PYRAF_NO_DISPLAY.
and the following bugs have been fixed:
- Tabbed filename completion had an issue where PyRAF would list
directory contents before they were requested by TAB.
- The 'pyraf -h' command now correctly displays all the possible
command-line options .
- PyRAF was not elegantly handling when a corruption in the
.pyraf_history file was found.
- An unused feature was removed: PyRAF had been adding its bin
directory to sys.path.
- EPAR: There was a bug in the parameter interface code causing EPAR
and TPAR to incorrectly save array-type parameters. A second bug
during Save was fixed for cases where the 'uparm' variable is
not correctly formed. Some EPAR GUI color fixes were made as well.
- Plots were not being fully drawn after imexamine commands via ds9
when using the newly sped-up matplotlib graphics kernel.
- A user found an issue with saveToFile() and with clear() when they
had no installed /usr/bin/tput.
- PyRAF installation notes state that the readline module is optional,
but it was causing an exception when missing.
In addition, code changes not affecting behavior are gradually being
made to PyRAF in an effort to support the eventual transition to Python
3. This release includes such changes.
pysynphot
=============
The version of pysynphot distributed with this release is the version used
by ETC 19.1.
A number of data changes are included with this release
- The Vega spectrum used to calculate vegamags has been updated
to alpha_lyr_stis_005.fits
- Wavecat entries have been added or modified to support new or
changed COS obsmodes
The following code changes were also made:
- Support for new flux units microjy (mujy) and nanojy (njy) was added
- The Box bandpass constructor now supports an optional waveunits
keyword, to define a box in wavelength units other than Angstroms
pytools
=============
Sphinx documentation for many of the modules in PyTOOLS has been added
to the pytools package, including the addition of the TEAL User Guide
to describe how to add a TEAL interface to any Python task.
fileutil:
- The 'osfn()' function in 'pytools.fileutil' has been updated to
return None if None is provided as input and to work properly for
*nix style variable names in filenames. Examples of filenames
that can now be expanded correctly when the variables are
defined include: $PYSYN_CDBS/mtab/t2605492m_tmg.fits and
mtab$t2605492m_tmg.fits.
- A new function was added to 'pytools.fileutil', countExtn(),
which returns the number of extensions with a given EXTNAME. This
new function will be used in conjunction with the STWCS functions
for restoring or archiving keyed-WCSs.
teal:
- A new function, getHelpFileAsString(), was added to pytools.teal
to make it easier to implement TEAL GUI's for tasks.
- The teal.teal() interface has an optional boolean arg named
canExecute, where default=True.
- Updated the mouse-wheel action to be move faster by default.
- A new function, teal.print_tasknames(), was added to print out the
names of all TEAL enabled tasks within a package. This can be
used in the __init__ module of a package to automatically report
all TEAL-enabled tasks to the user upon importing the package.
- The set of keywords understood by TEAL in the configobj file was
expanded to support more advanced behaviors. In particular,
the "trigger" kwd was changed to "triggers", the "is_set_by" kwd
was added to allow control by another parameter.
- Some GUI changes were made for basic usability: status widget
messages will be held and queued, browser-launching code is now
more flexible on more platforms, and a bug with focusOut events
(triggering rule actions) has been fixed.
wcsutil:
- The error messages when wcsutil.WCSObject raises an exception on
reading a WCS have been refined to report the keyword which
actually caused the failure. The bug and fix was reported by
F. Stoehr(ESO).
stwcs
=============
This release contains Version 0.8 of the STWCS package, a package which
provides suport for WCS based distortion models and coordinate
transformations. It relies on PyWCS (based on WCSLIB).
Bugs fixed:
- The default value for 'skew_coeffs' in STWCS.distortion.mutil needed
to be reset to None so that logic in STWCS.updatewcs.corrections
can recognize that there are no TDD coefficients available
from the IDCTAB for use. The default set of skew_coeffs in
STWCS.updatewcs needed to be updated to include the TDDORDER
key and value with a default value of 1 as appropriate for the
hard-coded default coeffs. These changes were needed to allow
betadrizzle/updatewcs to work correctly with IDCTAB files which
do not yet have the TDD coeffs in the PRIMARY header.
- Code which defines the reference chip based on instrument/detector
combination was changed to work with FITS files with arbitrary
number and order of extensions.
- Fixed a problem with STIS observations caused by DATE-OBS being in
each science extension instead of the primary header.
- Fixed a bug related to different keywords in STIS MAMA and CCD
observations.
- Added 'SAMP' and 'TIME' to the list of EXTNAME keywords to support
NICMOS observations.
- A number of changes were made to keep in sync with changes in PyWCS.
New Features:
- Support for alternate WCS's in headers as described in WCS Paper I
was added to STWCS.WCSUTIL. This includes functions for archiving,
restoring and deleting an alternate WCS and listing the available
WCS's by name and key.
- When UPDATEWCS is run with a new IDCTAB, all WCS's except the original
OPUS WCS are deleted now and a new one is computed.
Documentation:
- Added Sphinx based documentation.
- An initial draft of a document defining the headerlet and how it will
be used has been added to the STWCS Sphinx docs directory.
- A draft of a FITS Conventions TSR was added to the documentation
- A TSR on how the NPOL files were created and tested was finalized and
added to the documentation.
