STScI_Python Version 2.11 Release Notes December 2010 This release includes new versions of PyRAF, PyFITS, STWCS, pytools, Multidrizzle, and related tasks. This release also includes instrument-specific packages to support ACS, NICMOS, and COS. The convolve, image, and ndimage packages have been included with this release to insure compatibility with the rest of our code. Platform Support ================ Normally, platform support issues do not apply to Python tasks, as most Python code will run on all platforms on which Python has been installed. This distribution was tested to correctly support installation on Linux, Mac OS X 10.5 (Leopard), and Solaris, while also being provided for installation under Windows. The single exception is that the text-based epar functionality now available in PyRAF (in addition to the already existing GUI-based epar) is not available under Solaris, and likely will never be. As of the June 2010 release, PyRAF no longer requires the installation of IRAF. Note that the only IRAF distribution for Windows runs under Cygwin, but no testing of PyRAF has been conducted under Cygwin. Solaris Support Ending ---------------------- This version will be the last version which will be tested against Solaris. We will also no longer be building Solaris binaries for STSDAS after this release, although the source code can always be downloaded and compiled locally as needed. Documentation ============= Documentation for these tasks has been consolidated into a single location complete with an index viewable as a local web page. The documentation includes any available user guides and API documentation generated using 'EPYDOC' for the modules or packages. This index can be accessed using: --> import stscidocs --> stscidocs.viewdocs() --> stscidocs.help() This will automatically bring up the default web browser application to display the index page. All the available documentation for software packaged with this distribution can be accessed through this interface, as this documentation resides locally upon installation. Python Environment ================== This release has been tested using Python Versions 2.5.4 and 2.7. This release also requires at least numpy 1.3. acstools ============= This new package contains tasks to process ACS data after retrieval from the HST archive. These tasks include: - PixCteCorr: A stand-alone task to correct ACS _flt.fits files of the effects of CTE artifacts prior to further analysis or processing (such as cosmic-ray cleaning and combination by MultiDrizzle). - acs_destripe: A stand-alone task to remove the effects of row-to-row bias variations caused by the new electronics used to repair ACS during Servicing Mission 4 (SM4). Both of these tasks have been developed with TEAL interfaces, a GUI interface similar to the EPAR GUI under PYRAF without the dependence on IRAF or PyRAF. They also include fairly extensive Sphinx-based documentation to describe how to run the tasks. calcos ============= NOTE: This version (2.13.6) of calcos contains a bug that was discovered too late to be fixed for this release. If the input is a corrtag file for spectroscopic data, calcos will fail (KeyError: "Keyword 'ctype1' not found.") when trying to write the world coordinate system (WCS) keywords to the header of the flt or counts file. This can be worked around either by deleting keyword SPWCSTAB or by changing its value in the input corrtag header(s) to "N/A". The WCS keywords will then not be written. A bug which resulted in incorrect values in NUV x1dsum tables has been fixed. In some cases there was a gap in the data quality flags (the DQ extension in the flt and counts files) between the near-edge flag and the out-of-bounds flag. The data values were zero there, but because DQ was zero, the zero data values were included in the average in x1dsum. A bug was fixed in the code for comparing version strings. For example, 2.13.1 was not recognized as being later than 2.13. This comparison is done to check for compatibility between calcos and reference files. If the FLUXTAB does not have a row matching the observing configuration, calcos was quietly setting the values in the FLUX column to zero. Calcos now raises an exception if there is no matching row in the FLUXTAB. The stim count rate was not being computed correctly. This value is just printed for information pertaining to the deadtime correction. A new command-line argument --find was added. If specified, calcos will search for the target spectrum in the Y direction rather than relying completely on the offset from the wavecal spectrum. Options were added to the 1-D extraction to specify the location at which the spectrum should be extracted and/or the extraction height; these options are available via the x1dcorr task but not in calcos itself. For spectroscopic data, calcos now writes WCS keywords to the flt and counts images and to the corrtag table. This is disabled, however, if the SPWCSTAB keyword is missing or if its value is "N/A". There were changes in the names and interpretation of some of the data quality flags. In particular, one flag DQ_PHA_OUT_OF_BOUNDS is now used to flag pulse height out of bounds, rather than having separate flags for low pulse height and high pulse height. New columns NET and BACKGROUND are now written to the lampflash table. New segment-specific or stripe-specific keywords are used. Keyword SP_HGT has been changed to SP_HGT_A, SP_HGT_B, SP_HGT_C. Keywords EXPTIMEA, EXPTIMEB, NBADEVTA, NBADEVTB NEVENTSA, and NEVENTSB are now used for FUV. NEVENTS will be set to NEVENTSA + NEVENTSB in an x1d table. New keywords are now written when extracting a 1-D spectrum. SP_NOM_A, SP_NOM_B, SP_NOM_C are the nominal location (the expected value), and SP_OFF_A, SP_OFF_B, SP_OFF_C give the offset from nominal to where the spectrum was actually found. For most calibration steps for TIME-TAG data that use the exposure time, the value that will be used is the difference between the first and last events. The exposure time that has been corrected for time lost due to bursts and bad time intervals is used for converting counts to count rate. imagestats ============= Added Sphinx-based documentation to describe the code in this package. multidrizzle ============= Development on MultiDrizzle has focused on getting a replacement, currently named 'betadrizzle', tested and validated so that it can be publicly released. Only bug fixes required for continued use of MultiDrizzle in HST operations and the archive will be investigated and fixed until the replacement has been installed for use in HST operations and the archive. Only a few bug fixes required for pipeline use were implemented in MultiDrizzle since the last release; specifically, - the runfile now gets deleted when the parameter 'clean' is set to True - The changes to PyDrizzle's 'pydrizzle.distortion.mutil' module support the computation of the time-dependent distortion (TDD) terms based on arbitrary numbers of polynomial coefficients read in from the PRIMARY header of the IDCTAB. This allows separate solutions to be created for different epochs of data, including pre- and post-SM4 data. This change supports the use of updated versions of the IDCTAB reference files provided by the ACS team to support both pre- and post-SM4 data. - Logic for printing out warning message about using hard-coded TDD coefficients was corrected to allow the default values to be used for pre-SM4 data. A couple of small bugs affecting stand-alone use of MultiDrizzle by users were also addressed; namely, - the value of 'in_un' as written out to the runfile has both starting and ending quote marks to allow it to work as input to IRAF - Corrected a couple of typos in pydrizzle.process_input to address problems when trying to process a STIS association without the SPT files. This version of MultiDrizzle has been identified to contain a couple of bugs which will not be fixed in this code, although these issues have already been verified to work correctly in 'betadrizzle'. In particular, - WFC3/UVIS binned observations are not properly distortion corrected resulting in the chips being separated by over 2000 pixels instead of the expected ~10 pixels. - an small offset in the sky coordinates may be introduced into the output product when specifying a non-default output orientation. nictools ============= A new version of 'saaclean', Version 1.4(2010-10-20), includes all the changes necessary for creating an updated header that contains keywords derived from the input post-SAA-dark reference files, including the DATE keyword. numdisplay ============= A simple README file was added to the package to provide basic instructions for manually installing the numdisplay package. opuscoords ============= A problem with the inclusion of the 'from future' import statement in opuscoords was fixed to allow this code to be run in HST operations. PyFITS ============= This release contains Version 2.4.0 of PyFITS. - The following enhancements were added: - Checksum support now correctly conforms to the FITS standard. pyfits supports reading and writing both the old checksums and new standard-compliant checksums. The `fitscheck` command-line utility is provided to verify and update checksums. - Added a new optional keyword argument `do_not_scale_image_data` to the pyfits.open convenience function. When this argument is provided as True, and an ImageHDU is read that contains scaled data, the data is not automatically scaled when it is read. This option may be used when opening a fits file for update, when you only want to update some header data. Without the use of this argument, if the header updates required the size of the fits file to change, then when writing the updated information, the data would be read, scaled, and written back out in its scaled format (usually with a different data type) instead of in its non-scaled format. - Added a new optional keyword argument `disable_image_compression` to the pyfits.open function. When True, any compressed image HDU's will be read in like they are binary table HDU's. - Added a 'verify' keyword argument to the pyfits.append function. When False, append will assume the existing FITS file is already valid and simply append new content to the end of the file, resulting in a large speed up appending to large files. - Added HDU methods `update_ext_name` and `update_ext_version` for updating the name and version of an HDU. - Added HDU method `filebytes` to calculate the number of bytes that will be written to the file associated with the HDU. - Enhanced the section class to allow reading non-contiguous image data. Previously, the section class could only be used to read contiguous data. - Added method HDUList.fileinfo() that returns a dictionary with information about the location of header and data in the file associated with the HDU. - The following bugs were fixed: - Reading in some malformed FITS headers would cause a `NameError` exception, rather than information about the cause of the error. - pyfits can now handle non-compliant CONTINUE cards produced by Java FITS. - BinTable columns with TSCALn are now byte-swapped correctly. - Ensure that floating-point card values are no longer than 20 characters. - Updated flush so that when the data has changed in an HDU for a file opened in update mode, the header will be updated to match the changed data before writing out the HDU. - Allow hierarch cards to contain a keyword and value whose total character length is 69 characters. Previous length was limited at 68 characters. - Calls to FITS_rec['columnName'] now return an ndarray exactly the same as a call to FITS_rec.field('columnName') or FITS_rec.columnName. Previously, FITS_rec['columnName'] returned a much less useful fits_record object. - Corrected the append convenience function to eliminate the reading of the HDU data from the file that is being appended to. - Eliminated common symbols between the pyfitsComp module and the cfitsio and zlib libraries. These can cause problems on systems that use both PyFITS and cfitsio or zlib. PyRAF ============= PyRAF 1.10 has been released (bundled with stsci_python 2.11 as well as stand-alone). This release supports Python 2.5, 2.6, and 2.7. Since the 1.9 bundled release, the following enhancements have been made: - PyRAF commands can now be run as '-c' arguments to the 'pyraf' command, i.e. without entering the command-line shell. See also FAQ 2.9. - The 'unlearn' function is now supported for TEAL-enabled tasks. - The CL filename information has been added to any syntax errors thrown when PyRAF parses a CL file. - A change was made to EPAR (for TEAL-enabled tasks) to allow it to recognize sub-modules of previously imported task packages. - PyRAF can now be run when there is no widget library (i.e. Tkinter) installed on the node. As with no display, i.e. terminal interaction only, to run this way without error remember to setenv PYRAF_NO_DISPLAY. and the following bugs have been fixed: - Tabbed filename completion had an issue where PyRAF would list directory contents before they were requested by TAB. - The 'pyraf -h' command now correctly displays all the possible command-line options . - PyRAF was not elegantly handling when a corruption in the .pyraf_history file was found. - An unused feature was removed: PyRAF had been adding its bin directory to sys.path. - EPAR: There was a bug in the parameter interface code causing EPAR and TPAR to incorrectly save array-type parameters. A second bug during Save was fixed for cases where the 'uparm' variable is not correctly formed. Some EPAR GUI color fixes were made as well. - Plots were not being fully drawn after imexamine commands via ds9 when using the newly sped-up matplotlib graphics kernel. - A user found an issue with saveToFile() and with clear() when they had no installed /usr/bin/tput. - PyRAF installation notes state that the readline module is optional, but it was causing an exception when missing. In addition, code changes not affecting behavior are gradually being made to PyRAF in an effort to support the eventual transition to Python 3. This release includes such changes. pysynphot ============= The version of pysynphot distributed with this release is the version used by ETC 19.1. A number of data changes are included with this release - The Vega spectrum used to calculate vegamags has been updated to alpha_lyr_stis_005.fits - Wavecat entries have been added or modified to support new or changed COS obsmodes The following code changes were also made: - Support for new flux units microjy (mujy) and nanojy (njy) was added - The Box bandpass constructor now supports an optional waveunits keyword, to define a box in wavelength units other than Angstroms pytools ============= Sphinx documentation for many of the modules in PyTOOLS has been added to the pytools package, including the addition of the TEAL User Guide to describe how to add a TEAL interface to any Python task. fileutil: - The 'osfn()' function in 'pytools.fileutil' has been updated to return None if None is provided as input and to work properly for *nix style variable names in filenames. Examples of filenames that can now be expanded correctly when the variables are defined include: $PYSYN_CDBS/mtab/t2605492m_tmg.fits and mtab$t2605492m_tmg.fits. - A new function was added to 'pytools.fileutil', countExtn(), which returns the number of extensions with a given EXTNAME. This new function will be used in conjunction with the STWCS functions for restoring or archiving keyed-WCSs. teal: - A new function, getHelpFileAsString(), was added to pytools.teal to make it easier to implement TEAL GUI's for tasks. - The teal.teal() interface has an optional boolean arg named canExecute, where default=True. - Updated the mouse-wheel action to be move faster by default. - A new function, teal.print_tasknames(), was added to print out the names of all TEAL enabled tasks within a package. This can be used in the __init__ module of a package to automatically report all TEAL-enabled tasks to the user upon importing the package. - The set of keywords understood by TEAL in the configobj file was expanded to support more advanced behaviors. In particular, the "trigger" kwd was changed to "triggers", the "is_set_by" kwd was added to allow control by another parameter. - Some GUI changes were made for basic usability: status widget messages will be held and queued, browser-launching code is now more flexible on more platforms, and a bug with focusOut events (triggering rule actions) has been fixed. wcsutil: - The error messages when wcsutil.WCSObject raises an exception on reading a WCS have been refined to report the keyword which actually caused the failure. The bug and fix was reported by F. Stoehr(ESO). stwcs ============= This release contains Version 0.8 of the STWCS package, a package which provides suport for WCS based distortion models and coordinate transformations. It relies on PyWCS (based on WCSLIB). Bugs fixed: - The default value for 'skew_coeffs' in STWCS.distortion.mutil needed to be reset to None so that logic in STWCS.updatewcs.corrections can recognize that there are no TDD coefficients available from the IDCTAB for use. The default set of skew_coeffs in STWCS.updatewcs needed to be updated to include the TDDORDER key and value with a default value of 1 as appropriate for the hard-coded default coeffs. These changes were needed to allow betadrizzle/updatewcs to work correctly with IDCTAB files which do not yet have the TDD coeffs in the PRIMARY header. - Code which defines the reference chip based on instrument/detector combination was changed to work with FITS files with arbitrary number and order of extensions. - Fixed a problem with STIS observations caused by DATE-OBS being in each science extension instead of the primary header. - Fixed a bug related to different keywords in STIS MAMA and CCD observations. - Added 'SAMP' and 'TIME' to the list of EXTNAME keywords to support NICMOS observations. - A number of changes were made to keep in sync with changes in PyWCS. New Features: - Support for alternate WCS's in headers as described in WCS Paper I was added to STWCS.WCSUTIL. This includes functions for archiving, restoring and deleting an alternate WCS and listing the available WCS's by name and key. - When UPDATEWCS is run with a new IDCTAB, all WCS's except the original OPUS WCS are deleted now and a new one is computed. Documentation: - Added Sphinx based documentation. - An initial draft of a document defining the headerlet and how it will be used has been added to the STWCS Sphinx docs directory. - A draft of a FITS Conventions TSR was added to the documentation - A TSR on how the NPOL files were created and tested was finalized and added to the documentation.