sherpa/sherpa: CIAO 4.9

This is a CIAO release. Binaries will be provided with the next standalone release. Release notes are collected for all changes since CIAO 4.8, which may be included from multiple standalone releases. Release Notes Sherpa 4.9 now runs under both Python 2.7 and Python 3.5. The test infrastructure has been modified including simplification of the smoke test. Several bug fixes and enhancements are also included. Specific details are identified below. 14361: Sherpa and Chips wrapper need to replace execfile for python 3 compatibility This fix modifies the sherpa and chips wrapper scripts to replace the 'execfile' command with an 'exec' command sequence that allows the wrapper script to be utilized by python 2.7 or python 3.5. Without the change, the option to specify a command script at the sherpa or chips command prompt (ie. 'chips /pool1/runme.py') will not work on python 3.5 since the execfile does not exist in python 3.5. 107: Normalize plot labels. Plots created with plot_source used a different format to other plots when analysis=wavelength, in that LaTeX symbols were used for Angstrom and lambda (in other plots the string 'Angstrom' is used instead). The source plots now match the other plots. 138: improve and fix issues in save_all function. added a new argument to save_all: if outfile is None then the outfh argument is used to define the output handle (the argument can be any file-like argument, such as a file handle like sys.stdout or the output of open, or a StringIO object) setting the clobber argument to save_all now means that the output file (the outfile argument, if not None) is deleted if it already exists; prior to this, the file would be appended to instead the source expression is now saved correctly for most cases (e.g. when not using set_full_model); this is bug #97 but also affects non-PHA data sets the background model expression was not always written out correctly when using PHA data sets quality and grouping arrays of PHA data sets are now stored as 16-byte integers rather than a floating-point value (this has no affect on the results, but matches the OGIP standard) fixed up saving the grouping and quality arrays of background PHA data sets (this would only be an issue if the background is being fit, rather than subtracted) basic data sets created with the load_arrays function are now written out by save_all as part of the script; this is intended for small datasets and may have problems with precision if used with floating-point arrays calls to load_psf are now correctly restored (they may not have been written out correctly if multiple data sets were loaded) user models are now written out to disk; this consists of two parts: writing out the function that defines the model, which may or may not be possible (if not, a place-holder function is added to the output and a warning displayed). the necessary calls to load_user_model and add_user_pars are now included in the output the Python code created by save all has undergone several minor changes: it now explicitly imports the sherpa.astro.ui module, so that it can be run from the IPython prompt using the %run command, or directly as python it uses the create_model_component function rather than eval to create model components (this is CXC bug 12146) many optional arguments to functions are now given as name=value rather than being a positional argument, to make it clearer what the script is doing. calls to load_data have been replaced by more-specific versions - e.g. load_pha and load_image - if appropriate there have been several minor syntactic clean ups to better follow the suggestions from PEP8 When writing out code that defines a user-model, there is no attempt to make sure that modules used by the function are available. These will need to be added, either directly or imported, manually to the output. 153: Minor bug with calc_chi2datavar_errors (Fix #148). Make comparison test in calc_chi2datavar_errors less stringent, so to include the case where sqrt(x)=0. 155: Add argument to get_draws for supplying a covariance matrix. The get_draws function now accepts a user-provided covariance matrix. If no covariance matrix is provided, the covariance matrix computed by the default implementation is used. Note that covar() must be invoked before invoking get_draws if no covariance matrix is provided, otherwise get_draws will exit with an error. 165: Remove usage of deprecated numpy API. 185: Protect XPA command to avoid shell confusion. Fix the problem where if the working directory contained a file called x or y then the sherpa.astro.ui.image_data() function would fail with the message DS9Err: Could not display image 187: Issue a more meaningful message when sherpa.astro.io is imported directly and no fits backends are available. (Fix #92). 190: Datastack can be used if no plotter available. The datastack package can now be used even if there is no available plotting backend. In this case, plotting functions will not be available, but the rest of the datastack functionality will. (Fix #22). 195 Generalize calc_stat API + example of how to have several datasets with different fit statistics. Attempts to generalize the `calc_stat` API and enable simultaneous fits with different statistics for different data sets. 209: Fix docstrings for group_snr() and group_adapt_snr(). Updates to the docstrings for clarity. 210: New Smoke Test The smoke test has been greatly simplified: rather than running all the unit and regression tests that do not require test data, the smoke test now simply ensures that the basic installation works, i.e. that basic commands can be run and that dependencies can be reached. 211: Cleanup of documentation and code in sherpa.astro.utils. The calc_kcorr function is now exported by sherpa.astro.utils. Minor changes to the documentation in sherpa.astro.utils were also made to conform to Sphinx standards. 221 Add model documentation (Fix #217). Integrate existing model documentation (from external sources and the CIAO ahelp documentation system) into the model classes. 229: Code is both Python 2.7 and 3.5 compliant. (Fix #76). 242: Avoid use of inspect.argspec in Python 3 This change replaces the deprecated 'inspect.argspec' call with 'inspect.signature'. 252: Fix plot_photon_flux function. (Fix #241). A bug where plotting photon flux was fixed by adding a missing argument to the sample_photon_flux call. 253 Make sure background is taken into account in calc_stat_info (Fix #147). calc_stat_info call failed when wstat was selected, as the background was not taken into account. This issue has now been fixed. 254 Fix the documentation for set_rmf (Fix #236). The documentation for set_rmf incorrectly referred to ARF rather than RMF. 256 Fix docstring in set_quality (Fix #205). The docstring in set_quality now correctly indicates the quality flags. The previous documentation didn't describe the values of such flags properly. 257 Fix docstring for levmar tolerance (Fix #257). The documentation string for the Levenberg-Marquardt optimization function now correctly states that the parameter default values are equal to the single precision epsilon, rather than the square root of the double precision epsilon. 263 List_data_ids() fails on py3 with mixed id types (Fix #262). Sherpa was sorting the list of dataset IDs in a non-python3 compliant fashion, which resulted in issues when using strings and integers together as dataset IDs. This has now been fixed. 267 add wstat tests Add several regression tests for wstat. 282 Parallel_map not working on py3 with numcores=1 (Fix #277). The utils function parallel_map failed on Python 3 when called with numcores=1, i.e. on systems with only one processor/core. This has been fixed. 283 Sample flux and numpy deprecations (Fix #273 and #276). The sample_flux function was not working under Python 3 if the scales argument was provided. This has been fixed. Also, a DeprecationWarning was issued by numpy because during the sample_flux execution values were extracted from an array with non-integer indices. This has also been fixed. 284 String representation of data classes under py3 (Fix #275). Data classes DataPHA, DataARF, DataRMF, DataIMG, and DataIMGInt in sherpa.astro.data would throw an exception if users tried to print them as strings, under Python 3. This has been fixed. 287 Rewrite sherpa.stats.Stat.calc_stat and simplify sherpa.fit.Fit (fix #227 #248 #289 #292). In order to fix several issues related to the WStat support, and in order to make the code more maintainable, the sherpa.stats.Stat.calc_stat and sherpa.fit.Fit classes have gone through a round of refactoring. This fixes the following issues: #227 Issues using wstat when grouping/filtering data; #248 backscal column not treated properly for WStat; #289 calc_stat does not error out if background subtracted data is used with Likelihood statistics; #292 stat info does not include reduced stat/qval for wstat. 295 Fix display of pileup model in Python 3.5 (Fix #294). Fix display of instances of sherpa.astro.models.JDPileup so that, in Python 3.5, they can be displayed after the model has been evaluated. 304 replace file -> open (Fix #297). The save and restore functions used to use the file function which is not compatible with Python 3. This has now been fixed. 305 Fix python 3 issues with some session commands (Fix #303). The set_xlog, set_ylog, and show_bkg_model functions were not compatible with Python 3. This has now been fixed (Issue #303). 307 Move XSPEC table support to load_xstable_model and deprecate its support in load_table_model (Fix #270). Add the load_xstable_model routine to the sherpa.astro.ui module, which supports loading XSPEC additive or multiplicative (atable and mtable) models. The support for these models is still available via load_table_model in this release, but it is deprecated. The read_xstable_model routine has been added to the sherpa.astro.xspec module. 312 Fix over-zealous code clean up in PR #287 affecting sigmarej. Fits using the sigmarej iterated-fit method were broken if a filter had been applied to the data before the fit and there are any bins that get ignored at larger bin values than the filtered-out data. (This fixes a subtle regression introduced by #287). 313 Allow sequence=None when using gridsearch and Python 3.5 (Fix #309). Allow the gridsearch optimiser to be used with the sequence option set to None for Python 3.5. Caveats 319: image_getregion returns byte string on Py3. SH-2: The new test_save_restore test in the CIAO regression tests suite is failing on all platforms. We are investigating the reasons of the failure. The failure is triggered when the test is not run in isolation, and only when certain other tests are run before it. Also, this only applies to CIAO and not to standalone Sherpa. SH-3: Some tests are skipped during the CIAO regression tests. SH-4: Several OS X regression tests are failing. Note: The SH-2/3/4 caveats are issues with the tests themselves, not with the code, and only appear when running the full CIAO regression tests suite. User will not be affected by the above issues unless they run the full CIAO regression tests suite