fitting_summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of fitting experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tested:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- astropy.modeling.fitting\n",
    "  - Levenberg-Marquardt\n",
    "  - Simplex\n",
    "  - SLSQP\n",
    "- scipy.optimize.curve_fit\n",
    "- lmfit\n",
    "- ODRPACK\n",
    "- saba (Sherpa)\n",
    "  - Levenberg-Marquardt\n",
    "  - Moncar\n",
    "  - Nelder-Mead\n",
    "  \n",
    "Saba requires python 2.7, thus can't be used by specviz.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some features of these fitters:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Fitter             | method       | calls                  | depends on    | fix | ties | bounds | \n",
    "|:-------------------|:-------------|:-----------------------|:--------------|:---:|:----:|:------:|\n",
    "| astropy L-M        |              | optimize.leastsq       | MINPACK lmdif |   X |  X   |        |\n",
    "| astropy simplex    |              | optimize.fmin          |               |   X |  X   |        | \n",
    "| optimize.curve_fit | lm (default) | optimize.leastsq       | MINPACK lmdif |     |      |        |\n",
    "| optimize.curve_fit | trf          | optimize.least_squares |               |     |      |   X    |\n",
    "| optimize.curve_fit | dogbox       | optimize.least_squares |               |     |      |   X    | \n",
    "| odr                | lm (1)       |                        |               |   X |      |        | \n",
    "| lmfit              | several      | optimize.XXX           |               |     |  X   |   X    |\n",
    "\n",
    "(1) defaults to its own implementation of L-M when errors in the independent variable are ignored"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4> CURVE_FITt: </h4> is basically a wrapper around optimize.leastsq and optimize.least_squares, to facilitate their use in simpler problems (simplified calling sequence). \n",
    "\n",
    "optimize.least_squares supports bounded parameters when running the trf (Trust Region Reflective) and dogbox (rectangular trust regions) methods. Note that our experiment fails when using bounds, both in ROI 1 and ROI 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>LMFIT:</h4> Entry point for a large number of scipy optimizers. The default (as tested) reverts to L-M since it calls optimize.leastsq (minpack) via optimize.least_squares. \n",
    "\n",
    "Supported methods: L-M, simplex, L-BFGS-B , Powell, conjugate gradient, Newton-CG, COBYLA, truncated Newton, dogleg, Sequential Linear Squares programming, Differential Evolution, brute force (all from scipy.optimize).\n",
    "\n",
    "This would be an interesting avenue to pursue since it offers so large a variety of optimizers. It also supports bounded parameters, which is not a feature offered by the astropy fitters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4> ODR: </h4> Orthogonal Distance Regression is designed to handle the case where errors are present in both the independent and dependent variables. This is probably not the most common use case in spectroscopy in the UV-IR band. ODRPACK uses a modified trust-region L-M optimizer. According to the original paper, it falls back into ordinary L-M (as in optimize.leastsq) when errors in wavelength are to be ignored. Like astropy's fitters, it supports constrained (fixed) parameters, but no bounded parameters.\n",
    "\n",
    "This is the only tested software that has an implementation of basic L-M that is independent from the one offered in optimize.leastsq/MINPACK. This could be grounds for including it in the repertoire of specviz's fitters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitter behavior:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From tests in https://github.com/ibusko/experiments_specviz/blob/master/experiment_fiting.ipynb and https://github.com/ibusko/experiments_specviz/blob/master/curve_fit.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "|Fitter             | Initialization    |    ROI 1             |     ROI 2           | \n",
    "|:------------------|:------------------|:---------------------|:--------------------|\n",
    "| astropy L-M       | independent       | FIT WORKS            | FIT WORKS           |\n",
    "| astropy L-M       | compound          | FIT WORKS            | FIT WORKS           |\n",
    "| astropy L-M       | default           | moves closer         | moves closer        |\n",
    "|                   |                   |                      |                     |\n",
    "| astropy Simplex   | independent       | FIT WORKS            | moves closer        |\n",
    "| astropy Simplex   | compound          | FIT WORKS            | moves closer        |\n",
    "| astropy Simplex   | default           | Gaussian disappears  | Gaussian disappears |\n",
    "|                   |                   |                      |                     |\n",
    "| astropy SLSQP     | independent       | does nothing         | does nothing        |\n",
    "| astropy SLSQP     | compound          | does nothing         | does nothing        |\n",
    "| astropy SLSQP     | default           | diverges             | diverges            |\n",
    "|                   |                   |                      |                     |\n",
    "| scipy curve_fit   | default           | FIT WORKS            | FIT WORKS           |\n",
    "| scipy curve_fit   | compound          | never converges      | never converges     |\n",
    "|                   |                   |                      |                     |\n",
    "| least_squares TRF | compound, unbound | FIT WORKS            | not tested          |\n",
    "| least_squares TRF | compound, bounded | moves closer         | moves closer        |\n",
    "|                   |                   |                      |                     |\n",
    "| lmfit             | independent       | FIT WORKS            | FIT WORKS           |\n",
    "| lmfit             | compound          | FIT WORKS            | doesn't work        |\n",
    "|                   |                   |                      |                     |\n",
    "| odr               | compound          | FIT WORKS            | doesn't work        |\n",
    "|                   |                   |                      |                     |\n",
    "| Sherpa L-M        | independent       | FIT WORKS            | FIT WORKS           |\n",
    "| Sherpa L-M        | compound          | FIT WORKS            | FIT WORKS           |\n",
    "| Sherpa L-M        | default           | moves closer         | moves closer        |\n",
    "|                   |                   |                      |                     |\n",
    "| Sherpa Moncar     | independent       | FIT WORKS            | moves closer        |\n",
    "| Sherpa Moncar     | compound          | FIT WORKS            | moves closer        |\n",
    "| Sherpa Moncar     | default           | Gaussian disappears  | Gaussian disappears |\n",
    "|                   |                   |                      |                     |\n",
    "| Sherpa N-M        | independent       | moves closer         | FIT WORKS           |\n",
    "| Sherpa N-M        | compound          | moves closer         | FIT WORKS           |\n",
    "| Sherpa N-M        | default           | diverges             | diverges            |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"Initialization\" column says how the first guesses were defined:\n",
    "\n",
    "  - independent: this is how specviz does it\n",
    "  - compound: all five initial guesses are found at the same time\n",
    "  - default: whatever the fitter provides as initial values\n",
    "  \n",
    "ROI 1 contains only good data.\n",
    "\n",
    "ROI 2 contains bad data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}