# Minimizer Options

This section is used to declare the minimizers to use for each fitting software. If a fitting software has been selected in Fitting Options then a default set of minimizers for that solver will be run unless alternative minimizer options have been set. All minimizers for a software are included on the default list of minimizers unless otherwise stated.

Warning

Options set in this section will only have an effect if the related software is also set in Fitting Options (either explicitly, or as a default option).

## Bumps (`bumps`

)

Bumps is a set of data fitting (and Bayesian uncertainty analysis) routines.
It came out of the University of Maryland and NIST as part of the DANSE
(*Distributed Data Analysis of Neutron Scattering Experiments*) project.

FitBenchmarking currently supports the Bumps minimizers:

Nelder-Mead Simplex (

`amoeba`

)Levenberg-Marquardt (

`lm-bumps`

) This is mpfit, a translation of MINPACK to Python.Quasi-Newton BFGS (

`newton`

)scipy’s leastsq (

`scipy-leastsq`

) This calls scipy’s Levenberg-Marquardt method. Note that this was the default method for lm prior to Bumps v0.8.2.

**Licence** The main licence file for Bumps is here. Individual files have their own copyright and licence
– if you plan to incorporate this in your own software you should first check that the
licences used are compatible.

**Links** GitHub - bumps

The Bumps minimizers are set as follows:

```
[MINIMIZERS]
bumps: amoeba
lm-bumps
newton
de
scipy-leastsq
dream
```

Warning

The additional dependency Bumps must be installed for this to be available; See Extra dependencies.

Note

de is not included in the default list of minimizers for bumps. To run this solver, you must explicitly set the minimizer as seen above.

## Ceres Solver (`ceres`

)

Ceres Solver is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems.

FitBenchmarking currently supports the Ceres Solver minimizers:

Levenberg-Marquardt (

`Levenberg-Marquardt`

)Dogleg (

`Dogleg`

)Steepest Descent (

`steepest_descent`

)BFGS algorithm (

`BFGS`

)LBFGS algorithm (

`LBFGS`

)Fletcher-Reeves Non Linear Conjugate-Gradient (

`Fletcher_Reeves`

)Polak-Ribiere Non Linear Conjugate-Gradient (

`Polak_Ribiere`

)Hestenes-Stiefel Non Linear Conjugate-Gradient (

`Hestenes_Stiefel`

)

**Licence** Ceres Solver is available under the new BSD licence – details can be found here

**Links** Ceres Solver PyCeres - Ceres Python Bindings

The Ceres Solver minimizers are set as follows:

```
[MINIMIZERS]
ceres: Levenberg_Marquardt
Dogleg
BFGS
LBFGS
steepest_descent
Fletcher_Reeves
Polak_Ribiere
Hestenes_Stiefel
```

Warning

The additional dependency Ceres Solver must be installed for this to be available; See Extra dependencies.

Note

The PyCeres currently only works with Ceres Solver versions 2.0.0

## DFO (`dfo`

)

There are two Derivative-Free Optimization packages, DFO-LS and DFO-GN. They are derivative free optimization solvers that were developed by Lindon Roberts at the University of Oxford, in conjunction with NAG. They are particularly well suited for solving noisy problems.

FitBenchmarking currently supports the DFO minimizers:

**Licence** Both DFO-GN and DFO-LS are available under the GPL-3 licence. A proprietary licence is also available from NAG .

**Links** GitHub - DFO-GN GitHub - DFO-LS

The DFO minimizers are set as follows:

```
[MINIMIZERS]
dfo: dfols
dfogn
```

Warning

Additional dependencies DFO-GN and DFO-LS must be installed for these to be available; See Extra dependencies.

## GOFit (`gofit`

)

GOFit is a package of C++ algorithms with Python interfaces designed for the global optimization of parameters in curve fitting, i.e. for nonlinear least-squares problems arising from curve fitting. It is also included with Mantid since release 6.5.

FitBenchmarking currently supports the GOFit minimizers:

Multistart Global Minimizer (

`multistart`

)Alternating Multistart Global Minimizer (

`alternating`

)Quadratic Regularisation Local Minimizer (

`regularisation`

)

**Links** Documentation

**Licence** GOFit is available under a 3-clause BSD Licence

The GOFit minimizers are set as follows:

```
[MINIMIZERS]
gofit: multistart
alternating
regularisation
```

Note

The alternating minimizer currently only supports Crystal Field problems.

Warning

The additional dependency GOFit must be installed to use these minimizers. See Extra dependencies.

## Gradient-Free-Optimizers (`gradient_free`

)

Gradient-Free-Optimizers are a collection of gradient-free methods capable of solving various optimization problems. Please note that Gradient-Free-Optimizers must be run with problems that have finite bounds on all parameters.

Hill Climbing (

`HillClimbingOptimizer`

)Repulsing Hill Climbing (

`RepulsingHillClimbingOptimizer`

)Simulated Annealing (

`SimulatedAnnealingOptimizer`

)Random Search (

`RandomSearchOptimizer`

)Random Restart Hill Climbing (

`RandomRestartHillClimbingOptimizer`

)Random Annealing (

`RandomAnnealingOptimizer`

)Parallel Tempering (

`ParallelTemperingOptimizer`

)Particle Swarm (

`ParticleSwarmOptimizer`

)Evolution Strategy (

`EvolutionStrategyOptimizer`

)Bayesian (

`BayesianOptimizer`

)Tree Structured Parzen Estimators (

`TreeStructuredParzenEstimators`

)Decision Tree (

`DecisionTreeOptimizer`

)

**Licence** The Gradient-Free-Optimizers package is available under an MIT Licence .

The gradient_free minimizers are set as follows:

```
[MINIMIZERS]
gradient_free: HillClimbingOptimizer
RepulsingHillClimbingOptimizer
SimulatedAnnealingOptimizer
RandomSearchOptimizer
RandomRestartHillClimbingOptimizer
RandomAnnealingOptimizer
ParallelTemperingOptimizer
ParticleSwarmOptimizer
EvolutionStrategyOptimizer
BayesianOptimizer
TreeStructuredParzenEstimators
DecisionTreeOptimizer
```

Warning

The additional dependency Gradient-Free-Optimizers must be installed for this to be available; See Extra dependencies.

Note

BayesianOptimizer, TreeStructuredParzenEstimators and DecisionTreeOptimizer may be slow running and so are not run by default when gradient_free software is selected. To run these minimizers you must explicity set them as seen above.

## GSL (`gsl`

)

The GNU Scientific Library is a numerical library that provides a wide range of mathematical routines. We call GSL using the pyGSL Python interface.

The GSL routines have a number of parameters that need to be chosen, often without default suggestions. We have taken the values as used by Mantid.

We provide implementations for the following packages in the multiminimize and multifit sections of the library:

Levenberg-Marquardt (unscaled) (

`lmder`

)Levenberg-Marquardt (scaled) (

`lmsder`

)Nelder-Mead Simplex Algorithm (

`nmsimplex`

)Nelder-Mead Simplex Algorithm (version 2) (

`nmsimplex2`

)Polak-Ribiere Conjugate Gradient Algorithm (

`conjugate_pr`

)Fletcher-Reeves Conjugate-Gradient (

`conjugate_fr`

)The vector quasi-Newton BFGS method (

`vector_bfgs`

)The vector quasi-Newton BFGS method (version 2) (

`vector_bfgs2`

)Steepest Descent (

`steepest_descent`

)

**Links** SourceForge PyGSL

**Licence** The GNU Scientific Library is available under the GPL-3 licence .

The GSL minimizers are set as follows:

```
[MINIMIZERS]
gsl: lmsder
lmder
nmsimplex
nmsimplex2
conjugate_pr
conjugate_fr
vector_bfgs
vector_bfgs2
steepest_descent
```

Warning

The external packages GSL and pygsl must be installed to use these minimizers.

## Horace (`horace`

)

Horace is described as *a suite of programs for
the visiualization and analysis from time-of-flight neutron inelastic scattering
spectrometers.* We currently support:

Levenberg-Marquardt (

`lm-lsqr`

)

**Licence** Matlab must be installed to use Horace within FitBenchmarking, which is a
proprietary product.
Horace is made available under the the GPL-3 licence.

```
[MINIMIZERS]
horace: lm-lsqr
```

Note

If you have a non standard installation of Horace please set the HORACE_LOCATION (e.g on IDAaaS).

Warning

The Horace Toolbox and MATLAB must be installed for this to be available; see Installing External Software.

## Mantid (`mantid`

)

Mantid is a framework created to manipulate and analyze neutron scattering and muon spectroscopy data. It has support for a number of minimizers, most of which are from GSL.

BFGS (

`BFGS`

)Conjugate gradient (Fletcher-Reeves) (

`Conjugate gradient (Fletcher-Reeves imp.)`

)Conjugate gradient (Polak-Ribiere) (

`Conjugate gradient (Polak-Ribiere imp.)`

)Damped GaussNewton (

`Damped GaussNewton`

)FABADA (

`FABADA`

)Levenberg-Marquardt algorithm (

`Levenberg-Marquardt`

)Levenberg-Marquardt MD (

`Levenberg-MarquardtMD`

) - An implementation of Levenberg-Marquardt intended for MD workspaces, where work is divided into chunks to achieve a greater efficiency for a large number of data points.Simplex (

`Simplex`

)SteepestDescent (

`SteepestDescent`

)Trust Region (

`Trust Region`

) - An implementation of one of the algorithms available in RALFit.

**Licence** Mantid is available under the GPL-3 licence .

The Mantid minimizers are set as follows:

```
[MINIMIZERS]
mantid: BFGS
Conjugate gradient (Fletcher-Reeves imp.)
Conjugate gradient (Polak-Ribiere imp.)
Damped GaussNewton
FABADA
Levenberg-Marquardt
Levenberg-MarquardtMD
Simplex
SteepestDescent
Trust Region
```

Warning

The external package Mantid must be installed to use these minimizers.

## Levmar (`levmar`

)

The levmar package which implements the Levenberg-Marquardt method for nonlinear least-squares. We interface via the python interface available on PyPI.

Levenberg-Marquardt with supplied Jacobian (

`levmar`

) - the Levenberg-Marquardt method

**Licence** Levmar is available under the GPL-3 licence . A paid licence for proprietary commerical use is available from the author .

The levmar minimizer is set as follows:

```
[MINIMIZERS]
levmar: levmar
```

Warning

The additional dependency levmar must be installed for this to be available; See Extra dependencies. This package also requires the BLAS and LAPACK libraries to be present on the system.

## LMFIT (`lmfit`

)

The lmfit package provides simple tools to help you build complex fitting models for non-linear least-squares problems and apply these models to real data. Lmfit provides a high-level interface to non-linear optimization and curve fitting problems for Python. It builds on and extends many of the optimization methods of scipy.optimize.

Levenberg-Marquardt (

`leastsq`

)Least-Squares minimization, using Trust Region Reflective method (

`least_squares`

)Differential evolution (

`differential_evolution`

)Adaptive Memory Programming for Global Optimization (

`ampgo`

)Nelder-Mead (

`nelder`

)L-BFGS-B (

`lbfgsb`

)Powell (

`powell`

)Conjugate-Gradient (

`cg`

)Newton-CG (

`newton`

)Cobyla (

`cobyla`

)BFGS (

`bfgs`

)Truncated Newton (

`tnc`

)Newton-CG trust-region (

`trust-ncg`

)Nearly exact trust-region (

`trust-exact`

)Newton GLTR trust-region (

`trust-krylov`

)Trust-region for constrained optimization (

`trust-constr`

)Dog-leg trust-region (

`dogleg`

)Sequential Linear Squares Programming (

`slsqp`

)Simplicial Homology Global Optimization (

`shgo`

)Dual Annealing optimization (

`dual_annealing`

)

**Licence** LMFIT is available the new BSD-3 licence – details can be found here

The lmfit minimizer is set as follows:

```
[MINIMIZERS]
lmfit: differential_evolution
powell
cobyla
slsqp
emcee
nelder
least_squares
trust-ncg
trust-exact
trust-krylov
trust-constr
dogleg
leastsq
newton
tnc
lbfgsb
bfgs
cg
ampgo
shgo
dual_annealing
```

Note

The shgo solver is particularly slow running and should generally be avoided. As a result, this solver is not run by default when lmfit software is selected. In order to run this minimizer, you must explicitly set it as above.

Warning

emcee uses a Markov Chain Monte Carlo package and assumes that the prior is Uniform. This may not perform well for certain fitting problems.

## Matlab (`matlab`

)

We call the fminsearch function from MATLAB, using the MATLAB Engine API for Python.

Nelder-Mead Simplex (

`Nelder-Mead Simplex`

)

**Licence** Matlab is a proprietary product .

The matlab minimizer is set as follows:

```
[MINIMIZERS]
matlab: Nelder-Mead Simplex
```

Warning

MATLAB must be installed for this to be available; See Installing External Software.

## Matlab Curve Fitting Toolbox (`matlab_curve`

)

We call the fit function from the MATLAB Curve Fitting Toolbox, using the MATLAB Engine API for Python.

Levenberg-Marquardt (

`Levenberg-Marquardt`

)Trust-Region (

`Trust-Region`

)

**Licence** Matlab and the Curve Fitting Toolbox are both proprietary products .

The matlab_curve minimizers are set as follows:

```
[MINIMIZERS]
matlab_curve: Levenberg-Marquardt
Trust-Region
```

Warning

MATLAB Curve Fitting Toolbox must be installed for this to be available; See Installing External Software.

## Matlab Optimization Toolbox (`matlab_opt`

)

We call the lsqcurvefit function from the MATLAB Optimization Toolbox, using the MATLAB Engine API for Python.

Levenberg-Marquardt (

`levenberg-marquardt`

)Trust-Region-Reflective (

`trust-region-reflective`

)

**Licence** Matlab and the Optimization Toolbox are both proprietary products .

The matlab_opt minimizers are set as follows:

```
[MINIMIZERS]
matlab_opt: levenberg-marquardt
trust-region-reflective
```

Warning

MATLAB Optimization Toolbox must be installed for this to be available; See Installing External Software.

## Matlab Statistics Toolbox (`matlab_stats`

)

We call the nlinfit function from the MATLAB Statistics Toolbox, using the MATLAB Engine API for Python.

Levenberg-Marquardt (

`Levenberg-Marquardt`

)

**Licence** Matlab and the Statistics Toolbox are both proprietary products .

The matlab_stats minimizer is set as follows:

```
[MINIMIZERS]
matlab_stats: Levenberg-Marquardt
```

Warning

MATLAB Statistics Toolbox must be installed for this to be available; See Installing External Software.

## Minuit (`minuit`

)

CERN developed the Minuit 2 package to find the minimum value of a multi-parameter function, and also to compute the uncertainties. We interface via the python interface iminuit with support for the 2.x series.

Minuit’s MIGRAD (

`migrad`

)Minuit’s SIMPLEX (

`simplex`

)

**Links** Github - iminuit

**Licence** iminuit is released under the MIT licence, while Minuit 2 is LGPL v2 .

The Minuit minimizers are set as follows:

```
[MINIMIZERS]
minuit: migrad
simplex
```

Warning

The additional dependency Minuit must be installed for this to be available; See Extra dependencies.

## NLopt (`nlopt`

)

NLopt is a free/open-source library for nonlinear optimization, providing a common interface for a number of different free optimization routines available online as well as original implementations of various other algorithms

Bound Optimization BY Quadratic Approximation (

`LN_BOBYQA`

)NEW Unconstrained Optimization Algorithm (

`LN_NEWUOA`

)NEW Unconstrained Optimization Algorithm, bound algorithm (

`LN_NEWUOA_BOUND`

)Principal-axis method (

`LN_PRAXIS`

)Sequential Least Squares Programming (

`LD_SLSQP`

)Shifted limited-memory variable-metric (rank 1) (

`LD_VAR1`

)Shifted limited-memory variable-metric (rank 2) (

`LD_VAR2`

)Augmented Lagrangian local (

`AUGLAG`

)Augmented Lagrangian local (equality constraints) (

`AUGLAG_EQ`

)Nelder-Mead Simplex (

`LN_NELDERMEAD`

)Subplex (

`LN_SBPLX`

)Constrained Optimization BY Linear Approximations (

`LN_COBYLA`

)Conservative convex separable approximation (

`LD_CCSAQ`

)Method of Moving Asymptotes (

`LD_MMA`

)Newton (

`LD_TNEWTON`

)Newton preconditioned by the low-storage BFGS algorithm (

`LD_TNEWTON_PRECOND`

)Newton with steepest-descent restarting (

`LD_TNEWTON_RESTART`

)Newton preconditioned by the low-storage BFGS algorithm with steepest-descent restarting (

`LD_TNEWTON_PRECOND_RESTART`

)LBFGS, and derivative-free algorithm (

`LD_LBFGS`

)DIviding RECTangles (

`GN_DIRECT`

)DIviding RECTangles (locally biased) (

`GN_DIRECT_L`

)DIviding RECTangles (locally biased which uses some randomization) (

`GN_DIRECT_L_RAND`

)DIviding RECTangles (unscaled variant) (

`GNL_DIRECT_NOSCAL`

)DIviding RECTangles (locally biased and unscaled variant) (

`GN_DIRECT_L_NOSCAL`

)DIviding RECTangles (locally biased, unscaled variant which uses some randomization) (

`GN_DIRECT_L_RAND_NOSCAL`

)DIviding RECTangles (based on the original Fortran code by Gablonsky et al. (1998-2001)) (

`GN_ORIG_DIRECT`

)DIviding RECTangles (based on the original Fortran code by Gablonsky et al. (1998-2001) and locally biased ) (

`GN_ORIG_DIRECT_L`

)Controlled Random Search (

`GN_CRS2_LM`

)Multi-Level Single-Linkage, low-discrepancy sequence (

`G_MLSL_LDS`

)Multi-Level Single-Linkage (

`G_MLSL`

)Stochastic Global Optimization (

`GD_STOGO`

)Stochastic Global Optimizatiom, randomized variant (

`GD_STOGO_RAND`

)AGS (

`GN_AGS`

)Improved Stochastic Ranking Evolution Strategy (

`GN_ISRES`

)

The Nlopt minimizers are set as follows:

```
[MINIMIZERS]
nlopt: LN_BOBYQA
LN_NEWUOA
LN_NEWUOA_BOUND
LN_PRAXIS
LD_SLSQP
LD_VAR2
LD_VAR1
AUGLAG
AUGLAG_EQ
LN_NELDERMEAD
LN_SBPLX
LN_COBYLA
LD_CCSAQ
LD_MMA
LD_TNEWTON_PRECOND_RESTART
LD_TNEWTON_PRECOND
LD_TNEWTON_RESTART
LD_TNEWTON
LD_LBFGS
GN_DIRECT
GN_DIRECT_L
GN_DIRECT_L_RAND
GNL_DIRECT_NOSCAL
GN_DIRECT_L_NOSCAL
GN_DIRECT_L_RAND_NOSCAL
GN_ORIG_DIRECT
GN_ORIG_DIRECT_L
GN_CRS2_LM
G_MLSL_LDS
G_MLSL
GD_STOGO
GD_STOGO_RAND
GN_AGS
GN_ISRES
```

Note

The global optimization solvers are not run by default when nlopt software is selected. In order to run these minimizers, you must explicitly set them as above.

Note

The following 4 minimizers need a local optimizer selected to run. This has been set to use LD_LBFGS.

```
[MINIMIZERS]
nlopt: AUGLAG
AUGLAG_EQ
G_MLSL_LDS
G_MLSL
```

## RALFit (`ralfit`

)

RALFit is a nonlinear least-squares solver, the development of which was funded by the EPSRC grant Least-Squares: Fit for the Future. RALFit is designed to be able to take advantage of higher order derivatives, although only first order derivatives are currently utilized in FitBenchmarking.

Gauss-Newton, trust region method (

`gn`

)Hybrid Newton/Gauss-Newton, trust region method (

`hybrid`

)Newton, trust region method (

`newton`

)Newton-tensor, trust region method (

`newton-tensor`

)Gauss-Newton, regularization (

`gn_reg`

)Hybrid Newton/Gauss-Newton, regularization (

`hybrid_reg`

)Newton, regularization (

`newton_reg`

)Newton-tensor, regularization (

`newton-tensor_reg`

)

Note that the Newton-tensor methods take significantly longer than the other options to run (but may give a better solution in some cases). For this reason, they are not included in the default minimizers for RALFit, but must be turned on in the options file.

**Links** Github - RALFit. RALFit’s Documentation on: Gauss-Newton/Hybrid models, the trust region method and The regularization method

**Licence** RALFit is available under a 3-clause BSD Licence

The RALFit minimizers are set as follows:

```
[MINIMIZERS]
ralfit: gn
gn_reg
hybrid
hybrid_reg
```

Warning

The external package RALFit must be installed to use these minimizers.

## SciPy (`scipy`

)

SciPy is the standard python package for mathematical software. In particular, we use the minimize solver for general minimization problems from the optimization chapter of SciPy’s library. Currently we only use the algorithms that do not require Hessian information as inputs.

Nelder-Mead algorithm (

`Nelder-Mead`

)Powell algorithm (

`Powell`

)BFGS algorithm (

`BFGS`

)Newton-CG algorithm (

`Newton-CG`

)L-BFGS-B algorithm (

`L-BFGS-B`

)Newton-CG trust-region (

`trust-ncg`

)Nearly exact trust-region (

`trust-exact`

)Newton Generalized Lanczos Trust Region (

`trust-krylov`

)Trust-region for constrained optimization (

`trust-constr`

)Dog-leg trust-region (

`dogleg`

)

**Links** Github - SciPy minimize

**Licence** Scipy is available under a 3-clause BSD Licence. Individual packages may have their own (compatible) licences, as listed here.

The SciPy minimizers are set as follows:

```
[MINIMIZERS]
scipy: Nelder-Mead
Powell
CG
BFGS
Newton-CG
L-BFGS-B
TNC
SLSQP
COBYLA
trust-ncg
trust-exact
trust-krylov
trust-constr
dogleg
```

Note

The Hessian enabled solvers are not run by default when scipy software is selected. In order to run these minimizers, you must explicitly set them as above.

## SciPy LS (`scipy_ls`

)

SciPy is the standard python package for mathematical software. In particular, we use the least_squares solver for Least-Squares minimization problems from the optimization chapter of SciPy’s library.

Levenberg-Marquardt with supplied Jacobian (

`lm-scipy`

) - a wrapper around MINPACKThe Trust Region Reflective algorithm (

`trf`

)A dogleg algorithm with rectangular trust regions (

`dogbox`

)

**Links** Github - SciPy least_squares

**Licence** Scipy is available under a 3-clause BSD Licence. Individual packages many have their own (compatible) licences, as listed here.

The SciPy least squares minimizers are set as follows:

```
[MINIMIZERS]
scipy_ls: lm-scipy
trf
dogbox
```

## SciPy GO (`scipy_go`

)

SciPy is the standard python package for mathematical software. In particular, we use the Global Optimization solvers for global optimization problems from the optimization chapter of SciPy’s library.

Differential Evolution (derivative-free) (

`differential_evolution`

)Dual Annealing (

`dual_annealing`

)

**Links** Github - SciPy optimization

**Licence** Scipy is available under a 3-clause BSD Licence. Individual packages may have their own (compatible) licences, as listed here.

The SciPy global optimization minimizers are set as follows:

```
[MINIMIZERS]
scipy_go: differential_evolution
shgo
dual_annealing
```

Note

The shgo solver is particularly slow running and should generally be avoided. As a result, this solver is not run by default when scipy_go software is selected. In order to run this minimizer, you must explicitly set it as above.

## Theseus (`theseus`

)

Theseus is an efficient application-agnostic library for building custom nonlinear optimization layers in PyTorch to support constructing various problems in robotics and vision as end-to-end differentiable architectures.

Levenberg Marquardt (

`Levenberg_Marquardt`

)Gauss Newton (

`Gauss-Newton`

)

**Links** Paper- Theseus optimization

**Licence** Theseus is available under a MIT licence.

The theseus minimizers are set as follows:

```
[MINIMIZERS]
theseus: Levenberg_Marquardt
Gauss-Newton
```

Note

We strongly recommend you install Theseus in a venv or conda environment with Python 3.7-3.9