Adding Fitting Software
Controllers are used to interface FitBenchmarking with the various fitting
packages. Controllers are responsible for converting the problem into a format
that the fitting software can use, and converting the result back to a
standardised format (numpy arrays). As well as this, the controller must be
written so that the fitting is separated from the preparation wherever possible
in order to give accurate timings for the fitting. Supported controllers are
found in fitbenchmarking/controllers/
.
In order to add a new controller, you will need to:
Give the software a name
<software_name>
. This will be used by users when selecting this software.Create
fitbenchmarking/controllers/<software_name>_controller.py
which contains a new subclass ofController
.Note
Please note that if the fitting package being added uses Matlab, then the new controller should also inherit from the mixin class
MatlabMixin
.- class fitbenchmarking.controllers.matlab_mixin.MatlabMixin(cost_func)
Mixin class for matlab fitting software controllers
- py_to_mat(func)
Get the named function from the matlab version of the cost function
- Parameters
func (str) – The name of the function to retrieve
The new controller should implement four functions, as well as initializing the dictionary
algorithm_check
:
- Controller.algorithm_check = {'MCMC': [], 'all': [], 'bfgs': [], 'conjugate_gradient': [], 'deriv_free': [], 'gauss_newton': [], 'general': [], 'global_optimization': [], 'levenberg-marquardt': [], 'ls': [], 'simplex': [], 'steepest_descent': [], 'trust_region': []}
Within the controller class, you must initialize a dictionary,
algorithm_check
, such that the keys are given by:
all
- all minimizers
ls
- least-squares fitting algorithms
deriv_free
- derivative free algorithms (these are algorithms that cannot use information about derivatives – e.g., theSimplex
method inMantid
)
general
- minimizers which solve a generic min f(x)
simplex
- derivative free simplex based algorithms e.g. Nelder-Mead
trust_region
- algorithms which emply a trust region approach
levenberg-marquardt
- minimizers that use the Levenberg-Marquardt algorithm
gauss_newton
- minimizers that use the Gauss Newton algorithm
bfgs
- minimizers that use the BFGS algorithm
conjugate_gradient
- Conjugate Gradient algorithms
steepest_descent
- Steepest Descent algorithms
global_optimization
- Global Optimization algorithms
MCMC
- Markov Chain Monte Carlo algorithmsThe values of the dictionary are given as a list of minimizers for that specific controller that fit into each of the above categories. See for example the
GSL
controller.
- Controller.__init__()
Initialise anything that is needed specifically for the software, do any work that can be done without knowledge of the minimizer to use, or function to fit, and call
super(<software_name>Controller, self).__init__(problem)
(the base class’s__init__
implementation).
- Parameters
cost_func (subclass of
CostFunc
) – Cost function object selected from options.
- abstract Controller.setup()
Setup the specifics of the fitting.
Anything needed for “fit” that can only be done after knowing the minimizer to use and the function to fit should be done here. Any variables needed should be saved to self (as class attributes).
If a solver supports bounded problems, then this is where value_ranges should be set up for that specific solver. The default format is a list of tuples containing the lower and upper bounds for each parameter e.g. [(p1_lb, p2_ub), (p2_lb, p2_ub),…]
- abstract Controller.fit()
Run the fitting.
This will be timed so should include only what is needed to fit the data.
- abstract Controller.cleanup()
Retrieve the result as a numpy array and store results.
Convert the fitted parameters into a numpy array, saved to
self.final_params
, and store the error flag asself.flag
.The flag corresponds to the following messages:
- flag()
0: Successfully converged1: Software reported maximum number of iterations exceeded2: Software run but didn’t converge to solution3: Software raised an exception4: Solver doesn’t support bounded problems5: Solution doesn’t respect parameter bounds6: Solver has exceeded maximum allowed runtime7: Validation of the provided options failedBy default, a controller does not accept Jacobian or Hessian information. If the controller being added can use hessians and/or jacobians, then the following controller attributes should be set:
- Controller.jacobian_enabled_solvers = []
Within the controller class, you must define the list
jacobian_enabled_solvers
if any of the minimizers for the specific software are able to use jacobian information.
jacobian_enabled_solvers
: a list of minimizers in a specificsoftware that allow Jacobian information to be passed into the fitting algorithm
- Controller.hessian_enabled_solvers = []
Within the controller class, you must define the list
hessian_enabled_solvers
if any of the minimizers for the specific software are able to use hessian information.
hessian_enabled_solvers
: a list of minimizers in a specificsoftware that allow Hessian information to be passed into the fitting algorithm
Add the new software to the default options, following the instructions in Adding new Options.
Your new software is now fully hooked in with FitBenchmarking, and you can compare it with the current software. You are encouraged to contribute this to the repository so that other can use this package. To do this need to follow our Coding Standards and our Git Workflow, and you’ll also need to
Document the available minimizers (see Fitting Options, Minimizer Options), including licencing information, if appropriate. Note: make sure that you use
<software_name>
in these places so that the software links in the HTML tables link correctly to the documentation. Add the software toexamples/all_software.ini
.You should also ensure that the available minimizers are catagorised correctly in
self.algorithm_check
using the algorithm type options. Please refer to the Optimization Algorithms page for more information about each algorithm type.Create tests for the software in
fitbenchmarking/controllers/tests/test_controllers.py
. If the package ispip
installable then add the tests to theDefaultControllerTests
class and if not add to theExternalControllerTests
class. Unless the new controller is more complicated than the currently available controllers, this can be done by following the example of the others.If the software is deterministic, add the software to the regression tests in
fitbenchmarking/systests/test_regression.py
.If pip installable add to
install_requires
insetup.py
and add to the installation step in.github/workflows/release.yml
. If not, document the installation procedure in Installing External Software and update theFullInstall
Docker Container – the main developers will help you with this.
Note
For ease of maintenance, please add new controllers to a list of software in alphabetical order.
The FittingProblem
, CostFunc
and Jacobian
classes
When adding new minimizers, you will find it helpful to make use of the
following members of the
FittingProblem
, subclasses of
CostFunc
and subclasses of
Jacobian
classes:
- class fitbenchmarking.parsing.fitting_problem.FittingProblem(options)
Definition of a fitting problem, which will be populated by a parser from a problem definition file.
Onces populated, this should include the data, the function and any other additional requirements from the data.
- data_e
numpy array The errors or weights
- data_x
numpy array The x-data
- data_y
numpy array The y-data
- eval_model(params, **kwargs)
Function evaluation method
- Parameters
params (list) – parameter value(s)
- Returns
data values evaluated from the function of the problem
- Return type
numpy array
- set_value_ranges(value_ranges)
Function to format parameter bounds before passing to controllers, so self.value_ranges is a list of tuples, which contain lower and upper bounds (lb,ub) for each parameter in the problem
- Parameters
value_ranges (dict) –
- dictionary of bounded parameter names with
lower and upper bound values e.g.
{p1_name: [p1_min, p1_max], ...}
- class fitbenchmarking.cost_func.base_cost_func.CostFunc(problem)
Base class for the cost functions.
- abstract eval_cost(params, **kwargs)
Evaluate the cost function
- Parameters
params (list) – The parameters to calculate residuals for
- Returns
evaluated cost function
- Return type
float
- class fitbenchmarking.jacobian.base_jacobian.Jacobian(problem)
Base class for Jacobian.
- abstract eval(params, **kwargs)
Evaluates Jacobian of the model, \(\nabla_p f(x,p)\), at the point given by the parameters.
- Parameters
params (list) – The parameter values at which to evaluate the Jacobian
- Returns
Computed Jacobian
- Return type
numpy array