Adding Fitting Software

Controllers are used to interface FitBenchmarking with the various fitting packages. Controllers are responsible for converting the problem into a format that the fitting software can use, and converting the result back to a standardised format (numpy arrays). As well as this, the controller must be written so that the fitting is separated from the preparation wherever possible in order to give accurate timings for the fitting. Supported controllers are found in fitbenchmarking/controllers/.

In order to add a new controller, you will need to:

  1. Give the software a name <software_name>. This will be used by users when selecting this software.

  2. Create fitbenchmarking/controllers/<software_name>_controller.py which contains a new subclass of Controller.

    Note

    Please note that if the fitting package being added uses Matlab, then the new controller should also inherit from the mixin class MatlabMixin.

    class fitbenchmarking.controllers.matlab_mixin.MatlabMixin(cost_func)

    Mixin class for matlab fitting software controllers

    py_to_mat(func)

    Get the named function from the matlab version of the cost function

    Parameters:

    func (str) – The name of the function to retrieve

    The new controller should implement four functions, as well as initializing the dictionary algorithm_check:

  • Controller.algorithm_check = {'MCMC': [], 'all': [], 'bfgs': [], 'conjugate_gradient': [], 'deriv_free': [], 'gauss_newton': [], 'general': [], 'global_optimization': [], 'levenberg-marquardt': [], 'ls': [], 'simplex': [], 'steepest_descent': [], 'trust_region': []}

    Within the controller class, you must initialize a dictionary, algorithm_check, such that the keys are given by:

    • all - all minimizers

    • ls - least-squares fitting algorithms

    • deriv_free - derivative free algorithms (these are algorithms that cannot use information about derivatives – e.g., the Simplex method in Mantid)

    • general - minimizers which solve a generic min f(x)

    • simplex - derivative free simplex based algorithms e.g. Nelder-Mead

    • trust_region - algorithms which emply a trust region approach

    • levenberg-marquardt - minimizers that use the Levenberg-Marquardt algorithm

    • gauss_newton - minimizers that use the Gauss Newton algorithm

    • bfgs - minimizers that use the BFGS algorithm

    • conjugate_gradient - Conjugate Gradient algorithms

    • steepest_descent - Steepest Descent algorithms

    • global_optimization - Global Optimization algorithms

    • MCMC - Markov Chain Monte Carlo algorithms

    The values of the dictionary are given as a list of minimizers for that specific controller that fit into each of the above categories. See for example the GSL controller.

    The algorithm_check dictionary is used to determine which minimizers to run given the algorithm_type selected in Fitting Options. For guidance on how to catagorise minimizers, see the Optimization Algorithms section of the FitBenchmarking docs.

  • Controller.__init__()

    Initialise anything that is needed specifically for the software, do any work that can be done without knowledge of the minimizer to use, or function to fit, and call super(<software_name>Controller, self).__init__(problem) (the base class’s __init__ implementation).

    Parameters:

    cost_func (subclass of CostFunc) – Cost function object selected from options.

  • abstract Controller.setup()

    Setup the specifics of the fitting.

    Anything needed for “fit” that can only be done after knowing the minimizer to use and the function to fit should be done here. Any variables needed should be saved to self (as class attributes).

    If a solver supports bounded problems, then this is where value_ranges should be set up for that specific solver. The default format is a list of tuples containing the lower and upper bounds for each parameter e.g. [(p1_lb, p2_ub), (p2_lb, p2_ub),…]

  • abstract Controller.fit()

    Run the fitting.

    This will be timed so should include only what is needed to fit the data.

  • abstract Controller.cleanup()

    Retrieve the result as a numpy array and store results.

    Convert the fitted parameters into a numpy array, saved to self.final_params, and store the error flag as self.flag.

    The flag corresponds to the following messages:

    flag()
    0: Successfully converged
    1: Software reported maximum number of iterations exceeded
    2: Software run but didn’t converge to solution
    3: Software raised an exception
    4: Solver doesn’t support bounded problems
    5: Solution doesn’t respect parameter bounds
    6: Solver has exceeded maximum allowed runtime
    7: Validation of the provided options failed
    8: Confidence in fit could not be calculated

By default, a controller does not accept Jacobian or Hessian information. If the controller being added can use hessians and/or jacobians, then the following controller attributes should be set:

  • Controller.jacobian_enabled_solvers = []

    Within the controller class, you must define the list jacobian_enabled_solvers if any of the minimizers for the specific software are able to use jacobian information.

    • jacobian_enabled_solvers: a list of minimizers in a specific

    software that allow Jacobian information to be passed into the fitting algorithm

  • Controller.hessian_enabled_solvers = []

    Within the controller class, you must define the list hessian_enabled_solvers if any of the minimizers for the specific software are able to use hessian information.

    • hessian_enabled_solvers: a list of minimizers in a specific

    software that allow Hessian information to be passed into the fitting algorithm

  1. Add the new software to the default options, following the instructions in Adding new Options.

Your new software is now fully hooked in with FitBenchmarking, and you can compare it with the current software. You are encouraged to contribute this to the repository so that other can use this package. To do this need to follow our Coding Standards and our Git Workflow, and you’ll also need to

  1. Document the available minimizers (see Fitting Options, Minimizer Options), including licencing information, if appropriate. Note: make sure that you use <software_name> in these places so that the software links in the HTML tables link correctly to the documentation. Add the software to examples/all_software.ini.

    You should also ensure that the available minimizers are catagorised correctly in self.algorithm_check using the algorithm type options. Please refer to the Optimization Algorithms page for more information about each algorithm type.

  2. Create tests for the software in fitbenchmarking/controllers/tests/test_controllers.py. If the package is pip installable then add the tests to the DefaultControllerTests class and if not add to the ExternalControllerTests class. Unless the new controller is more complicated than the currently available controllers, this can be done by following the example of the others.

  3. If the software is deterministic, add the software to the regression tests in fitbenchmarking/systests/test_regression.py.

  4. If pip installable add to [project.optional-dependencies] in pyproject.toml and add to the installation step in .github/workflows/release.yml. If not, document the installation procedure in Installing External Software and update the FullInstall Docker Container – the main developers will help you with this.

Note

For ease of maintenance, please add new controllers to a list of software in alphabetical order.

The FittingProblem, CostFunc and Jacobian classes

When adding new minimizers, you will find it helpful to make use of the following members of the FittingProblem, subclasses of CostFunc and subclasses of Jacobian classes:

class fitbenchmarking.parsing.fitting_problem.FittingProblem(options)

Definition of a fitting problem, which will be populated by a parser from a problem definition file.

Onces populated, this should include the data, the function and any other additional requirements from the data.

data_e

numpy array The errors or weights

data_x

numpy array The x-data

data_y

numpy array The y-data

eval_model(params, **kwargs)

Function evaluation method

Parameters:

params (list) – parameter value(s)

Returns:

data values evaluated from the function of the problem

Return type:

numpy array

set_value_ranges(value_ranges)

Function to format parameter bounds before passing to controllers, so self.value_ranges is a list of tuples, which contain lower and upper bounds (lb,ub) for each parameter in the problem

Parameters:

value_ranges (dict) –

dictionary of bounded parameter names with

lower and upper bound values e.g.

{p1_name: [p1_min, p1_max], ...}

class fitbenchmarking.cost_func.base_cost_func.CostFunc(problem)

Base class for the cost functions.

abstract eval_cost(params, **kwargs)

Evaluate the cost function

Parameters:

params (list) – The parameters to calculate residuals for

Returns:

evaluated cost function

Return type:

float

class fitbenchmarking.jacobian.base_jacobian.Jacobian(problem)

Base class for Jacobian.

abstract eval(params, **kwargs)

Evaluates Jacobian of the model, \(\nabla_p f(x,p)\), at the point given by the parameters.

Parameters:

params (list) – The parameter values at which to evaluate the Jacobian

Returns:

Computed Jacobian

Return type:

numpy array