Suggestions for a global optimization contest

The following are some suggestions that would serve to make a global optimization contest most useful to the scientific community.

How should the test problems be selected?

Apart from well known standard problems, a good test set should also contain

• penalty problems since this is how many people currently solve constrained global problems.
• problems that only have a single minimum that lies on a curved valley, such as the Rosenbrock function. There are enough problems in practice where one does not know whether there are one or more minima, so global algorithms should not perform too badly on single minimum problems.
• problems with very wide bounds since many algorithms deteriorate drastically on some problems when there is little information on the magnitude of some component(s) of the expected solution.

I think that for a contest, each test problem should have a parameter in it that may be varied, and the average count for solving five problem instances, say, with a fixed set of parameters should be the basis of evaluation. One should specify in the contest announcement the parametric form and a range for the parameter in the problem description, but keep secret the actual parameter set to be used until all submissions are in. Then tuning to the specific instance would be impossible, it is not so easy to tune the algorithm to the problem, and performance can be expected to be more like that for an unknown future problem.

Two relevant submission criteria

An important problem that one faces when one wants to use an algorithm for an expensive practical problem is that one generally has not the resources to play with the parameters and needs a first shot success. Thus each algorithm should come with an easy-to-use recipe for picking all tuning parameters, and the methods should be compared based on this default setting, at least as one of the criteria for quality.

Another important point, especially for methods for which it is easy to vary details of the algorithm, is that a comparison always compares specific implementations, not `methods'. Sometimes changing what seem minor details may have a large positive or negative effect on the quality of a method. Therefore, part of the submission requirements should be a specific implementation that is made available online. Then people can verify the results for themselves, and fast dissemination of the fruits of the best investigations is guaranteed.

This also gives a basis for selecting a new test set for a subsequent contest. Indeed, one can try this new test set on the old algorithms, and pick those problems where the old algorithms differ in performance from the general trend, or where all perform disappointingly. Because these are the areas where the need for improvement is most pressing.