[WIP] Adding Auto-Sklearn

[herilalaina]

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

This PR partially fixes #618. It extends ML hyperparameter tuning problems to Auto-Sklearn search space.

How Has This Been Tested (if it applies)

Checklist

• [TODO] The documentation is up-to-date with the changes I made.
• I have read the CONTRIBUTING document and completed the CLA (see CLA).
• [TODO] All tests passed, and additional code has been covered with new tests.

[herilalaina]

There is a scikit-learn version conflict here: kuti (requires scikit-learn<0.23.0,>=0.22.1) and Auto-Sklearn (requires scikit-learn<0.25.0,>=0.24.0).

This PR works only for auto-sklearn==0.12.6.

@teytaud You can reproduce error on BO with: pytest -s nevergrad/functions/automl/ -k "test_bo_error"

[herilalaina]

Everything works fine (except on Windows: not supported) after downgrading Auto-Sklearn to 0.11.0

[jrapin]

Everything works fine (except on Windows: not supported) after downgrading Auto-Sklearn to 0.11.0

Given it cannot be installed on Windows, we cannot keep it in the least of requirements.
So a few changes required:

• remove autosklearn from the requirements
• don't import autosklearn on top of the module
• when instantiating the class, raise ng.errors.UnsupportedExperiment in case we're on Windows (eg:

  nevergrad/nevergrad/functions/images/imagelosses.py

  Lines 113 to 119 in 2fb7d17

  	if os.name != "nt":
  	# pylint: disable=import-outside-toplevel
  	from koncept.models import Koncept512 as K512Model
  	MODELS[key] = K512Model()
  	else:
  	raise UnsupportedExperiment("Koncept512 is not working properly under Windows")

• import autosklearn when needed, inside the class, using a try/except clause that explains that autosklearn needs to be installed first (or maybe there's no needs, it will just to an ImportError anyway.
• manually install autosklearn within .circleci/config.yml

[jrapin]

All these tests use some optimization, it would be nice to have at least one thaat just calls func(whatever_parameters) and checks that it works independently of anything else.

[teytaud]

This PR looks exciting and almost OK. Any reason for interrupting ? If Windows is an issue we can disable that.

[teytaud]

@herilalaina I think the problem is unrelated to your pr and now fixed in main
so maybe just:
(1) git pull origin main <=== this rebase will fix pointless bugs
(2) black nevergrad <=== this will fix static tests
(3) then push again and maybe everything will be fine (or maybe not :-) )
(you might also merge my suggestion by clicking on commit above, so that your work is registered in the dashboard :-) )

[herilalaina]

Thanks @teytaud ! Tests are now passing except for a few mypy errors (btw unrelated to the PR ?).

Do you have recommendations for the list of algorithms and number of datasets to consider in experiments.py?

[teytaud]

Thanks @teytaud ! Tests are now passing except for a few mypy errors (btw unrelated to the PR ?).

That's a complete mystery, and I would suggest #type: ignore if jrapin does not kill me for writing this...

Do you have recommendations for the list of algorithms and number of datasets to consider in experiments.py?

For problems like hp tuning, maybe HyperOpt, Cobyla, BO, NGOpt39, OnePlusOne, TwoPointsDE, GeneticDE, MetaModel, CMA...

[teytaud]

@herilalaina I suggest #type: ignore for the two remaining issues in static...

[teytaud]

Deprecated by #1389