Rework experiment options

[chriseclectic]

Summary

Currently it is difficult to set custom options for an experiment. This is in part due to the different layers of options that are involved including:

• Experiment options (which effect circuit generation)
• Transpilation options
• Backend / execution options
• Analysis options

This PR attempts to improve how options are handled for both experiment design (for documenting available options and setting default values), and for the user (for customizing options when running experiments). This is done by adding set options methods in the same way that it is currently done for configurable Qiskit backends (Aer, IBMQ).

Note that this uses the same Options class as backends from terra, which currently is not mappable (so you need to do **options.__dict__ for example), however there is currently a PR open to make this class mappable.

Details and comments

The main changes here are:

• Adds options, transpile_options, backend_options, and analysis_options properties to the BaseExperiment class. These contain all default/set options for the current experiment and will be used by the run method to allow customizing execution. When designing an experiment default values for each of these are set by overriding the corresponding _default_options methods. For users to update options they use the corresponding set_options, set_transpiler_options etc methods.
• Removed kwargs from BaseExperiments.circuits, these are intended to all be handled with options/set_options.
• Adds **options to BaseExperiment.__init__ for setting custom option values at initialization.
• Removes transpiled_circuits method: this method should intead be done using a call to the terra transpile function with the experiments transpile_options as qiskit.transpile(expr.circuits(backend), backend, **exp.transpile_options).

With these changes an example of using custom options for running an experiment might look something like:

qubits = [0, 2]
expr = MyExperiment(qubits, param=val, seed=1)
expr.set_transpile_options(optimization_level=3, scheduling_method='asap')
expr.set_analysis_options(method=something)

# Run experiment
expdata = expr.run(backend, shots=2000)

[eggerdj]

This approach makes sense to me. It is a bit hard to see what the default options are. Is there a way we can make it easier for a user to figure out what the defaults are?

[eggerdj]

Can you add some more details on how the options are handled in the docstrings? Perhaps doing this at the module level instead of in the init or class docstring would be good.

[nkanazawa1989]

I agree with having more description would be great, but it seems difficult. The Options class defines nothing.
https://github.com/Qiskit/qiskit-terra/blob/9fd5436ba30b98d3bfd2578cc7f4ce564dfffd55/qiskit/providers/options.py#L18-L27

This options is provider dependent, but the experiment is provider agnostic (since this is analysis option, this can be experiment dependent).

[chriseclectic]

Sure, sounds good. I think we should probably add documentation of all experiment and analysis options to the class level docstring of each experiment and analysis options to the analysis class (this will duplicate analysis options, but I think this is good to do since most people will only interact via the Experiment class and not use the analysis classes directly).

@nkanazawa1989 The Options class isn't provider dependent, its just a container in the qiskit provider module since it is currently only used in backends for IBMQ and Aer. Aer and IBMQ add their backend dependent options in the same way as here using _default_options class methods.

[eggerdj]

Makes sense. Are there any other such cases we need to worry about?

[chriseclectic]

Not that I can immediately think of for circuits, maybe for calibration you will have some scheduling specific ones that should not be user changed though? If so we can add them as needed

[eggerdj]

This almost looks like something that the subclasses would like to control. E.g. perhaps some RB experiments would want to run on a high optimization level while calibration experiments would want to avoid this. I suppose subclasses can always override set_transpile_options.

[nkanazawa1989]

sounds like we can override _default_transpile_options instead.

[nkanazawa1989]

This PR makes the base class cleaner and I really like it. My main point is suggestion of removal of kwargs from Experiment.run method (though I understand this is convenient for single line execution, which is important concept of experiment).

[nkanazawa1989]

The options contains both immutable and runtime options? This overrides class default setting in constructor and runtime. For me it looks too flexible.

[chriseclectic]

This was matching the current way options are handled for simulators. When a class is initialized it initializes a copy of the default options specified in the class design, and then everything in options is mutable. For analysis options it's currently coded so you can pass option values at construction Analysis(**options) which is equivalent to setting them after default construction, Analysis().set_options(**options). These are equivalent. The Analysis.run method can also be called with options, which will override any stored values in analysis.options just for that execution, without changing any of the stored options.

However thinking about this more, since analysis options are also stored in experiment maybe this is overkill and we could remove the set_options ahd just have the default options and run(**options) to override them to keep the analysis clas more like a function without side effects (like it was before). I'll try make some changes to simplify this.

[nkanazawa1989]

Sounds good, thank you :)

[nkanazawa1989]

I agree with having more description would be great, but it seems difficult. The Options class defines nothing.
https://github.com/Qiskit/qiskit-terra/blob/9fd5436ba30b98d3bfd2578cc7f4ce564dfffd55/qiskit/providers/options.py#L18-L27

This options is provider dependent, but the experiment is provider agnostic (since this is analysis option, this can be experiment dependent).

[nkanazawa1989]

Do you think it is possible to have default data processor? I think this should depend on backend (execution) options, i.e. meas_level and return_type. Where the default data processor should be defined?

[eggerdj]

There is probably a few ways to do this. One option is in the analysis class, see e.g. the work in progress here: #31

[chriseclectic]

It probably makes sense to put something into the ExperimentData class for meas level. In ExperimentData when you add job/results and it pulls out circuit data it could for example store the meas_level / result_type in the metadata (if we didn't already add it as experiment metadata).

[nkanazawa1989]

This is bit vague. **backend_options? I think this run method internally calls transpile, execute, and analysis. This makes me think this option should be able to handle all of these options. Otherwise we should remove runtime option to avoid confusion. i.e.

exp.run(backend, optimization_level=0)

doesn't look strange to me (since Options class defines nothing we cannot raise any warning or error with current implementation).

[chriseclectic]

The intent is options passed to exeriment.run are directly passed through to backend.run as run options (and backend.run doesnt support transpiler options like old execute)

You can't filter out transpiler (or analysis) options without keeping an exhaustive list of all allowed option values for the transpiler like we had before, which is part of what this PR is trying to fix. So the intent is any transpiler/analysis options be specifically set using the corresponding set methods.

[nkanazawa1989]

Another comment. Do we really need to expose transpile to end users? This sounds like a jargon. I prefer something like circuit_optimization_option.

[mtreinish]

One thing here to note is I'm trying to make the Options class a bit more useable in: Qiskit/qiskit#6355 . It's still a free form container but offers the mapping interface now (which enables func(**options)). On top of that it makes the expected contract of the class a bit more explicit, ie you can create any fields in the options you want at __init__ time, but after the object is initialized the fields are fixed and only the values are mutable. This fits with how _default_options is expected to work as a classmethod it defines which options/fields are available for the class and what their default values are (which can be None).

[chriseclectic]

@mtreinish not being able to add fields after construction will have some issues with transpiler/run/analysis options which do not provide an exhaustive list of all available fields in the default method (we don't want to duplicate every kwarg and default value from transpile, and each backend will have different options that could be used as run options).

[mtreinish]

@mtreinish not being able to add fields after construction will have some issues with transpiler/run/analysis options which do not provide an exhaustive list of all available fields in the default method (we don't want to duplicate every kwarg and default value from transpile, and each backend will have different options that could be used as run options).

You can always create a new Options object if you need to add fields dynamically. I'm hesitant to enable adding fields after initialization because at a certain point it just ends up being a dictionary and I think the Options class should have more of a fixed structure so people can have more guarantees than a dictionary.

For multiple backends being wrapped by an experiment class you can do something like what I did for the fake backends and just return the upstream backend's options object: https://github.com/Qiskit/qiskit-terra/blob/main/qiskit/test/mock/fake_backend.py#L112-L116

For the transpile function, I'm wondering if maybe we want to expose the options through a global Options object. We're kind of half way there with the user config file for some options.

[yaelbh]

Thanks @chriseclectic, it looks good. Will it solve #17 ?

[chriseclectic]

@yaelbh It should, if you set the analysis options for each sub experiment. Something like:

exp1 = Experiment(**params1)
exp1.set_analysis_otions(**opts1)
exp2 = Experiment(**params2)
exp2.set_analysis_options(**opts2)
exp12 = ParallelExperiment([exp1, exp2])
expdata = exp12.run(backend)

[eggerdj]

Currently in the main branch all run-time options get passed to the analysis. Therefore, any circuit generation options that the analysis does not need also get passed to the analysis. To avoid the analysis from throwing an exception we need a **kwargs in the arguments of the _run_analysis method. It looks like this PR solves this issue. Is that the case @chriseclectic ?

[chriseclectic]

@eggerdj Yes, that it one of the issues this PR addresses. To set analysis options now you could do

exp = Experiment(...)
exp.set_analysis_options(**analysis_opts)
expdata = exp.run(backend,  **run_opts)

or if you wanted to break execution and analysis into 2 separate steps:

exp = Experiment(...)
expdata = exp.run(backend, analysis=False, **run_opts)
exp.run_analysis(expdata, **analysis_options)

[eggerdj]

This looks good to me. A few small questions/comments.

[eggerdj]

A few lines above is written:

             self.run_analysis(experiment_data)

i.e. I see no options. Is this not inconsistent with the method signature? Do we need **options here?

[chriseclectic]

The direct call to run_analysis allows overriding the experiments stored analysis options with the kwarg ones (like run does with runtime options). At the moment for split execution/analysis the following are basically equivalent:

# 1
exp.set_analysis_options(**analysis_opts)
expdata = exp.run(backend, analysis=False)
exp.run_analysis(expdata)

# 2
expdata = exp.run(backend, analysis=False)
exp.run_analysis(expdata, **analysis_opts)

Removing the kwargs from run_analysis would remove option 2. Opt 2 is not strictly necessary, but I thought it might be convenient in cases where you are running analysis separately to the main execution, or analyze expdata loaded from a DB.

[eggerdj]

That makes sense. Thanks. At the risk of being pedantic, can we document this in the docstring to make the two usage options clear and that an analysis can be run outside the context of an experiment?

[eggerdj]

This almost looks like something that the subclasses would like to control. E.g. perhaps some RB experiments would want to run on a high optimization level while calibration experiments would want to avoid this. I suppose subclasses can always override set_transpile_options.

[nkanazawa1989]

sounds like we can override _default_transpile_options instead.

[nkanazawa1989]

Do you think it is worth adding a link to the API docs? Of course this is provider agnostic but user cannot find what is run_options with this. You can say something like this is provider dependent but in IBM Quantum you can refer to ...