Add SCIP Persistent Support #3200
Conversation
|
Two open questions left for this pull request:
For info of how I tested the code:
|
|
@Opt-Mucca - answering your questions:
|
|
@mrmundt Thanks for the fast reply!
|
|
@Opt-Mucca - No problem!
|
|
I have now added:
|
| try: | ||
| import pyscipopt | ||
|
|
||
| scip_available = True | ||
| except ImportError: | ||
| scip_available = False |
There was a problem hiding this comment.
I theorize that @jsiirola may want this logic instead put in the import attempt code: https://github.com/Pyomo/pyomo/blob/main/pyomo/common/dependencies.py#L1037
There was a problem hiding this comment.
I will ping @jsiirola again about this thought.
| try: | ||
| import pyscipopt | ||
|
|
||
| scip_available = True | ||
| except ImportError: | ||
| scip_available = False |
There was a problem hiding this comment.
I theorize that @jsiirola may want this logic instead put in the import attempt code: https://github.com/Pyomo/pyomo/blob/main/pyomo/common/dependencies.py#L1037
There was a problem hiding this comment.
In general, yes - but as this is in a testing file, it is not a big deal.
|
@Opt-Mucca - Please make sure you're on the most recent version of |
Co-authored-by: Miranda Mundt <[email protected]>
|
@mrmundt Black should be run and all the issues responded to. |
|
@Opt-Mucca - Spelling is the new failure. (You are suffering the same issue as all the core devs... The auto-linter and spell checker.) |
|
@mrmundt TIL there are auto-linters that double as spell checkers. (I'll fix the spelling errors that come up each time the test fails. If it's just the two errors then I'm properly amazed.) |
|
@Opt-Mucca - If you could have heard the griping in our weekly dev call when I introduced the spell checker... |
|
@mrmundt I can imagine (would probably be guilty of griping even if makes sense) For the currently failing runs: I've added a fix that should work, but I will ask some colleagues tomorrow. (I'm not the greatest the warm start code). The fixes raise some warning where a loaded solution has variables of incorrect values (this may stem from non specified variables having a default value of 0) The current reason of the pipeline failure: SCIP was trying to load an invalid solution before checking for feasibility (I think this is a longstanding python error with the interface and one of the functions) The added solution: I simply added a check on the feasibility of the solution before adding it to the solution storage. I will check with colleagues, but I'm not sure on how SCIP handles partial solutions. I am not familiar or don't know if we have any functionality like Gurobi that tries to complete the solution. This also holds for trying to repair a given infeasible solution. |
|
Support for partial solution loading is now added. The failed tests are now passing locally too. |
|
@Opt-Mucca - If you need help with your tests, please let us know. We tend to ignore PRs until the tests are passing. |
|
FYI, the doc build/doc test failures are NOT RELATED. I have #3479 open to resolve it. |
|
I added a minor change: I no longer check what type the |
|
@Opt-Mucca - can you do me one last favor please? I updated all of the existing files recently to reflect that our copyright is 2008 - 2025 now (see PR #3515 ) - could you please update your new files here to reflect that as well? |
There was a problem hiding this comment.
Apologies for how long it took to get to this. Overall, this is not in too bad of shape, but there are a couple things that would be very good to rework:
- avoid the use of StandardRepn, and instead use / write a walker based on
StreamBasedExpressionVisitor. - rework this so we only need to walk each expression once (this is a performance issue, and is particularly important for persistent interfaces where performance is one of the big motivators).
| f"PySCIPOpt through Pyomo does not yet support expression type {type(pyomo_expr)}" | ||
| ) | ||
|
|
||
| new_expr += get_nl_expr_recursively(repn.nonlinear_expr) |
There was a problem hiding this comment.
get_nl_expr_recursively should be reimplemented by building on StreamBasedExpressionVisitor - among other things, that routine has a graceful fallback to a nonrecursive implementation for very deep / unbalanced expressions (which we can get in Pyomo).
| repn = generate_standard_repn(expr, quadratic=True) | ||
| else: | ||
| repn = generate_standard_repn(expr, quadratic=False) |
There was a problem hiding this comment.
It would be better to build off the pyomo.repn.quadratic/QuadraticRepnVisitor and not use generate_standard_repn - generate_standard_repn has several bugs and is going to be deprecated as soon as we can get everything ported off it.
| if var.is_fixed(): | ||
| val = var.value | ||
| return val, val |
There was a problem hiding this comment.
We should verify that the var is not fixed outside it's bounds (i.e., catch trivially infeasible models)
| e = sys.exc_info()[1] | ||
| msg = ( | ||
| "Unable to create SCIP model. " | ||
| f"Have you installed PySCIPOpt correctly?\n\n\t Error message: {e}" | ||
| ) | ||
| raise Exception(msg) |
There was a problem hiding this comment.
It would be better to log this message to the logger as an error and then re-raise the original exception.
| degree = repn.polynomial_degree() | ||
| if (max_degree is not None) and (degree > max_degree): | ||
| raise DegreeError( | ||
| "While SCIP supports general non-linear constraints, the objective must be linear. " | ||
| "Please reformulate the objective by introducing a new variable. " | ||
| "For min problems: min z s.t z >= f(x). For max problems: max z s.t z <= f(x). " | ||
| "f(x) is the original non-linear objective." | ||
| ) |
There was a problem hiding this comment.
I would avoid calling polynomial_degree and instead do this check after building the QuadraticRepn: polynomial_degree is almost as slow as just building the repn, and you can get the degree from the repn by seeing if there are any quadratic terms and if the nonlinear expression is None.
| if len(repn.linear_vars) > 0: | ||
| referenced_vars.update(repn.linear_vars) | ||
| new_expr += sum( | ||
| repn.linear_coefs[i] * self._pyomo_var_to_solver_var_expr_map[var] | ||
| for i, var in enumerate(repn.linear_vars) | ||
| ) | ||
|
|
||
| for i, v in enumerate(repn.quadratic_vars): | ||
| x, y = v | ||
| new_expr += ( | ||
| repn.quadratic_coefs[i] | ||
| * self._pyomo_var_to_solver_var_expr_map[x] | ||
| * self._pyomo_var_to_solver_var_expr_map[y] | ||
| ) | ||
| referenced_vars.add(x) | ||
| referenced_vars.add(y) |
There was a problem hiding this comment.
Fundamental question: why are you using a representation compiler at all if you are just going to create a new expression tree? You could just use a single expression walker to generate the new expression directly without going through the compiled intermediary.
There was a problem hiding this comment.
Hey @jsiirola thanks for the review! It's now my turn to apologise for taking so long though...... Have started a new job and simply been waiting on a burst of motivation some evening to look at this.
I used the representation compiler because it was the first thing that I found and seemed to fit what I wanted. I do not have an overview whatsoever of the various Pyomo data structures, so that's on me for not doing enough research. I will try and look into StreamBasedExpressionVisitor and make the relevant changes (No complaints from me if someone else on the team wants to do the changes themself). Is there some particular date for when it would be good if this PR was finished?
There was a problem hiding this comment.
@Opt-Mucca: No worries! The internals of Pyomo are still rather poorly documented (not enough nights and weekends), so there is no reason to expect you to have stumbled across the "right" set of utilities. If you are looking for a potentially useful inspiration, you might look at the Pyomo2SympyVisitor walker for converting Pyomo expressions to sympy expressions (in pyomo/core/expr/sympy_tools.py)
There was a problem hiding this comment.
Oh - and you asked about timeline... We are currently planning for the next release on 6 August, so ideally, we would aim for all PRs to be in final review about a week before that.
| if (max_degree is not None) and (degree > max_degree): | ||
| raise DegreeError( | ||
| "While SCIP supports general non-linear constraints, the objective must be linear. " | ||
| "Please reformulate the objective by introducing a new variable. " |
There was a problem hiding this comment.
Is there a reason not to just do that reformulation for the user here?
There was a problem hiding this comment.
I am of the opinion that the modelling language should not be doing something that the user didn't specifically ask for. For example, creating some "shadow" variable that is likely never revealed to the user and would cause a mismatch in number of variables in the model and the number of variables in the users list / dict of "standard" variables. We have had many debates about this in PySCIPOpt.
Completely fair to have the stronger opinion of "Make it as easy as possible for the user"
There was a problem hiding this comment.
In general, I completely agree. However, we have historically made exceptions for trivial changes that
- can only be implemented in a single way
- enable a solver to be run on a model that a rational user would expect the solver to run on (that is, we want users to be able to easily switch between solvers without having to recode their model)
Examples of changes that we do make are:
- Adding a trivial (fixed) variable to LP files so that we can add constant terms to objectives (several solvers do not accept a constant as part of the objective expression in LP format)
- Converting the optimization sense (max to min) for solvers that only accept one sense.
In my opinion, this would fall into the same category... But I also think that this is on the edge, and tossing a warning for the user would be completely warranted. (I can also be convinced that this reformulation is a "bridge too far" and the error is the right path forward).
There was a problem hiding this comment.
You've now rather convinced me..... I still don't like the choice, but I do believe that if there's already some exceptions then this would be an additional sensible one,
| try: | ||
| import pyscipopt | ||
|
|
||
| scip_available = True | ||
| except ImportError: | ||
| scip_available = False |
There was a problem hiding this comment.
In general, yes - but as this is in a testing file, it is not a big deal.
|
@Opt-Mucca, we are refactoring the solver interfaces in pyomo/contrib/solver (https://pyomo.readthedocs.io/en/stable/explanation/experimental/solvers.html). We would really like this functionality there instead of in pyomo/solvers/plugins/solvers (which will eventually become deprecated). I plan to build off your branch by moving the files into pyomo/contrib/solver and reworking them to match the new interfaces. Please let us know if you have any objections/concerns. |
|
@michaelbynum No complaints at all. Sorry in advance if the code is a bit sloppy (mainly the SCIP naming and constant checks for |
Fixes # .
None.
Summary/Motivation:
Adds support for SCIP persistent solving. Persistent solving uses the python interface to SCIP directly as opposed to the current standard that involves read and writing a temporary model file.
The new dependency that it introduces is
PySCIPOptChanges proposed in this PR:
Legal Acknowledgement
By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution: