Verbosity System Description

[jClugstor]

The current way to specify verbosity for solvers is through the Bool keyword argument verbose, which just causes the solver to print out more information if it's true. We can have a lot more flexibility by adding an AbstractVerbosity abstract type and concrete subtypes for every SciMLProblem type.

For example we'll have a BVPVerbosity <: AbstractVerbosity that will have toggle-able Bool fields for everything that we would want to print during a BVP solve.

We can have constructors so that BVPVerbosity(true) has every field true, BVPVerbosity(false) has every field false, and BVPVerbosity(nothing) is the default which has some things true and other things false.

This will allow the user to have much greater control over what is printed during the solution of any problem.

[jClugstor]

Related to SciML/BoundaryValueDiffEq.jl#290 (comment)

[ChrisRackauckas]

Some things to read, especially from R's global warning system, which I think has done the best for user-level toggling of messaging:

• https://stackoverflow.com/questions/11239428/how-to-change-warning-setting
• https://stat.ethz.ch/R-manual/R-devel/library/base/html/warning.html
• https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/warnings
• https://adv-r.hadley.nz/conditions.html

The idea is that we want to get something with similar-ish functionality, but without relying on globals, which is not thread safe and has performance issues, instead allowing this to be at a function level and uniform across SciML. So let's keep that goal in mind.

The basic kwargs on everything should be:

• verbose: highest level toggle on all printing and warning defaults. Defaults to Verbosity.Default, Allowed: None, Default, All
• error_control: toggles handling on error control and adaptivity checking algorithms, such as convergence issues and correctness guarantees (e.x. dt < dtmin). Defaults to Verbosity.Warn. Allowed Verbosity: None, Warn, Error.
• performance: toggles handling on performance issues, such as mismatched input/output types in out of place functions and other known performance traps. Defaults to Verbosity.Warn. Allowed Verbosity: None, Warn, Error.
• numerical: toggles handling of potential numerical issues, such as large condition numbers and detection of other potential floating point issues. Defaults to Verbosity.None. Allowed: None, Edge, All.

Edge = only prints the info on edge cases, like high condition numbers and changes to QR pivoting.
Info = prints every step. It's a debugging mode. This could also have things like solver_choice, where it can print on things like "I switched which ODE solver I'm using here".

Then each of these should be handling individual settings like convergence, dtmin, condition, etc. which are flipped by the higher commands, so error_control = Verbosity.Error sets all of the error control pieces like convergence = Verbosity.Error

verbose is then treated as a default where others refine. All just turns on all printing available, so then every condition number prints out etc, and then you can condition = Verbosity.None to override that.

The normal usage should match what we have right now, i.e. that "most" things try not to print but warnings of error control do print to let the user know why the solver exited early. Then we expect users to do things like:

verbose = ODEVerbosity(verbose = Verbosity.None) # I want no printing ever, quickest way to write that
verbose = ODEVerbosity(verbose = Verbosity.All) # I want all printing and I'm just going to copy paste it into an issue and Chris why the solver is failing
verbose = ODEVerbosity(error_control = Verbosity.Error) # I want error messages instead of warnings if the solver fails
verbose = ODEVerbosity(performance= Verbosity.Warn) # I want warnings for solver exiting but also if something is slow 
verbose = ODEVerbosity(performance= Verbosity.Edge) # I'm scared that my equation is hard and want more information

Those are probably the most common scenarios right there.

In addition we want to start controlling logging locations. Right now we assume everything goes to the REPL. But if we start to print out giant lists of condition numbers, usually I don't Revise that to my REPL but slap that into a file. So it would be nice to give users easy way to do this. In fact, think of some of these options as "wouldn't it be great if the user gave me everything in a bug report so I didn't have to run their code myself?" The point of this is to make that as dead simple as possible, so users know how to fix things better but also give us better bug reports.

Again, something someone like @oscardssmith or I would do is add printing statements into OrdinaryDiffEq to do things like, print if the solver switched from stiff to non-stiff, and print the time stamp of when that happened, so if verbose = ODEVerbosity(performance= Verbosity.All) did exactly that and slapped it into a file, we could just ask the user to run that report and we could diagnose what's going on from our phones without having to run the code ourselves. Quicker debugging and user feedback.

So for logging control we probably want some stuff like warn_location, info_location, etc. where it defaults to REPL, but then has an option for FILE, where then a filename is provided. This would be nice for example so that the exit warnings still go to the REPL, but you can have the extra verbose information printing out to a file.

There's probably many other ideas to go on this. @ranocha @devmotion @oscardssmith @AayushSabharwal @BenChung @Vaibhavdixit02 @avik-pal @baggepinnen @bradcarman @asinghvi17 @SebastianM-C probably have feedback, along with maybe @LilithHafner who has done some with Julia Base and likely has some suggests of things we should use to accomplish this.

[AayushSabharwal]

Then each of these should be handling individual settings like convergence, dtmin, condition, etc. which are flipped by the higher commands, so error_control = Verbosity.Error sets all of the error control pieces like convergence = Verbosity.Error

Could we make the hierarchy more explicit? So we have a struct ErrorControlVerbosity <: AbstractVerbosity and in the top level ODEVerbosity struct the field is error_control::Union{Verbosity, ErrorControlVerbosity}. The user either sets error_control = Verbosity.Warn or error_control = ErrorControlVerbosity(; dtmin = Verbosity.Warn). We can define convert(::Type{<:AbstractVerbosity}, ::Verbosity) to automatically convert Verbosity.Warn into an ErrorControlVerbosity. This could also enable a nice way to say "Everything but X", by doing

error_control = ErrorControlVerbosity(Verbosity.None)
error_control.dtmin = Verbosity.Warn

[SebastianM-C]

One aspect that I want to mention on this is that the Logging stdlib & LoggingExtras have several of the features that we want, such as

• printing to file
• various log levels
• adding metadata to logs that can be used for filtering

One issue though is that if you have a log that is disabled, such as the default @debug behavior, that can still lead to a performance hit since the codegen still contains logging code, so adding this kind of logging would not be zero cost. It would be very nice if base julia could have some options for statically defining log levels or something similar.

[ranocha]

I agree with the last part - this should not come at the price of making something like solving many small ODEs on a GPU in parallel terribly slow.

[ChrisRackauckas]

Yes, it should all compile out when the logging isn't used. That's pretty essential and would break static compilation if that isn't setup. So that's definitely in the requirements.