Too much "not ready for production" messaging in docs/examples

[j-wags]

Continuing from OpenFreeEnergy/IndustryBenchmarks2024#314 (comment)

@mattwthompson had brought up the extent to which we recommend against using our products or speak inconsistently about recommended ways to do things. Matt led an effort to dig up all the instances of this and the report is at the end of this message.

The big action I'd propose is to change the messaging around our ff14SB port to NOT use the phrase "not ready for production", because:

1. People have been using these workflows in production for a while, both internally in industry and in academic publications
2. Given the huge scope of what people might want to use our tools for, I can't fathom what testing I'd do to say that things are "ready for production", other than doing the potential energy comparisons we've done on enumerated sequences and folded proteins, and releasing the FF for a while and not having people report critical bugs, which is more-or-less what's already happened.

The major courses of action I can think of here are:

• Stop saying that Sage + ff14SB isn't ready for production, instead say something like In our testing, we've found that the SMIRNOFF ff14SB port has very minor (0.X%) energy differences from the original AMBER implementation due to unavoidable differences in parameter assignment methodology. People have used it in their workflows for years and not reported any major energy discrepancies or incompatibilities with OpenFF's Parsley and Sage line of force fields.
• Update messaging around no-longer-experimental Interchange functionality (if it isn't already updated - Matt did a great job making users acknowledge using experimental features by requiring them to set an environment variable to activate it)
• Replace wording around deterministic AM1BCC charge assignment by saying OpenFF tooling attempts to provide the same charges for the same molecular graph (independent of the input conformation) by generating one or more conformers internally before running AM1BCC during parameter assignment. While we try to get consistent behavior from this conformer generation step, this method is not perfect and the resulting charges may vary somewhat between runs. Our upcoming force fields will use a [charge assignment method](NAGL) that only considers the 2D structure of input molecules to assign partial charges deterministically to a given molecular graph.
• Address all straightforward fixes in Matt's report as well

I'm interested if folks have major hesitations or alternatives for this plan!

Matt's report

More than zero of our examples have warnings which prevent people from believing our products are suitable for production. Some of these warnings are significantly out of date and likely a barrier to adoption in their current state. Here is one such example: openmm/openmm#4806 (comment)

Specific examples

Toolkit showcase (openff-toolkit):

• users explicitly told not to use ff14SB port

Protein-ligand systems (openff-interchange)

• users explicitly told this code is not production-ready
• users explicitly told they should not use ff14SB port
• users explicitly told Interchange.combine is experimental

OpenFF force fields in Amber and GROMACS
(note: somewhat unclear which example this refers to - I don't see these here or here)

• users explicitly told not suitable for multi-component systems
• outdated GROMACS installation command

Mixture simulation

• says OpenMM is used for loading topologies
• says the toolkit produces deterministic charges
• promises that 1.0.0 will implement a SMIRNOFF "definition of semiempirical partial charge treatment"
• promises that 1.0.0 will load charges from SDF/mol2

Conformer energies

• says the toolkit produces deterministic charge
• uses OpenMM instead of Interchange.minimize

Mixing parameters

• broken link to residue template generators
• probably functionally out-of-date

Inspect parameters

• "This API" (which API?) is "not final"

Other inconsistencies

• Toolkit showcase:frequency of writing to disk
• Protein-ligand example typo "functionaln"
• virtual sites "oxygens planar carbonyls"

[mrshirts]

I support this effort!

[Yoshanuikabundi]

• I agree we should drop the "not ready for production" verbiage, probably everywhere but definitely for the ff14sb port - we wrote it when we were pretty dubious about the port, but we have a lot of confidence in it now!
• Ditto the "experimental"/"subject to change"/"not final" warnings about the API - that's what semver is for!
• I don't see anything in Matt's report that I would oppose fixing/removing
• We probably need a page, somewhere in the docs, that describes expectations, caveats, limitations, AND publications that have come out of our ff14sb port - that way we can just link to that page when we use the port and let users make up their own minds. To be honest, at this point we probably have enough confidence in the port that the word "port" is all the heads up we need!
• The mixture simulation example sounds like it's just wrong and needs to be revised immediately

---

I think it'd be nice to have a policy about what we make promises about in the docs. I think there are a few issues that have encouraged us to hedge in the past:

1. Do we think we're likely to change an API in the near future? (eg, "not final" warnings)
2. Are we confident that this implementation of software is correct? (eg, Interchange.combine)
3. Are we confident that the ff14sb Amber port is correct? (eg, some of our "ff14sb not ready for production" warnings)
4. Are we confident that it makes sense to combine these force field parameters? (eg, most of our "ff14sb not ready for production" warnings)

There's one other area where I've resisted giving warnings:

5. Are we confident that this simulation protocol is correct? (eg, Showcase write-to-disk frequency)

Each of these I think deserves separate reasoning:

1. Do we think we're likely to change an API in the near future?

We should not warn people about this, that's what SEMVER is for. The exception might be if we literally have a ticket on the board to make the change, the design work is all done, and we can give an accurate ETA - in that case, it might make sense to warn users that we're about to break something.

2. Are we confident that this implementation of software is correct?

This is a real problem that Matt solved nicely with the experimental decorator and environment variable, and the Python warning associated with that decorator should provide enough heads up that we don't need to repeat it in the examples (especially if we then forget to remove it when it becomes out-of-date). The decorator even generates a little warning in the API docs that then gets automatically removed when the decorator is removed! I think should adopt this more widely - it's tightly scoped, highly visible in the code, it's easy to tell it to go away, and the whole shebang gets removed in one fell swoop when the feature is no longer experimental.

3. Are we confident that the ff14sb Amber port is correct?

We need a policy for force fields we're not sure about - maybe a more formal release candidate or pre-release thing? - but surely ff14sb is no longer one of them

4. Are we confident that it makes sense to combine these force field parameters?

We need a policy on combining force fields, and it might be as simple as "Parsley and Sage are compatible with Amber force fields".

5. Are we confident that this simulation protocol is correct?

I think this should be an explicit non-goal of our documentation. Our responsibilities to accuracy end when the user has system parameters in their engine of choice, and what they do with that engine is up to them. In examples I write, I try to signal this by doing obviously wrong things like writing to disk constantly, running only very short simulations, omitting equilibration, not doing analysis, and (frankly) running the simulation inside the notebook. It's nice to be able to give the user the pay-off of seeing their system wiggle, but the purpose of the notebook was finished several steps before.

I admit that this has resulted in an issue or two in the past, I suspect from 1st year grad students and other beginners, but I think the appropriate response is "yeah, this example isn't about the simulation protocol". I would love to write a Lemkul-style beginners guide to MD using OpenFF and OpenMM, and if I ever get to do that then we can make promises about the protocol in that guide, but in general our documentation is targeted at people who already know their way around their engine of choice. This is an important separation-of-concerns issue that allows us to focus on force fields, rather than trying to do the whole simulation pipeline.

[mattwthompson]

Fantastic write-up, Josh!

[davidlmobley]

I broadly agree too; my students keep telling me they can't do X because there's a warning about it not being production ready, and I keep having to convince them it is OK to ignore the warning. The reality is most code all of us use (from other products) is academic and "not production ready" so the different standard we're holding ourselves too creates lots of confusion.

[pavankum]

Adding PackMol wrapper in Interchange to the list, which sometimes get tagged with ''experimental Interchange PackMOL wrapper".

[j-wags]

You'll be happy to see https://github.com/openforcefield/openff-interchange/pull/1252/files in the next Interchange release :-)