Testing strategies

[dbutenhof]

Rather than creating an issue that's primarily to begin a discussion, I figured I might as well just start it as a discussion. Throw in ideas and comments, and we can work towards defining a "testing epic" in Jira that we can boil down into discrete GitHub issues over time.

Right now, we have a fragmented testing environment including true unit tests, "fake" unit tests (e.g., server tests relying on a real database, and on the Flask package to make client API calls in order to test our Resource classes), and the "gold" unit tests (which are really pseudo-functional tests relying on a mixture of source code hackery and external setup), and pytest "functional tests" which drive real commands in an isolated contrived environment.

We need a true functional test environment, where we create a real server instance with backend services (PostgreSQL, Elasticsearch, Apache, NGINX, etc) which we can drive with real client interactions and additional observability hooks (including journald, database visualizer tools like Kibana and psql) to validate and diagnose the behavior of our code in an automated framework.

I think we also need to "simplify" our current testing frameworks to a straightforward pytest environment to truly isolate individual objects and methods for individual interface testing with all external interactions cleanly mocked. We shouldn't be making client POST/GET calls here; we should be calling methods of the class. We shouldn't be making DB calls; we should be mocking the results of all methods outside the specific class under test. The way we do it now is wasting time and resources testing our dependency code (SQLAlchemy, Flask, etc.) as "unit" tests. Validating all of that external integration is important, but it's not unit testing because it doesn't limit the failure domain to our own code.

[webbnh]

Yes.