MAINT: drop support for pandas 1.x

After the next release, we can probably drop support for pandas 1.5 (certainly now nearing a pandas 3.0 release, we can support pandas 2.x and 3.x for a while)

(this would mostly allow some cleaning up in the tests)