Implement automatic dependency bumping

[njsmith]

Dependabot is cool, but has some issues:

• It's super noisy. I got 20 emails from it yesterday, which is not unusual. There are dozens of people watching our repos, so multiply by that. And probably we'll have more repos in the future. I've hesitated to add it to new repos because of this. And 95% of these emails are completely irrelevant to any human, which makes it hard to find the 5% that we care about.

• It can't auto-merge into repos with a restricted committer list, like this one. (There's no way to add a "github app" to the allowed committers list.)

• Its auto-merge feature is going to be problematic once we switch to a bors-like workflow where a bot handles merges in serial order. The bors workflow assumes that only the bors-bot commits to the master branch; we'd want the dependency updates to get routed into the bors-bot merge queue, but dependabot can't do that.

Our dependency bumping needs are actually super simple, and could be handled by a bot that does something like this:

• On some schedule, pull down the repo, and run pip-compile
• If this results in changes, push them back as a branch, and watch to see if the tests pass
• If the tests pass, merge it automatically without hassling anyone
• If the tests don't pass, convert the branch into a PR and leave it for a human to sort out

So maybe snekomatic should grow the ability to handle dependency bumping, integrated into any future bors-like features.

[webknjaz]

@greysteil ^

[greysteil]

Thanks for the ping. We're in the process of fixing (2), which will hopefully be of use for other apps (and would allow bors to act as an app). I'm also keen to add support for Dependabot making @bors r+ commands if it's useful.

Issue (1) is a bit trickier! I wouldn't expect that many PRs on a repo like this one, but Dependabot is built around a philosophy of creating one PR per update. We'd like to offer the ability to group updates in future, but realistically it won't be in the next 3-6 months.

[njsmith]

@greysteil Thanks for dependabot! This issue might come off as a bit complain-y, but dependabot has been super helpful to us; this is more "hmm, I wonder if our specific situation puts us in the part of the tradeoff continuum where it would be even better to roll something custom".

To explain the issue with notifications a bit more:

It's not just this repo... This org currently has 25 repos in it. I think right now dependabot is only registered on 4 of them? But it's still enough to cause quite a few PRs.

And each PR causes multiple emails: typically 1 when it's opened, 2 for codecov posting stats (why 2 instead of 1 I don't know, codecov is pretty mysterious, but that's another issue), and then 1 when dependabot auto-merges the PR.

And whenever we invite a new volunteer to join the org, they get subscribed to all these emails. (Right now we have 65 members, and it's rising all the time.)

Multiply those together and it adds up to a lot!

And then the thing is, most of the time, the CI tests pass, dependabot auto-merges the PR, and there's no reason at all for any human to be involved, so the notifications are pure spam. The only time I need to look is when the auto merge fails.

I totally get why on a lot of projects you'd want to see the updates happening even if the tests pass, in case they break something that wasn't being tested. But we're mostly developing libraries, not apps; when we actually ship our code, it's with vague dependency metadata and we leave it up to our users to decide how to handle pinning. We only pin stuff for purposes of CI testing, to avoid surprise breakage. So if the CI passes, that is a 100% reliable indication that this change isn't going to break our CI :-), and there's nothing else for a human to check even in principle.

So the net effect is a ton of email, and the thing I have to watch out for and notice is every once in a while when a certain email doesn't arrive.

[greysteil]

That's useful feedback - thanks for taking the time to write it. In future I'd like to add some settings to Dependabot to make it more appropriate for library setups like that, so that you only get a notification for out-of-range updates and for in-range dependency updates that break the build. There's not way to do that at the moment, however.

[webknjaz]

@njsmith FYI https://developer.github.com/changes/2019-09-05-apps-protected-branches-api/

[njsmith]

Ah cool, that's a nice change! I probably still won't use it to enable dependabot auto-merge here, b/c I'm not confident I understand dependabot's auto-merge rules well enough to know what the actual security implications would be. (Like, if someone who doesn't have permissions to master modifies the dependabot branch, will dependabot still auto-merge their changes to master?) But it's definitely a step forward :-)

[webknjaz]

FWIW with https://github.com/marketplace/actions/alls-green#why, it should be safer to use GitHub's built-in auto-merge feature now.

Also, I've got an interesting automation PoC at https://github.com/cherrypy/cheroot/blob/main/.github/workflows/pip-tools.yml — it bumps env-specific constraints files that are then auto-used in tox.