RFC: change our Ruby version support policy for RSpec 4

[myronmarston]

TL;DR

I propose that we change our Ruby version support policy for RSpec 4 so that dropping support for old ruby versions is no longer coupled to major version releases. Instead, I recommend we adopt a policy based around supporting old Ruby versions for N months after the Ruby core team has EOL'd
it (for some reasonable value of N).

The Current State of Things

We currently support 10 6 versions of MRI:

• 1.8.7
• 1.9.2
• 1.9.3
• 2.0
• 2.1
• 2.2
• 2.3
• 2.4
• 2.5
• 2.6
• 2.7
• 3.0

In addition, we have been strictly interpreting SemVer to mean that the only time we can drop support for an old version of Ruby is when we do a major release. As a result, we drop support for old versions very, very rarely. In fact, in all my years of involvement with RSpec (dating back to 2010), I can only think of one version of Ruby we've ever dropped support for: we stopped supporting 1.8.6 when RSpec 3 was released.

I think the amount of effort we expend supporting old versions vastly exceeds the utility to our users.

Why is this a problem?

Supporting old Ruby versions has a very real maintenance cost:

• Supporting old versions bloats our Travis build matrix, costing Travis more resources, and slowing down our feedback cycle from our CI builds.
• Our CI build does not consistently pass on old Ruby versions for reasons outside our control. For example, our CI builds get intermittent segfaults on MRI 1.9.2 and REE. Dealing with those failures (kicking the build, etc) has a very real maintenance cost.
• Continuing to support old Ruby versions has prevented us from upgrading our dependencies. For example, we are still on Cucumber 1.x, even though Cucumber 3.1 5.2 is out. Cucumber does not support ruby 1.8.7 starting with Cucumber 2, so we haven't been able to upgrade. Staying on old versions of dependencies causes us to accumulate excess tech debt.
• In my experience, there's often extra work to get my PRs green against the old Ruby versions. For example, I often accidentally use the newer hash syntax, which 1.8.7 cannot parse. After the CI build finishes, I see the failure, fix it, and force push to address the issue. This creates extra work for what should be simple changes.
• Supporting old Ruby versions limits our ability to take advantage of new Ruby features.

Regarding SemVer

The SemVer spec does not specifically document when a library is allowed to drop support for an old version of a language. I'd argue that it's a grey area that we're allowed to decide on for our project. And even if it did explicitly document it, it's up to us to decide if we want to strictly follow it. IMO, the important thing about SemVer is that your project documents the semantics of your versioning and sticks to it--not necessarily that you follow the SemVer spec. IMO, if we realize that strict adherence to the SemVer spec was inhibiting our ability to serve our users due to extra work it requires, it would be wise and appropriate to choose not to follow the spec anymore, and decide on our own semantic that we think better serves our users.

That said, I don't believe the SemVer spec requires that we maintain support for old versions of the language until the next major release.

Coupling Version Drops to Major Releases is a problem

While we strive to follow SemVer, bumping RSpec's major version is not purely a technical decision. It's also a marketing decision. The Ruby community has certain expectations around a major release of a project like RSpec containing major new features. In addition, there are lots of published materials (such as my own book) that reference RSpec's version number.

That means that we can't simply rev the major version number more often, as a means to drop old ruby versions. Besides, I think our users appreciate the stability of a major RSpec version line lasting for years.

Given that major new RSpec versions have historically only happened every 3-5 years, and our recent productivity is, if anything, less than it's been in the past, I think that coupling Ruby version support to major releases is problematic. We could better serve our users (by more productively addressing their issues, building new features, etc) if we were able to drop support for old versions of Ruby between major versions.

So, I believe that beginning with RSpec 4, we should come up with a new Ruby version policy that is not coupled to major releases.

What should we do instead?

I propose that we allow ourselves to drop support for old versions of Ruby as part of a minor release (e.g. x.y.0) in addition to major releases. In addition, I think it would be beneficial to have a specific policy about when we will drop old versions, so that it's not arbitrary based on when we feel like it. The Ruby core team announces the support schedule for each version of Ruby, and that includes an EOL date. I think we should adopt a policy of dropping support for old versions of ruby in the next minor release that comes N months after a version is EOL'd (for some reasonable value of N, such as 6, 12, 18 or 24).

Here's a concrete example. Ruby 2.0 was EOL'd on 2016-02-24. If we had adopted this policy for RSpec 3, and chose N=24, we would be dropping support for Ruby 2.0 in RSpec 3.8, which is the next minor release, since 2 years has elapsed since it was EOL'd.

How would this effect users?

• For users who use RSpec to test applications: I do not expect this to have much effect (if any) on these users. This could really only effect users who are running really old versions of Ruby that are months or years past the point where the Ruby core team is supporting it. I doubt there are any users who are allowing their Ruby version to remain so far behind while also staying on the latest version of RSpec.
• For users who use RSpec to test gems: users who support old versions of Ruby that we no longer support will be effected. They won't be able to upgrade RSpec without dropping support for old ruby versions. However, if we pick a large enough value of N, the number of users who are effected by this should be small.

Questions

1. Thoughts on the idea as a whole?
2. If we went with this proposal, what value of N should we pick?

[iGEL]

I believe that users who use a very old Ruby version but need to be on the latest Rspec version are very rare and not worth the effort you correctly mentioned. In my opinion, Nshould be 6 or max. 12 months.

[dnch]

1. Yes.
2. 12 months. Drive upgrades.

[xaviershay]

I'm most worried about ensuring we support bundler, though maybe with that being folded into rubygems it gets the same EOL policy as Ruby? In which case I'm more inclined towards a more aggressive N, maybe even matching them directly.

@indirect do you have any concerns about RSpec dropping old ruby version support (above policy)?

From the travis config, looks like bundler 2 is supporting 2.3+ but bundler 1.X is still supporting 1.8.7 https://github.com/bundler/bundler/blob/master/.travis.yml#L57

[JonRowe]

Older versions of Ruby, especially those > 2 do not cause us significant maintenance, so I'm heavily against a short N and in general not in favour of this. I'm ok with us dropping < 2 support in RSpec 4, but I don't really like the idea of dropping Ruby support in anything less than major versions, particularly because Bundler / rubygems will attempt to install the latest version regardless of your ruby specified e.g. so people who say use ~> 4.x will get error messages when we drop ruby versions without them doing anything, where as if we do it on major versions

I really don't think we should worry about the major version "expectation" we should just publicly state what we're doing and carry on, I'm also no at all concerned about references to versions in media, people expect those to be out of date, especially printed books.

[myronmarston]

Older versions of Ruby, especially those > 2 do not cause us significant maintenance,

I disagree. Every additional version of ruby that we support adds to our maintenance burden. It is a qualitatively different burden for Ruby 2.x compared to 1.x, but it's still there. And even if we felt like the 1.x burden is the main thing needing to be addressed, I'd argue that's reason enough to consider a different policy. The reason we still support 1.x is because we chose a policy that dictates that the only time we can drop support for a version is as part of a major version release, which only gives us the opportunity every few years. When we release a new version, I don't want the ruby versions supported at that time to impose a hard requirement on the RSpec maintainers for the next 3+ years due to our versioning policy.

particularly because Bundler / rubygems will attempt to install the latest version regardless of your ruby specified e.g. so people who say use ~> 4.x will get error messages when we drop ruby versions without them doing anything, where as if we do it on major versions

I assume we'd use the required_ruby_version field of our gemspec to declare what ruby versions we support in the metadata of our published gems. I thought bundler took this into account and wouldn't try to upgrade to a version that can't be installed according to it's required_ruby_version?

If bundler doesn't take that into account, it sounds like something we should push to have improved there.

I really don't think we should worry about the major version "expectation" we should just publicly state what we're doing and carry on, I'm also no at all concerned about references to versions in media, people expect those to be out of date, especially printed books.

My point was simply that we can't rev the major version as often as we'd like to be able to drop support for an old Ruby version. For example, my publisher really wouldn't have appreciated us releasing RSpec 4 right after they published my book on RSpec 3 (which took significant time and resources on their part). So I do not consider reving the major version of RSpec more often as a realistic possibility.

[JonRowe]

I disagree. Every additional version of ruby that we support adds to our maintenance burden. It is a qualitatively different burden for Ruby 2.x compared to 1.x, but it's still there.

Sure theoretically, in practise it's minimal. We spend more time solving peoples trivial issues that we do solving EOL'd Ruby bugs.

If support burden is the only reason we drop Rubies we could drop RBX and JRuby which cause us the most grief.

I assume we'd use the required_ruby_version field of our gemspec to declare what ruby versions we support in the metadata of our published gems. I thought bundler took this into account and wouldn't try to upgrade to a version that can't be installed according to it's required_ruby_version?

If bundler doesn't take that into account, it sounds like something we should push to have improved there.

It doesn't. Our own builds fall foul of this, which is why we have ruby version if blocks in them.

My point was simply that we can't rev the major version as often as we'd like to be able to drop support for an old Ruby version.

We haven't done it because we haven't felt the time is right, it's not that we can't.

For example, my publisher really wouldn't have appreciated us releasing RSpec 4 right after they published my book on RSpec 3

If we made major releases more common, without the expectation of major features, this becomes a non issue. It was only an issue for 2 -> 3 because we are made other radical feature changes at the same time. Arguably having a 3 -> 4 release (as we've discussed before) that only discards Ruby versions makes us more stable not less.

[iGEL]

It's possible that I missed the point of what you said, but it seems to me that bundler takes the required_ruby_version into account. I just tried to install Rails on Ruby 2.1/Bundler 1.15.1 and got version 4.2.10, not 5. My Gemspec was:

source "https://rubygems.org"

gem "rails"

When I tried to force version "5.1.0", I got:

Gem::InstallError: rack requires Ruby version >= 2.2.2.
An error occurred while installing rack (2.0.4), and Bundler cannot continue.
Make sure that `gem install rack -v '2.0.4'` succeeds before bundling.

In Gemfile:
  rails was resolved to 5.1.0, which depends on
    actioncable was resolved to 5.1.0, which depends on
      actionpack was resolved to 5.1.0, which depends on
        rack-test was resolved to 0.6.3, which depends on
          rack

[myronmarston]

Thanks for actually trying it, @iGEL :). Seems like we don't have to worry about bundler auto-upgrading users to a version they can't use as long as we declare required_ruby_version correctly.

Sure theoretically, in practise it's minimal. We spend more time solving peoples trivial issues that we do solving EOL'd Ruby bugs.

EOL'd Ruby bugs are a tiny part of the maintenance cost I'm talking about. Here are some things I'm talking about:

• Slower travis builds because we have so many parts to our build matrix
• More frequent flaky travis builds (again, due to so many parts of the build matrix)
• Inability to upgrade our dependencies. While dropping 1.8.7 and 1.9.2 would allow us to upgrade to cucumber 2.x, we can't upgrade to cucumber 3.x until we drop Ruby 1.9.3, 2.0 and 2.1.
• Inability to take advantage of new Ruby features. For example, it would be really nice to take advantage of Ruby 2's keyword args (especially required keyword args available in ruby 2.1+). We can't do that until we've dropped support for ruby < 2.1.

Maybe you haven't felt that continuing to support old versions of Ruby is burdensome, but I certainly have.

[xaviershay]

FWIW in practice I've found it pretty painful supporting old Ruby versions. Particularly when I was working actively with mocks.

[JonRowe]

Thanks for actually trying it, @iGEL :). Seems like we don't have to worry about bundler auto-upgrading users to a version they can't use as long as we declare required_ruby_version correctly.

Our own builds say otherwise.

From https://travis-ci.org/rspec/rspec-expectations/jobs/323984512

We, by merit of being one of the most popular testing framework have to be the lowest bar when it comes to Ruby support. If we drop a Ruby version we are effectively orphaning everyone on that Ruby version that depends on us. Currently we are able to say "upgrade to the latest version of RSpec" for a lot of enquiries from older versions, if we start dropping Ruby versions we may well have to start back porting bug fixes to the maintenance branches. I feel that burden outweighs the burden of supporting multiple older Rubies on the current branch.

Especially when we don't have any stats of how many people actively use RSpec on those versions, if rubygems was able to provide us that I'd feel more confident we could do this safely.

[tomstuart]

Speaking only as a grateful user of RSpec who otherwise has no skin in the game:

I absolutely agree that you should find a way to gracefully drop support for EOL Ruby versions. I wish you had unlimited resources but I understand that you don’t. Your time is limited and precious, so please use it to maintain & improve RSpec for the vast majority of Ruby users.

However, I want to say that I do disagree with the point about “community expectations” around the content, significance and rarity of major RSpec releases. It’s a somewhat circular argument: yes, at the moment it’s years between major RSpec versions, so they are de facto A Big Deal. But if you’re going to change something, that is an essentially harmless thing to change, versus violating the assumption that new minor releases don’t break compatibility. The scope of the “RSpec major releases are a big deal” expectation is only as large as the more engaged portion of the RSpec userbase; the scope of “minor releases don’t break things” is orders of magnitude larger, so it’s much harder to push against and, crucially, not something that you can ever change outside of your own project. But you can easily choose to cut major RSpec versions more often, and everyone will just get used to it.

In summary, my gut feeling is that you’ll piss more people off by dropping Ruby support in minor releases than you will by simply releasing more frequent major versions of RSpec. People would quickly adapt to the latter, especially given the changes proposed in rspec/rspec-rails#1967.