Ship precompiled gems for linux

[flavorjones]

Hello! If you're arriving here to give feedback on the v1.11.0 Release Candidates, please make sure to:

• tell us which linux distro you're using!
• (If you can) a docker image in which we can reproduce the failure you're seeing!

Thanks so much for using Nokogiri and for giving us feedback on this experiment!

---

See #1571 for the PR from @larskanis that enables building precompiled libraries for linux.

This issue is for discussion on how to ship those precompiled libraries in a gem for linux users in a way that won't break the installation experience for people.

Some things that we'll need to figure out were mentioned in #1571 by @larskanis:

1. Bundler isn't clever at selection of unsupported ruby versions. See Add two specs for choosing platform specific gems rubygems/bundler#6247 . This requires users to explicit opt-out of binary gems, when they want to run nokogiri on ruby-trunk.

2. There is a known issue on Alpine Linux: Cross building doesn't work for alpine3.7 rake-compiler/rake-compiler-dock#20 . I did a manual test on the alpine docker image and it reproducible fails at Nokogiri::XML::Reader#namespaces somewhere within the tests: https://gist.github.com/larskanis/94920901e6466c9383360ee1346f9f34 . I didn't analyze the reason for the segfault so far, but I think we should before the release.

And he later mentioned:

Issue 1 in the above comment will probably finally be fixed by the next bundler release. See rubygems/bundler#7522 . This way bundler will select the platform=ruby gem automatically, if fat binary gem doesn't fit to the ruby release.

I think we'll probably also be able to fix / work around issue 2, so that I hope we'll be able to ship binary releases some day.

Finally, I'd like for VersionInfo to contain some indication that the user is using precompiled libraries. A patch that worked at one point on the #1571 branch (but now needs to be rewritten) is:

From 20f69908732b50258be0ee18e1d78b841d462dc4 Mon Sep 17 00:00:00 2001
From: Mike Dalessio <mike.dalessio@gmail.com>
Date: Sun, 13 Jan 2019 01:15:06 -0500
Subject: [PATCH] `nokogiri -v` emits info about precompiled binaries

It may be useful information to be able to tell if the libraries were
compiled on installation, or whether they were precompiled in a fat
binary native gem.
---
 Rakefile                | 3 +++
 ext/nokogiri/nokogiri.c | 6 ++++++
 lib/nokogiri/version.rb | 6 ++++++
 3 files changed, 15 insertions(+)

diff --git a/Rakefile b/Rakefile
index 5ef937d..72edb7e 100644
--- a/Rakefile
+++ b/Rakefile
@@ -383,6 +383,9 @@ task :cross do
     raise "rake-compiler has not installed any cross rubies. Use rake-compiler-dock or 'rake gem:windows' for building binary windows gems."
   end
 
+  ENV['CFLAGS'] ||= ""
+  ENV['CFLAGS'] += " -DNOKOGIRI_USE_PRECOMPILED_NATIVE"
+
   CROSS_RUBIES.each do |cross_ruby|
     task "tmp/#{cross_ruby.platform}/nokogiri/#{cross_ruby.ver}/nokogiri.so" do |t|
       # To reduce the gem file size strip mingw32 dlls before packaging
diff --git a/ext/nokogiri/nokogiri.c b/ext/nokogiri/nokogiri.c
index 740b30d..5b32500 100644
--- a/ext/nokogiri/nokogiri.c
+++ b/ext/nokogiri/nokogiri.c
@@ -104,6 +104,12 @@ void Init_nokogiri()
   rb_const_set(mNokogiri, rb_intern("NOKOGIRI_LIBXSLT_PATCHES"), Qnil);
 #endif
 
+#ifdef NOKOGIRI_USE_PRECOMPILED_NATIVE
+  rb_const_set(mNokogiri, rb_intern("NOKOGIRI_USE_PRECOMPILED_NATIVE"), Qtrue);
+#else
+  rb_const_set(mNokogiri, rb_intern("NOKOGIRI_USE_PRECOMPILED_NATIVE"), Qfalse);
+#endif
+
 #ifdef LIBXML_ICONV_ENABLED
   rb_const_set(mNokogiri, rb_intern("LIBXML_ICONV_ENABLED"), Qtrue);
 #else
diff --git a/lib/nokogiri/version.rb b/lib/nokogiri/version.rb
index 44c076a..c05e1a0 100644
--- a/lib/nokogiri/version.rb
+++ b/lib/nokogiri/version.rb
@@ -46,6 +46,10 @@ module Nokogiri
       NOKOGIRI_USE_PACKAGED_LIBRARIES
     end
 
+    def libxml2_using_precompiled_native?
+      NOKOGIRI_USE_PRECOMPILED_NATIVE
+    end
+
     def warnings
       warnings = []
 
@@ -78,6 +82,7 @@ module Nokogiri
           vi["libxml"] = {}.tap do |libxml|
             if libxml2_using_packaged?
               libxml["source"] = "packaged"
+              libxml["precompiled"] = libxml2_using_precompiled_native?
               libxml["patches"] = NOKOGIRI_LIBXML2_PATCHES
             else
               libxml["source"] = "system"
@@ -89,6 +94,7 @@ module Nokogiri
           vi["libxslt"] = {}.tap do |libxslt|
             if libxml2_using_packaged?
               libxslt["source"] = "packaged"
+              libxslt["precompiled"] = libxml2_using_precompiled_native?
               libxslt["patches"] = NOKOGIRI_LIBXSLT_PATCHES
             else
               libxslt["source"] = "system"
-- 
2.17.1

And tagging @tenderlove for visibility because he had some thoughts on the previous #1571 thread.

[larskanis]

bundler-2.2.0 will fix the issues with binary gems. I already played a bit with it's improved multiplatform support on Windows. I didn't test nokogiri on Linux so far, but a nokogiri pre-release will allow easy testing.

Regarding Alpine Linux (aka musl libc) I did some research and found out that sprintf() doesn't work properly but segfaults as soon as any arguments are used. It is only used once here and can easily get replaced by strcat() / strcpy().

After fixing the sprintf segfault, there is one failing test case on Alpine with binary gem:

Nokogiri::HTML::TestDocumentEncoding#test_encoding_without_charset:
ArgumentError: invalid byte sequence in UTF-8
    /comcard/nokogiri/test/html/test_document_encoding.rb:26:in `test_encoding_without_charset'

The test case succeeds with Shift_JIS encoding but fails with cp932 although they are mostly interchangeable. I'll try to find the root cause.

[flavorjones]

@larskanis Just want to confirm: you think I should ship an RC for precompiled linux? I can do that on Monday if so.

I can add Alpine to the CI pipelines -- that might be the easiest way to tell if something's broken. I'll try to do that in the next day or so.

[flavorjones]

@larskanis They're out there! https://twitter.com/flavorjones/status/1224334774753513473

[casperisfine]

Some feedback on precompiled binaries in general, not nokogiri per say.

On paper precompiled binaries are great as they are much faster to install, however they are a huge pain whenever a new Ruby version is released.

I just had this issue with the grpc and google-protobuf gems. I wanted to try the 2.7.0 RCs but couldn't because precompiled gems put a upper requirement on ruby versions.

For instance https://rubygems.org/gems/nokogiri/versions/1.11.0.rc1-x86-linux has:

REQUIRED RUBY VERSION:
>= 2.4, < 2.8.DEV

So let's imagine I use that version in my app, and by the end of 2020 I try to test the Ruby 3 RCs (or the 3.0 release in January 2021).

I won't be able to bundle again because of that ruby version requirement. In theory bundler could fallback to the source gem (https://rubygems.org/gems/nokogiri/versions/1.11.0.rc1), but there's only one global flag which isn't very practical.

Aaron had the same problem with grpc: https://twitter.com/tenderlove/status/1222937689751605248

I realize this is more of a Bundler feedback than a nokogiri one, but my personal opinion is that bundler/rubygems/whatever would need to be improved before popular gems start pushing precompiled binaries.

[flavorjones]

@casperisfine Thanks for the feedback. This should be addressed in Bundler 2.2, see rubygems/bundler#7522 (mentioned above) for more context.

[casperisfine]

🤦‍♂ Sorry, I read the issue description way too quickly. Great to hear this soon won't be a problem anymore.

[voxik]

Overall, the best way to install nokogiri on Fedora is via dnf install rubygem-nokogiri and Fedora user does not need to care.

Nevertheless, the runtime dependency on mini_portile2 is annoying for pre-built packages.