Bug 1482641 - ruby:2.3 image fails to build nokogiri 1.8
Summary: ruby:2.3 image fails to build nokogiri 1.8
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat Software Collections
Classification: Red Hat
Component: rh-ruby23-container
Version: rh-ruby23
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: ruby maint
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-17 19:19 UTC by Aleksandar Kostadinov
Modified: 2019-06-14 14:18 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-14 14:18:27 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Aleksandar Kostadinov 2017-08-17 19:19:09 UTC
Description of problem:
Trying to build an app via s2i it fails on building nokogiri 1.8.0

Nokogiri is a mandatory gem for most (all) rails applications. I think it is high priority to support building it in our standard s2i images.

Version-Release number of selected component (if applicable):
registry.access.redhat.com/rhscl/ruby-23-rhel7 
"Digest": "sha256:7b96ad3a8bfcd29a880d80d8d22f33a32cb91562c16d78d3fcdfafc997c1e9ba"

How reproducible:


Steps to Reproduce:
1. try to build an app with nokogiri 1.8.0 in Gemfile

Actual results:

> ---> Installing application source ...
> ---> Building your Ruby application from source ...
> ---> Running 'bundle install --deployment --without development:test' ...
> Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
> Fetching gem metadata from https://rubygems.org/.........
> Fetching version metadata from https://rubygems.org/...
> Fetching dependency metadata from https://rubygems.org/..
> Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running `gem install bundler`.
> Installing rake 12.0.0
> Installing concurrent-ruby 1.0.5
> Installing i18n 0.8.6
> Installing minitest 5.10.3
> Installing thread_safe 0.3.6
> Installing tzinfo 1.2.3
> Installing activesupport 5.0.5
> Installing builder 3.2.3
> Installing erubis 2.7.0
> Installing nokogiri 1.8.0 with native extensions
> Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
> current directory: /opt/app-root/src/bundle/ruby/2.3.0/gems/nokogiri-1.8.0/ext/nokogiri
> /opt/rh/rh-ruby23/root/usr/bin/ruby -r ./siteconf20170817-16-eixitt.rb extconf.rb
> checking if the C compiler accepts ... yes
> Building nokogiri using packaged libraries.
> *** extconf.rb failed ***
> Could not create Makefile due to some reason, probably lack of necessary
> libraries and/or headers.  Check the mkmf.log file for more details.  You may
> need configuration options.
> Provided configuration options:
> --with-opt-dir
> --without-opt-dir
> --with-opt-include
> --without-opt-include=${opt-dir}/include
> --with-opt-lib
> --without-opt-lib=${opt-dir}/lib64
> --with-make-prog
> --without-make-prog
> --srcdir=.
> --curdir
> --ruby=/opt/rh/rh-ruby23/root/usr/bin/$(RUBY_BASE_NAME)
> --help
> --clean
> --use-system-libraries
> /opt/rh/rh-ruby23/root/usr/share/rubygems/rubygems/dependency.rb:319:in `to_specs': Could not find 'mini_portile2' (~> 2.2.0) among 9 total gem(s) (Gem::LoadError)
> Checked in 'GEM_PATH=/opt/app-root/src/bundle/ruby/2.3.0', execute `gem env` for more information
> from /opt/rh/rh-ruby23/root/usr/share/rubygems/rubygems/dependency.rb:328:in `to_spec'
> from /opt/rh/rh-ruby23/root/usr/share/rubygems/rubygems/core_ext/kernel_gem.rb:65:in `gem'
> from extconf.rb:460:in `<main>'
> To see why this extension failed to compile, please check the mkmf.log which can be found here:
> /opt/app-root/src/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0/nokogiri-1.8.0/mkmf.log
> extconf failed, exit code 1
> Gem files will remain installed in /opt/app-root/src/bundle/ruby/2.3.0/gems/nokogiri-1.8.0 for inspection.
> Results logged to /opt/app-root/src/bundle/ruby/2.3.0/extensions/x86_64-linux/2.3.0/nokogiri-1.8.0/gem_make.out
> An error occurred while installing nokogiri (1.8.0), and Bundler cannot
continue.
> Make sure that `gem install nokogiri -v '1.8.0'` succeeds before bundling.
> error: build error: non-zero (13) exit code from registry.access.redhat.com/rhscl/ruby-23-rhel7

Expected results:
build completes successfully

Comment 2 Aleksandar Kostadinov 2017-08-17 19:32:35 UTC
FYI just confirmed nokogiri 1.7.2 builds fine. We shouldn't be forcing people to use old gems though.

Comment 4 Vít Ondruch 2017-09-14 06:53:27 UTC
Well ... this is a bit ironic, since in reality, you don't need mini_portile2 neither to build nor to run Nokogiri. Not sure if there is any solution, since I don't think we are going to add mini_portile2 gem into image.

But this is workaround:

~~~
$ gem fetch nokogiri
Fetching: nokogiri-1.8.0.gem (100%)
Downloaded nokogiri-1.8.0

$ gem unpack nokogiri-1.8.0.gem 
Unpacked gem: '/home/vondruch/ff/nokogiri-1.8.0'

$ gem spec --ruby nokogiri-1.8.0.gem > nokogiri.gemspec

$ cd nokogiri-1.8.0/

$ sed -i '/mini_portile/ s/^/#/' nokogiri.gemspec 

$ cd nokogiri-1.8.0/

$ gem build ../nokogiri.gemspec 

$ NOKOGIRI_USE_SYSTEM_LIBRARIES=true gem install nokogiri-1.8.0.gem
~~~

Comment 5 Vít Ondruch 2017-09-14 06:55:44 UTC
Ups, made one mistake there :/ the "cd" after "gem spec" should not be there. Sorry.

Comment 6 Aleksandar Kostadinov 2017-09-14 08:44:14 UTC
I don't think this is a workaround because users would do:

> oc new-app registry.access.redhat.com/rhscl/ruby-23-rhel7~https://github.com/openshift-qe/myapp

You don't provide a script how your app is built. It is the s2i logic that installs gems.

If nokogiri 1.8 has broken spec, then perhaps somebody can submit a pull request?

Otherwise customer s2i builds will fail with nokogiri 1.8. And nokogiri is a requirement for all Rails apps AFAIK. I don't think it is reasonable from our side to not support that.

Comment 7 Aleksandar Kostadinov 2017-09-14 09:51:37 UTC
I created a simple test app and here is the exact command to reproduce the issue. App is not rails, I just added nokogiri to deps so it works quicker.

> oc new-app "registry.access.redhat.com/rhscl/ruby-23-rhel7~https://github.com/openshift-qe/ruby-ex#nokogiri18"

To see error by:

> oc logs bc/ruby-ex

What should we advice customers building apps with a recent version of nokogiri in the Gemfile?

Or should we submit a patch to nokogiri?

Comment 8 Vít Ondruch 2017-09-14 16:18:24 UTC
Ok, scratch my previous comment, although there is relation ...

The issue is with your Gemfile.lock. It seems you have created your Gemfile.lock using packaged version of rubygem-nokogiri, which we have either in Fedora or in some RHSCL collection. This package differs a bit from the upstream in a way, that it drops the dependency on mini_portile2 (the comment 4 actually demonstrates how and why we do that). If you generated the Gemfile.lock using upstream nokogiri, you would have the dependency on mini_portile2 in your Gemfile.lock. But you don't have it there ...

So there are several workarounds possible:

1) Drop the Gemfile.lock from your repository and everything should work.
2) Use the upstream version of nokogiri on the system where you are generating the Gemfile.lock (this is actually similar to what the ruby container setup, which is using just ruby and rubygem-bundler, all the remaining packages are pristine upstream version downloaded from rubygems.org).
3) There used to be ror-container in earlier versions of OS. This container was using RHSCL on background and was shipping with nokogiri if I am not mistaken. Not sure if it is still available or supported ...

[1] https://github.com/sclorg/ror-container

Comment 9 Pavel Valena 2017-09-14 16:47:47 UTC
(In reply to Vít Ondruch from comment #8)
</snip>
> So there are several workarounds possible:
> 
> 1) Drop the Gemfile.lock from your repository and everything should work.

I can confirm that, using:
https://github.com/pvalena/ruby-ex/tree/no-lock

and running this app(as in Comment 6). The container builds fine:
```
$ oc logs ruby-ex-1-build
Cloning "https://github.com/pvalena/ruby-ex" ...
        Commit: 0ab8ede0f44c76ab41d9305b04421df64472a78d (Remove Gemfile.lock)
        Author: Pavel Valena <pvalena@redhat.com>
        Date:   Thu Sep 14 18:33:48 2017 +0200
---> Installing application source ...
---> Building your Ruby application from source ...
---> Running 'bundle install  --without development:test' ...
Fetching gem metadata from https://rubygems.org/..............
Fetching version metadata from https://rubygems.org/..
Resolving dependencies...
Installing mini_portile2 2.2.0
Installing nokogiri 1.8.0 with native extensions
 . . . 

```

and the resulting pod:
```
$ oc rsh ruby-ex-1-gs55b
sh-4.2$ bundle exec bash
bash-4.2$ gem list | grep nokogiri
nokogiri (1.8.0)

```

> 2) Use the upstream version of nokogiri on the system where you are
> generating the Gemfile.lock (this is actually similar to what the ruby
> container setup, which is using just ruby and rubygem-bundler, all the
> remaining packages are pristine upstream version downloaded from
> rubygems.org).

Using 'scl enable rh-ruby23' on Red Hat Enterprise Linux 7 (no nokogiri installed) and running 'bundle install', results in:

https://github.com/pvalena/ruby-ex/tree/nokogiri18-fix

Which works, same as above.

> 3) There used to be ror-container in earlier versions of OS. This container
> was using RHSCL on background and was shipping with nokogiri if I am not
> mistaken. Not sure if it is still available or supported ...

AFAIK you cannot use that reasonably in OpenShift, as it lacks the s2i functionality.

Comment 10 Aleksandar Kostadinov 2017-09-14 18:05:20 UTC
Wow, I didn't realize I'm using packaged nokogiri. I think it is a very bad practice to package gem with same name and same version but modified from upstream. This is how developers using RHEL and Fedora can be highly confused.

If we want to ship a fork, then we should also change the name of the gem. If we truly believe mini_portile2 should not be part of nokogiri spec, then submit upstream pull request.

Could you advise where should bug be mored to reach responsible RHEL nokogiri package maintainer? I'm reopening the bug because current package of nokogiri is IMO broken.

Comment 11 Aleksandar Kostadinov 2017-09-14 18:13:45 UTC
Or maybe, if we have nokogiri installed by default in the image, it will also workaround the issue. The problem is I guess, that we can't know which version of Fedora/RHEL user used to generate Gemfile.lcok, thus one version of nokogiri can't do the job.

In any case, I believe we need to make it more user friendly or issue a meaningful error message in logs how user can fix the problem.

Comment 12 Vít Ondruch 2017-09-15 15:31:09 UTC
(In reply to Aleksandar Kostadinov from comment #10)
> Wow, I didn't realize I'm using packaged nokogiri.

Unfortunately you still did not provided enough information about your environment. It would be nice if you give use some insights. Was it RHEL and RHSCL? Which collections you had enabled? Or was it Fedora?

Anyway, I don't think we are going to change Fedora neither RHSCL, since mini_portile2 is useless to us and it does things which goes precisely against what the principles distributions are trying to achieve. I.e. it download source code of libraries and bundles them instead of using the system libraries.

And neither I see upstream is going to change. It won't change as long as bundler is installing libraries during runtime.

It would be nice to provide some better error message, but unfortunately, I don't see any way how we could achieve it in some generic manner, but I am open to suggestions.

At the end, I think that you should make sure you are using the same environment for development as you are using for production, otherwise you'll face challenges as this one.

Comment 13 Pavel Valena 2017-09-15 16:03:58 UTC
(In reply to Aleksandar Kostadinov from comment #10)
> Wow, I didn't realize I'm using packaged nokogiri. I think it is a very bad
> practice to package gem with same name and same version but modified from
> upstream. This is how developers using RHEL and Fedora can be highly
> confused.

FTR: since Nokogiri 1.6.0 (2013), system libraries in Fedora are used:
https://src.fedoraproject.org/rpms/rubygem-nokogiri/c/7b46db2a696f69641e5a37d0d6a1aca9ee343892

Comment 15 Aleksandar Kostadinov 2017-09-25 21:48:23 UTC
I am using Fedora 26. In any case, by forking the nokogiri gem, we are breaking normal operation of bundle. And the whole point of bundle is to allow ruby dependencies setup in any env you run it.

I see as the only reasonable option to make mini-portile2 unsupported but ship it to avoid dependency issues. How will that hurt? I've read in the wild comments that we are breaking some ruby stuff and recommendation was to better use upstream versions.

This is the first time I see it happen (probably because I deploy on RH/fedora) but I find it pretty frustrating. I think it will be beneficial to community and our distribution to stop shipping any breaking changes.

What exactly is the point in shipping these libraries under the same name if they will not behave in the same way?

Comment 16 Pavel Valena 2017-09-26 13:54:38 UTC
(In reply to Aleksandar Kostadinov from comment #15)
> I am using Fedora 26. In any case, by forking the nokogiri gem, we are
> breaking normal operation of bundle. And the whole point of bundle is to
> allow ruby dependencies setup in any env you run it.

I'm afraid this is not caused by anything we ship in container. Your Gemfile.lock is incorrect/incompatible with upstream Nokogiri.

Due to this, I belive it is a Fedora issue/incompatibility and not caused by container.
However, only the dependencies differ, as we use the system ones(Fedora in this case), but functionality is completely same.

You can resolve it by using upstream Nokogiri. IOW, having the same environement as in container. Alternatively, this command should (on any OS) produce correct Gemfile.lock:

```
$ bundle install --disable-shared-gems

```

> I see as the only reasonable option to make mini-portile2 unsupported but
> ship it to avoid dependency issues. How will that hurt? I've read in the
> wild comments that we are breaking some ruby stuff and recommendation was to
> better use upstream versions.

From my POV it does not make sense to ship a dependency because of custom/incompatible developer's environment, that has no direct connection to the container.
Other distributions could have similar/other differencies. Do we intent to keep compatible with all/most of them? As of now, container is fully compatible with upstream. Fedora is not(with regard to the dependency). On the contrary, by packaging other dependencies(mini_portile2), we could introduce some incompatibilities(however improbably).

Note that this could also by solved by packaging Nokogiri into the container(IOW using ror-container).

> This is the first time I see it happen (probably because I deploy on
> RH/fedora) but I find it pretty frustrating. I think it will be beneficial
> to community and our distribution to stop shipping any breaking changes.

This is not deployment issue. I suppose you'd have the same issue with other containers(based on different OS).

> What exactly is the point in shipping these libraries under the same name if
> they will not behave in the same way?

Please see Comment 4 with regard to that.

Comment 17 Aleksandar Kostadinov 2017-09-26 14:29:53 UTC
I agree the container is compatible with upstream gems. This at least is good.

If is indeed an incompatibility in Fedora.

But is also incompatibility of nokogiri that we ship with **SCL** (rh-ror42) according to comment 16. My concern is that such incompatibilities sooner or later bite the user and user becomes frustrated.

I don't see what other distros have to do with our decision how to ship things. We should aim at convenience and compatibility between our product and the community standard which is upstream ruby and rubygem repo.

I don't need workarounds. I try to use bits shipped by Red Hat exactly to check they work fine. And I report bugs if they don't. It is easy for me to always use RVM. I'm frustrated that I do the extra mile just to hit issues but then we keep the incompatible behavior.

I don't feel like we are providing an enjoyable dev environment in this way. My request is to reconsider the UX around our ruby packaging wherever it is incompatible with upstream. Otherwise the work for packaging and maintaining such bits is not nearly as worthy as it could be in my opinion.

That's it from me. If such bugs are rejected, I guess I'll just ignore strange behavior that can be fixed by using upstream bits going forward. My desire is just to make RHEL and Fedora a more enjoyable dev environment (and they are, just we shouldn't stop making them more so).

Regards,
Aleksandar

P.S. wrt comment 4, mini_portile being shipped doesn't mean it must be used for building the gem.

Comment 20 Joe Orton 2019-03-14 11:02:42 UTC
Red Hat does not currently plan to provide any further changes to this collection in a Red Hat Software Collections update release.

This software collection is nearing the retirement date (May 2019) after which customers are encouraged to upgrade to a later release.

Please contact Red Hat Support if you have further questions, or refer to the support lifecycle page for more information. https://access.redhat.com/support/policy/updates/rhscl/

Comment 21 Joe Orton 2019-06-14 14:18:27 UTC
In accordance with the Red Hat Software Collections Product Life Cycle, the support period for this collection has ended.

New bug fix, enhancement, and security errata updates, as well as technical support services will no longer be made available for this collection.

Customers are encouraged to upgrade to a later release.

Please contact Red Hat Support if you have further questions, or refer to the support lifecycle page for more information. https://access.redhat.com/support/policy/updates/rhscl/


Note You need to log in before you can comment on or make changes to this bug.