Bug 1411594 - Cannot install as Vagrant plug-in using the embedded Ruby
Summary: Cannot install as Vagrant plug-in using the embedded Ruby
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine-sdk-ruby
Classification: oVirt
Component: Build
Version: 4.0.6
Hardware: x86_64
OS: Linux
unspecified
medium vote
Target Milestone: ovirt-4.1.0-rc
: 4.0.7
Assignee: Juan Hernández
QA Contact: Pavel Stehlik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-10 03:34 UTC by marcus young
Modified: 2017-02-01 16:08 UTC (History)
4 users (show)

Fixed In Version: 4.0.7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-01 16:08:30 UTC
oVirt Team: Infra
rule-engine: ovirt-4.1+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github https://github.com/mitchellh vagrant issues 8184 None None None 2020-02-06 00:34:06 UTC
oVirt gerrit 69992 'None' MERGED Enable installation in Vagrant with embedded Ruby 2020-02-06 00:34:06 UTC
oVirt gerrit 70062 'None' MERGED Enable installation in Vagrant with embedded Ruby 2020-02-06 00:34:06 UTC
oVirt gerrit 70063 'None' MERGED Enable installation in Vagrant with embedded Ruby 2020-02-06 00:34:06 UTC

Description marcus young 2017-01-10 03:34:49 UTC
Description of problem:

For the vagrant-ovirt4 project I cannot get an embedded version of the sdk to compile.

Version-Release number of selected component (if applicable):
4.0.6

How reproducible:
always

Steps to Reproduce:
1. vagrant plugin install ovirt-engine-sdk

Actual results:
Installing the 'ovirt-engine-sdk' plugin. This can take a few minutes...
Building native extensions.  This could take a while...
Bundler, the underlying system Vagrant uses to install plugins,
reported an error. The error is shown below. These errors are usually
caused by misconfigured plugin installations or transient network
issues. The error from Bundler is:

ERROR: Failed to build gem native extension.

    /opt/vagrant/embedded/bin/ruby -r ./siteconf20170110-12000-sc6cd9.rb extconf.rb
checking for xml2-config... yes
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling ov_error.c
compiling ov_http_client.c
compiling ov_http_request.c
compiling ov_http_response.c
compiling ov_module.c
compiling ov_xml_reader.c
ov_xml_reader.c:24:30: fatal error: libxml/xmlreader.h: No such file or directory
 #include <libxml/xmlreader.h>
                              ^
compilation terminated.
make: *** [ov_xml_reader.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/myoung/.vagrant.d/gems/2.2.5/gems/ovirt-engine-sdk-4.0.6 for inspection.
Results logged to /home/myoung/.vagrant.d/gems/2.2.5/extensions/x86_64-linux/2.2.0/ovirt-engine-sdk-4.0.6/gem_make.out


Expected results:
install successfully

Additional info:
This is more a problem in general over linkages. The problem existed in both FFI and nokogiri but were resolved. The nokogiri fix: https://github.com/sparklemotion/nokogiri/commit/3c0137c3c284506b0eb010cbe15950b838478550

Related background issues: 
https://github.com/sparklemotion/nokogiri/issues/1104
https://github.com/mitchellh/vagrant/issues/3769 (see yveslaroche 's comment)
https://github.com/mitchellh/vagrant/issues/3769

Also existed in libffi. 

```
  dev.localdomain  myoung  ~  repos  github  ovirt-engine-sdk-ruby  gem install ovirt-engine-sdk
Building native extensions.  This could take a while...
Successfully installed ovirt-engine-sdk-4.0.6
Parsing documentation for ovirt-engine-sdk-4.0.6
Done installing documentation for ovirt-engine-sdk after 25 seconds
1 gem installed
  dev.localdomain  myoung  ~  repos  github  ovirt-engine-sdk-ruby  vagrant plugin install ovirt-engine-sdk
Installing the 'ovirt-engine-sdk' plugin. This can take a few minutes...
Building native extensions.  This could take a while...
Bundler, the underlying system Vagrant uses to install plugins,
reported an error. The error is shown below. These errors are usually
caused by misconfigured plugin installations or transient network
issues. The error from Bundler is:

ERROR: Failed to build gem native extension.

    /opt/vagrant/embedded/bin/ruby -r ./siteconf20170110-12425-1bjujuj.rb extconf.rb
checking for xml2-config... yes
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling ov_error.c
compiling ov_http_client.c
compiling ov_http_request.c
compiling ov_http_response.c
compiling ov_module.c
compiling ov_xml_reader.c
ov_xml_reader.c:24:30: fatal error: libxml/xmlreader.h: No such file or directory
 #include <libxml/xmlreader.h>
                              ^
compilation terminated.
make: *** [ov_xml_reader.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/myoung/.vagrant.d/gems/2.2.5/gems/ovirt-engine-sdk-4.0.6 for inspection.
Results logged to /home/myoung/.vagrant.d/gems/2.2.5/extensions/x86_64-linux/2.2.0/ovirt-engine-sdk-4.0.6/gem_make.out```

extconf.rb calls ```
  dev.localdomain  myoung  ~  repos  github  ovirt-engine-sdk-ruby  xml2-config --cflags
-I/usr/include/libxml2```

```
  dev.localdomain  myoung  ~  repos  github  ovirt-engine-sdk-ruby  ls -alh /opt/vagrant/embedded/include/libxml2/libxml/xmlreader.h
-rw-r--r--. 1 root root 13K Jun 23  2016 /opt/vagrant/embedded/include/libxml2/libxml/xmlreader.h
  dev.localdomain  myoung  ~  repos  github  ovirt-engine-sdk-ruby  sudo find /usr -name xmlreader.h
/usr/include/libxml2/libxml/xmlreader.h
```

Comment 1 Juan Hernández 2017-01-10 09:00:56 UTC
Some parts of the Ruby SDK are written in C, to interact with libcurl and libxml2, before building it you need to manually install 'libcurl-devel' and 'libxml2-devel', as described here:

  https://github.com/oVirt/ovirt-engine-sdk-ruby/#building

That is for a RPM based distribution, but should be similar for other distributions. Did you do that?

Comment 2 christian.grundmann 2017-01-11 08:40:25 UTC
(In reply to Juan Hernández from comment #1)
> Some parts of the Ruby SDK are written in C, to interact with libcurl and
> libxml2, before building it you need to manually install 'libcurl-devel' and
> 'libxml2-devel', as described here:
> 
>   https://github.com/oVirt/ovirt-engine-sdk-ruby/#building
> 
> That is for a RPM based distribution, but should be similar for other
> distributions. Did you do that?

The libcurl-devel and libxml2-devel packages are installed,
otherwise the installation using "gem install ovirt-engine-sdk" wont work.
Problem is that the installation doesn't work using the embedded Ruby in vagrant using "vagrant plugin install ovirt-engine-sdk" which is needed

Comment 3 Juan Hernández 2017-01-11 09:09:36 UTC
Looking at the 'mkmf.log' file generated when the build runs with the Ruby embedded in Vagrant, I see the following (partial) gcc command:

  gcc ... -I/vagrant-substrate/staging/embedded/include/libxml2

This makes me think that the build of the SDK is using the embedded 'xml2-config' tool to find the required header files and libraries, which makes sense. But the directories that that generates seem incorrect:

  $ /opt/vagrant/embedded/bin/xml2-config --cflags
  -I/vagrant-substrate/staging/embedded/include/libxml2

That points to a '/vagrant-substrate' directory that doesn't exist, at least it isn't installed with the version of vagrant that I installed (CentOS 7, vagrant-1.9.1-1.x86_64).

The same happens with the 'curl-config' command. May this be an issue in Vagrant itself?

I solved it making a link to the directory that does exist:

  # mkdir -p /opt/vagrant/embedded
  # ln -s /opt/vagrant/embedded /vagrant-substrate/staging/embedded

Doing that the SDK is installed correctly:

  $ vagrant plugin install ovirt-engine-sdk
  Installing the 'ovirt-engine-sdk' plugin. This can take a few minutes...
  Building native extensions.  This could take a while...
  Installed the plugin 'ovirt-engine-sdk (4.0.6)'!

That could be a temporary solution, so you can continue the development of your plugin, but we better permanent solution. Any suggestion?

Comment 4 Juan Hernández 2017-01-11 09:14:04 UTC
I see the same wrong directory in all the .pc files included in vagrant:

  # grep 'prefix=' /opt/vagrant/embedded/lib/pkgconfig/*.pc

Maybe is an issue with this particular Vagrant installation, I mean, with the RPM for CentOS. Would be nice if you can check with other Vagrant installations.

Comment 5 Juan Hernández 2017-01-11 09:49:28 UTC
I created the following Vagrant issue to request help from their community:

  The .pc files inside RPM packages contain incorrect 'prefix', pointing to '/vagrant-substrate'
  https://github.com/mitchellh/vagrant/issues/8184

Comment 6 Juan Hernández 2017-01-11 09:59:51 UTC
The workaround I wrote in comment 3 was incorrectly described, it should be like this:

  # mkdir -p /vagrant-substrate/staging
  # ln -s /opt/vagrant/embedded /vagrant-substrate/staging/embedded

Comment 7 Juan Hernández 2017-01-11 12:22:06 UTC
I have submitted a patch that modifies the 'extconf.rb' file of the SDK so that it explicitly fixes the incorrect path returned by the 'xml2-config' tool embedded in Vagrant:

  Enable installation in Vagrant with embedded Ruby
  https://gerrit.ovirt.org/69992

I checked that with that change the SDK can be installed with "vagrant plugin install /path/to/the/fixed/gem". But I have no enough knowledge to check if that will actually work. Any chance that you can test that? The modified .gem file is available here:

  https://jhernand.fedorapeople.org/ovirt-engine-sdk-4.0.6.gem

If that works for you, then I will merge the change and do a new release of the gem.

Comment 8 marcus young 2017-01-11 13:38:40 UTC
I'll check and have an answer in the next few hours. Thanks for the quick feedback

Comment 9 marcus young 2017-01-11 14:35:21 UTC
It seems to install but not load. It doesn't look like it built the extensions.

  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant plugin install ovirt-engine-sdk-4.0.6.gem
Installing the 'ovirt-engine-sdk-4.0.6.gem' plugin. This can take a few minutes...
Building native extensions.  This could take a while...
Installed the plugin 'ovirt-engine-sdk (4.0.6)'!
  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant plugin install vagrant-ovirt4-0.0.1.gem
Installing the 'vagrant-ovirt4-0.0.1.gem' plugin. This can take a few minutes...
Installed the plugin 'vagrant-ovirt4 (0.0.1)'!
  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant plugin list
ovirt-engine-sdk (4.0.6)
  - Version Constraint: 4.0.6
vagrant-ovirt4 (0.0.1)
  - Version Constraint: 0.0.1
vagrant-share (1.1.6, system)

  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant up --provider=vagrant-ovirt4
Ignoring ovirt-engine-sdk-4.0.6 because its extensions are not built.  Try: gem pristine ovirt-engine-sdk --version 4.0.6
Vagrant failed to initialize at a very early stage:

The plugins failed to load properly. The error message given is
shown below.

cannot load such file -- ovirt-engine-sdk
  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  ack ovirt-engine-sdk
Gemfile.lock
28:      ovirt-engine-sdk (>= 4.0.5)
76:    ovirt-engine-sdk (4.0.6)

vagrant-ovirt4.gemspec
19:  gem.add_runtime_dependency 'ovirt-engine-sdk', '>=4.0.5'

Comment 10 Juan Hernández 2017-01-11 15:05:30 UTC
In my environment I see that when the SDK is installed as a plugin Vagrant is looking for an "ovirt-engine-sdk.rb" or "ovirt-engine-sdk.so" file, which doesn't exist, it is named "ovirtsdk4.so". Can you try to un-install the SDK, so that it doesnt appear in the "plugins.json" file?

  vagrant plugin uninstall ovirt-engine-sdk

Then try again to install your own plugin. This should pull the SDK, but as a normal gem, not as a plugion. Then, I guess that Vagrant won't look for that non-existing file, and may work.

Comment 11 Juan Hernández 2017-01-11 19:48:24 UTC
After some more testing I found that once installed, the SDK is loaded correctly when required by other plugins. But apparently the vagrant-share plugin tries to load it again, in a different way, and that causes the problem. I can't check why that happens, because the vagrant-share plugin isn't open source, so I am not able to debug what is happening inside. I removed it manually from /opt/vagrant/embedded/plugins.json, and then things seem to work.

Comment 12 marcus young 2017-01-11 19:59:32 UTC
Looks like it's working, however there's a warning (although it doesn't prevent it from seemign to work). 

The warning:  $ vagrant up
Ignoring ovirt-engine-sdk-4.0.6 because its extensions are not built.  Try: gem pristine ovirt-engine-sdk --version 4.0.6

The full output of me basically starting from scratch:

 dev.localdomain  myoung  ~  sudo yum uninstall -y vagrant >/dev/null 2>&1 && sudo rm -rf /opt/vagrant
 dev.localdomain  myoung  ~  sudo yum install -y https://releases.hashicorp.com/vagrant/1.9.1/vagrant_1.9.1_x86_64.rpm
Loaded plugins: fastestmirror
vagrant_1.9.1_x86_64.rpm                                                                               |  83 MB  00:00:24
Examining /var/tmp/yum-root-n_iPIE/vagrant_1.9.1_x86_64.rpm: 1:vagrant-1.9.1-1.x86_64
Marking /var/tmp/yum-root-n_iPIE/vagrant_1.9.1_x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package vagrant.x86_64 1:1.9.1-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================
 Package                   Arch                     Version                     Repository                               Size
==============================================================================================================================
Installing:
 vagrant                   x86_64                   1:1.9.1-1                   /vagrant_1.9.1_x86_64                   199 M

Transaction Summary
==============================================================================================================================
Install  1 Package

Total size: 199 M
Installed size: 199 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:vagrant-1.9.1-1.x86_64                                                                                   1/1
  Verifying  : 1:vagrant-1.9.1-1.x86_64                                                                                   1/1

Installed:
  vagrant.x86_64 1:1.9.1-1

Complete!

 dev.localdomain  myoung  ~  rm -rf ~/.vagrant.d/
 dev.localdomain  myoung  ~  for i in $(gem list | awk '{print $1}'); do gem uninstall -y $i >/dev/null 2>&1; done 

 dev.localdomain  myoung  ~  vagrant plugin list
vagrant-share (1.1.6, system)

 dev.localdomain  myoung  ~  cd repos/github/vagrant-ovirt4

 dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  rm -rf *.gem && gem build *.gemspec
WARNING:  open-ended dependency on ovirt-engine-sdk (>= 4.0.5) is not recommended
  if ovirt-engine-sdk is semantically versioned, use:
    add_runtime_dependency 'ovirt-engine-sdk', '~> 4.0', '>= 4.0.5'
WARNING:  open-ended dependency on rspec-its (>= 0, development) is not recommended
  if rspec-its is semantically versioned, use:
    add_development_dependency 'rspec-its', '~> 0'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: vagrant-ovirt4
  Version: 0.0.1
  File: vagrant-ovirt4-0.0.1.gem


 dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  wget https://jhernand.fedorapeople.org/ovirt-engine-sdk-4.0.6.gem
--2017-01-11 19:53:17--  https://jhernand.fedorapeople.org/ovirt-engine-sdk-4.0.6.gem
Resolving jhernand.fedorapeople.org (jhernand.fedorapeople.org)... 2610:28:3090:3001:5054:ff:fea7:9474, 152.19.134.199
Connecting to jhernand.fedorapeople.org (jhernand.fedorapeople.org)|2610:28:3090:3001:5054:ff:fea7:9474|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 243712 (238K) [application/x-troff-man]
Saving to: ‘ovirt-engine-sdk-4.0.6.gem’

100%[====================================================================================>] 243,712     1.28MB/s   in 0.2s

2017-01-11 19:53:18 (1.28 MB/s) - ‘ovirt-engine-sdk-4.0.6.gem’ saved [243712/243712]

  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  gem install ovirt-engine-sdk-4.0.6.gem
Building native extensions.  This could take a while...
Successfully installed ovirt-engine-sdk-4.0.6
Parsing documentation for ovirt-engine-sdk-4.0.6
Installing ri documentation for ovirt-engine-sdk-4.0.6
Done installing documentation for ovirt-engine-sdk after 28 seconds
1 gem installed
  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant plugin install vagrant-ovirt4-0.0.1.gem
Installing the 'vagrant-ovirt4-0.0.1.gem' plugin. This can take a few minutes...
Building native extensions.  This could take a while...
Installed the plugin 'vagrant-ovirt4 (0.0.1)'!
  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant up
Ignoring ovirt-engine-sdk-4.0.6 because its extensions are not built.  Try: gem pristine ovirt-engine-sdk --version 4.0.6
Vagrant.require_plugin is deprecated and has no effect any longer.
Use `vagrant plugin` commands to manage plugins. This warning will
be removed in the next version of Vagrant.
Bringing machine 'default' up with 'ovirt4' provider...
==> default: Creating VM with the following settings...
==> default:  -- Name:          jenkins
==> default:  -- Cluster:       Default
==> default:  -- Template:      Vagrant-Centos7-test
==> default:  -- Console Type:  vnc
==> default: An error occured. Recovering..
==> default: VM is not created. Please run `vagrant up` first.

Comment 13 marcus young 2017-01-11 20:11:56 UTC
I also tried to pristine all but warning still exists. Not sure if there's more we should try to do since it seems as though it's willing to work with the API

  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  gem pristine --all
Restoring gems to pristine condition...
Skipped bigdecimal-1.2.6, it is a default gem
Restored bundler-unload-1.0.2
Building native extensions.  This could take a while...
# In case of problems run the following command to update binstubs:
gem regenerate_binstubs
Restored executable-hooks-1.3.2
Building native extensions.  This could take a while...
Restored gem-wrappers-1.2.7
Skipped io-console-0.4.3, it is a default gem
Skipped json-1.8.1, it is a default gem
Restored minitest-5.4.3
Building native extensions.  This could take a while...
Restored ovirt-engine-sdk-4.0.6
Restored power_assert-0.2.2
Skipped psych-2.0.8, it is a default gem
Skipped rake-10.4.2, it is a default gem
Skipped rdoc-4.2.0, it is a default gem
Restored rubygems-bundler-1.4.4
Restored rvm-1.11.3.9
Restored test-unit-3.0.8

  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant up
Ignoring ovirt-engine-sdk-4.0.6 because its extensions are not built.  Try: gem pristine ovirt-engine-sdk --version 4.0.6
Vagrant.require_plugin is deprecated and has no effect any longer.
Use `vagrant plugin` commands to manage plugins. This warning will
be removed in the next version of Vagrant.
Bringing machine 'default' up with 'ovirt4' provider...
==> default: Creating VM with the following settings...
==> default:  -- Name:          test
==> default:  -- Cluster:       Default
==> default:  -- Template:      Vagrant-Centos7-test
==> default:  -- Console Type:  vnc
==> default: Waiting for VM to become "ready" to start...
==> default: Starting VM.
==> default: Waiting for VM to get an IP address...
==> default: Machine is booted and ready for use!

  dev.localdomain  myoung  ~  repos  github  vagrant-ovirt4  vagrant destroy -f
Ignoring ovirt-engine-sdk-4.0.6 because its extensions are not built.  Try: gem pristine ovirt-engine-sdk --version 4.0.6
Vagrant.require_plugin is deprecated and has no effect any longer.
Use `vagrant plugin` commands to manage plugins. This warning will
be removed in the next version of Vagrant.
==> default: Halting VM...
==> default: Waiting for VM to shutdown...
==> default: Removing VM...

Comment 14 Juan Hernández 2017-01-12 08:41:51 UTC
OK, as things seem to work I will merge the patch and close this bug. Please reopen if at some point you think that we should do something about the warning.

Comment 15 Juan Hernández 2017-01-12 08:57:52 UTC
The fix for this bug is included in release 4.0.7:

  https://rubygems.org/gems/ovirt-engine-sdk/versions/4.0.7

It will also be included in the upcoming oVirt 4.0.7 umbrella release, then the bug will be moved to ON_QA.

Comment 16 Sandro Bonazzola 2017-01-25 07:55:53 UTC
4.0.6 has been the last oVirt 4.0 release, please re-target this bug.

Comment 17 Sandro Bonazzola 2017-02-01 16:02:44 UTC
oVirt 4.1.0 GA has been released, re-targeting to 4.1.1.
Please check if this issue is correctly targeted or already included in 4.1.0.

Comment 18 Juan Hernández 2017-02-01 16:08:30 UTC
This was fixed in version 4.0.7 of the SDK, and is also included in version 4.1.0 of the SDK, which was included in release 4.1.0 of the oVirt project.


Note You need to log in before you can comment on or make changes to this bug.