Bug 1266918 - If 'palemoon' package is installed, libguestfs fails to build appliance with error: /usr/sbin/supermin5 exited with error
If 'palemoon' package is installed, libguestfs fails to build appliance with ...
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: supermin (Show other bugs)
7.2
Unspecified Linux
high Severity high
: rc
: ---
Assigned To: Richard W.M. Jones
Virtualization Bugs
:
Depends On: 1271255
Blocks: 1288337 1301891
  Show dependency treegraph
 
Reported: 2015-09-28 09:51 EDT by Gajanan
Modified: 2016-07-01 09:52 EDT (History)
9 users (show)

See Also:
Fixed In Version: supermin-5.1.14-2.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-01 09:52:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libguestfs log file (5.44 KB, text/plain)
2015-09-30 12:57 EDT, Jacob Hunt
no flags Details
supermin5 log file (1.45 MB, text/plain)
2015-09-30 12:58 EDT, Jacob Hunt
no flags Details
SYMCpddea.requires (111 bytes, application/x-shellscript)
2015-10-08 11:45 EDT, Jacob Hunt
no flags Details
SYMCpddea.provides (2.91 KB, text/plain)
2015-10-08 11:45 EDT, Jacob Hunt
no flags Details
palemoon.spec (8.63 KB, text/plain)
2015-10-12 17:58 EDT, Richard W.M. Jones
no flags Details

  None (edit)
Description Gajanan 2015-09-28 09:51:51 EDT
Description of problem:

Customer reported this issue on RHEL 7.2 Beta 

On certain systems virt-sysprep fails while not on others when run on a new qcow2 image with the same backing file. 

The error returned is:

~~~
 virt-sysprep: error: libguestfs error: /usr/bin/supermin5 exited with error 
status 1, see debug messages above
~~~

When running virt-sysprep with the verbose flag I see:

~~~
supermin: kernel: picked kernel vmlinuz-3.10.0-316.el7.x86_64
supermin: kernel: picked modules path /lib/modules/3.10.0-316.el7.x86_64
supermin: kernel: kernel_version 3.10.0-316.el7.x86_64
supermin: kernel: modules /lib/modules/3.10.0-316.el7.x86_64
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.rk20t5r5/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: *** parent directory not found ***
supermin: When reporting this error:
supermin: please include ALL the debugging information below
supermin: AND tell us what system you are running this on.
     src=/usr/lib64/firefox/browser/defaults/preferences
    dest=/usr/lib64/firefox/browser/defaults/preferences
 dirname=/usr/lib64/firefox/browser/defaults
basename=preferences
supermin: ext2fs_namei: parent directory not found: /usr/lib64/firefox/browser/defaults: File not found by ext2_lookup
supermin: failure: ext2fs_namei: parent directory not found
libguestfs: clear_socket_create_context: setsockcreatecon failed: NULL: Invalid argument [you can ignore this UNLESS using SELinux + sVirt]
~~~

Here are the commands I'm running:

~~~
/usr/bin/qemu-img create -f qcow2 -b ${BASEIMGPATH}/$1.img /disk1/images/${KVMNAME}.img
/usr/bin/virt-sysprep -v --firstboot /share/lair/kvm/kvmfirstboot.sh --hostname ${KVMNAME} --operations defaults,-ssh-userdir -a /disk1/images/${KVMNAME}.img
~~~

  
This happens on a RHEL 7.2 snapshot 2 workstation.

It happens every time I run the command.
Comment 3 Richard W.M. Jones 2015-09-28 10:02:15 EDT
$ ls -l /usr/lib64/firefox/browser/defaults
total 0
lrwxrwxrwx. 1 root root 39 Sep 28 09:54 preferences -> /usr/lib64/firefox/defaults/preferences
$ ls -l /usr/lib64/firefox/defaults
total 0
drwxr-xr-x. 2 root root 6 Jan  7  2015 preferences

I can't reproduce this on RHEL 7.2.

Please run:

  libguestfs-test-tool |& tee /tmp/log

and attach the /tmp/log file to this BZ.

Also, please run:

  /usr/bin/supermin5 --build --verbose --copy-kernel -f ext2 --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /var/tmp/appliance |& tee /tmp/log2

and attach /tmp/log2 to the BZ.
Comment 4 Richard W.M. Jones 2015-09-28 10:03:17 EDT
(In reply to Richard W.M. Jones from comment #3)

Sorry, second command was wrong, it should be:

/usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /var/tmp/appliance |& tee /tmp/log2
Comment 5 Gajanan 2015-09-29 06:46:42 EDT
Asked for logs, awaiting for updates from customer. 

Regards,
Gajanan
Comment 6 Jacob Hunt 2015-09-30 12:57 EDT
Created attachment 1078746 [details]
libguestfs log file
Comment 7 Jacob Hunt 2015-09-30 12:58 EDT
Created attachment 1078747 [details]
supermin5 log file
Comment 8 Jacob Hunt 2015-09-30 12:59:58 EDT
I've attached the logs requested in comments #3 and #4.
Comment 9 Richard W.M. Jones 2015-09-30 15:23:37 EDT
Supermin builds a small appliance when libguestfs starts up the first
time.

Looking at the supermin log file (comment 7), you can see the package
list that supermin builds near the top of that file, and the package
list has 246 packages in it.  For comparison, on RHEL 7.2 the package
list normally contains 184 packages, although it's allowed to vary
slightly.

More seriously the package list contains firefox, which should really
never be on that list at all.

Supermin starts with the list of seed packages from
/usr/lib64/guestfs/supermin.d/packages, and (in this old version) runs
various RPM commands in order to build a complete list of packages
which includes all dependencies.  Commands that get run look like this
example:

$ rpm -qR coreutils | awk '{print $1}' | \
    xargs rpm -q --qf '%{name}\n' --whatprovides | \
    grep -v 'no package provides' | sort -u
bash
coreutils
glibc
gmp
grep
info
libacl
libattr
libcap
libselinux
ncurses
openssl-libs

Ref: https://github.com/libguestfs/supermin/blob/5873007f8cd40217fb38b56647cc38c48a77b90d/src/rpm.ml#L227-L250

The question is why are we pulling in firefox, which would not
normally happen on RHEL 7.

I suspect the answer has something to do with the strange Symantec
package installed on the machine: 'SYMCpddea-8.0.2.0-0.x86_64'.

I don't have access to this package, but looking around, it looks as
if this package RPM-Provides all sorts of core libraries, which of
course it should be doing.  See:
http://www.symantec.com/connect/it/forums/netbackup-7504-upgrade-pdde-plugin-dependency-error#comment-7756221

Is it possible we can get the customer to run the following commands
for us?

rpm -q --provides SYMCpddea
rpm -q --requires SYMCpddea
Comment 10 Richard W.M. Jones 2015-09-30 15:25:31 EDT
> but looking around, it looks as
> if this package RPM-Provides all sorts of core libraries, which of
> course it should be doing.
                 ^not^
> Is it possible we can get the customer to run the following commands
> for us?
> 
> rpm -q --provides SYMCpddea
> rpm -q --requires SYMCpddea
Comment 11 Jacob Hunt 2015-10-08 11:45 EDT
Created attachment 1081050 [details]
SYMCpddea.requires
Comment 12 Jacob Hunt 2015-10-08 11:45 EDT
Created attachment 1081051 [details]
SYMCpddea.provides
Comment 14 Richard W.M. Jones 2015-10-08 12:32:01 EDT
The SYMCpddea is providing a lot of core packages, eg:

libxml2.so.2()(64bit)
libz.so.1()(64bit)
libcurl.so.4()(64bit)
libssh2.so.1()(64bit)
libssl.so.0.9.8()(64bit)
etc.

That explains why SYMCpddea gets pulled into the appliance, although
not why firefox and the other browser gets pulled in.

So I still don't understand why this bug happens.
Comment 21 Richard W.M. Jones 2015-10-12 17:56:16 EDT
Please thank the customer for persisting and providing this extra
detail.  Using the information provided by the customer I was able
to diagnose and reproduce the bug.

The problem is the 'palemoon-25.7.0-1.el7.nux.x86_64' package --
a web browser.  This RPM package 'Provides: libnss3.so()(64bit)'
(and a few other core libraries, but this provides alone is sufficient).
This would normally be provided only by 'nss'.  It is required by
the appliance packages such as openldap, and hence palemoon is
pulled into the appliance by supermin's dependency resolution.

In the next step, palemoon 'Requires: libxul.so()(64bit)', and
since firefox provides this dependency, it too is pulled in to
the appliance.  Since our firefox package is buggy (bug 1269973),
that causes the ultimate supermin failure.  However we should never
have got so far as pulling in palemoon nor firefox in the first
place.

The workaround is to uninstall 'palemoon'.  I am going to add fixes
to supermin upstream and in RHEL 7.3 so that this problem can be
avoided completely.
Comment 22 Richard W.M. Jones 2015-10-12 17:58 EDT
Created attachment 1082150 [details]
palemoon.spec

This is a reproducer.  The attachment is an empty 'palemoon.spec'
(it's not the web browser).  It just has the same 'Provides' and
'Requires' as the real package.

If you build and install this fake package on a RHEL 7.2 box, then
run 'libguestfs-test-tool', it will fail in the same way as the
customer described:

...
supermin: *** parent directory not found ***
supermin: When reporting this error:
supermin: please include ALL the debugging information below
supermin: AND tell us what system you are running this on.
     src=/usr/lib64/firefox/browser/defaults/preferences
    dest=/usr/lib64/firefox/browser/defaults/preferences
 dirname=/usr/lib64/firefox/browser/defaults
basename=preferences
supermin: ext2fs_namei: parent directory not found: /usr/lib64/firefox/browser/defaults: File not found by ext2_lookup
supermin: failure: ext2fs_namei: parent directory not found
libguestfs: error: /usr/bin/supermin5 exited with error status 1, see debug messages above
Comment 23 Richard W.M. Jones 2015-10-12 18:02:14 EDT
(In reply to Richard W.M. Jones from comment #22)
> Created attachment 1082150 [details]
> palemoon.spec
> 
> This is a reproducer.  The attachment is an empty 'palemoon.spec'
> (it's not the web browser).  It just has the same 'Provides' and
> 'Requires' as the real package.
> 
> If you build and install this fake package on a RHEL 7.2 box, then
> run 'libguestfs-test-tool',

I should add that you also need 'firefox' installed to reproduce
the bug.

With fake palemoon + firefox installed, you will see the bug.

If you uninstall palemoon (but leave firefox installed) the bug
will go away and libguestfs will work normally again.
Comment 24 Richard W.M. Jones 2015-10-13 09:38:06 EDT
Upstream patch posted:

https://www.redhat.com/archives/libguestfs/2015-October/msg00136.html

However this requires us to rebase to a later supermin,
since we switched from running rpm commands to using librpm
upstream (much faster too).
Comment 29 Xianghua Chen 2016-06-28 01:38:13 EDT
It still fail with with packages:
libguestfs-1.32.5-6.el7.x86_64
supermin5-5.1.16-2.el7.x86_64

Steps:
1. Assure that you have firefox, then install palemoon or just install a fake one using the attachment provided:
# wget -O palemoon.spec https://bugzilla.redhat.com/attachment.cgi?id=1082150
# rpmbuild -bb palemoon.spec
# rpm -ivh /root/rpmbuild/RPMS/x86_64/palemoon-25.7.0-1.el7.nux.x86_64.rpm 
2. 
# libguestfs-test-tools
... ...
supermin: *** parent directory not found ***
supermin: When reporting this error:
supermin: please include ALL the debugging information below
supermin: AND tell us what system you are running this on.
     src=/usr/lib64/firefox/browser/defaults/preferences
    dest=/usr/lib64/firefox/browser/defaults/preferences
 dirname=/usr/lib64/firefox/browser/defaults
basename=preferences
supermin: ext2fs_namei: parent directory not found: /usr/lib64/firefox/browser/defaults: File not found by ext2_lookup
supermin: failure: ext2fs_namei: parent directory not found
libguestfs: error: /usr/bin/supermin5 exited with error status 1, see debug messages above
libguestfs-test-tool: failed to launch appliance
Comment 30 Richard W.M. Jones 2016-06-28 03:55:16 EDT
Yes, I can reproduce the error too.  Doh.
Comment 31 Richard W.M. Jones 2016-07-01 09:46:00 EDT
I enabled debugging and found the problem is our "shortest provider"
algorithm doesn't work for the nss-softokn packages, eg:

supermin: rpm: multiple providers: requirement libfreebl3.so()(64bit): providers
: nss-softokn-freebl palemoon
supermin: rpm: multiple providers: picked palemoon

supermin: rpm: multiple providers: requirement libnssdbm3.so()(64bit): providers
: nss-softokn palemoon
supermin: rpm: multiple providers: picked palemoon

Notice that len(palemoon) < len(nss-softokn*).
Comment 32 Richard W.M. Jones 2016-07-01 09:52:04 EDT
This is not very simple to fix.  The good news is that the nux repository
has deleted the 'palemoon' package entirely, so that means hopefully
new customers will not hit this particular bug in future.

Since the customer closed the bug and we have a workaround
(either uninstall palemoon or uninstall firefox) I am going to
close the bug as WONTFIX and drop it from the erratum.

Note You need to log in before you can comment on or make changes to this bug.