Bug 1266918

$ ls -l /usr/lib64/firefox/browser/defaults
total 0
lrwxrwxrwx. 1 root root 39 Sep 28 09:54 preferences -> /usr/lib64/firefox/defaults/preferences
$ ls -l /usr/lib64/firefox/defaults
total 0
drwxr-xr-x. 2 root root 6 Jan  7  2015 preferences

I can't reproduce this on RHEL 7.2.

Please run:

  libguestfs-test-tool |& tee /tmp/log

and attach the /tmp/log file to this BZ.

Also, please run:

  /usr/bin/supermin5 --build --verbose --copy-kernel -f ext2 --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /var/tmp/appliance |& tee /tmp/log2

and attach /tmp/log2 to the BZ.

(In reply to Richard W.M. Jones from comment #3)

Sorry, second command was wrong, it should be:

/usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /var/tmp/appliance |& tee /tmp/log2

Asked for logs, awaiting for updates from customer. 

Regards,
Gajanan

Created attachment 1078746 [details]
libguestfs log file

Created attachment 1078747 [details]
supermin5 log file

I've attached the logs requested in comments #3 and #4.

Supermin builds a small appliance when libguestfs starts up the first
time.

Looking at the supermin log file (comment 7), you can see the package
list that supermin builds near the top of that file, and the package
list has 246 packages in it.  For comparison, on RHEL 7.2 the package
list normally contains 184 packages, although it's allowed to vary
slightly.

More seriously the package list contains firefox, which should really
never be on that list at all.

Supermin starts with the list of seed packages from
/usr/lib64/guestfs/supermin.d/packages, and (in this old version) runs
various RPM commands in order to build a complete list of packages
which includes all dependencies.  Commands that get run look like this
example:

$ rpm -qR coreutils | awk '{print $1}' | \
    xargs rpm -q --qf '%{name}\n' --whatprovides | \
    grep -v 'no package provides' | sort -u
bash
coreutils
glibc
gmp
grep
info
libacl
libattr
libcap
libselinux
ncurses
openssl-libs

Ref: https://github.com/libguestfs/supermin/blob/5873007f8cd40217fb38b56647cc38c48a77b90d/src/rpm.ml#L227-L250

The question is why are we pulling in firefox, which would not
normally happen on RHEL 7.

I suspect the answer has something to do with the strange Symantec
package installed on the machine: 'SYMCpddea-8.0.2.0-0.x86_64'.

I don't have access to this package, but looking around, it looks as
if this package RPM-Provides all sorts of core libraries, which of
course it should be doing.  See:
http://www.symantec.com/connect/it/forums/netbackup-7504-upgrade-pdde-plugin-dependency-error#comment-7756221

Is it possible we can get the customer to run the following commands
for us?

rpm -q --provides SYMCpddea
rpm -q --requires SYMCpddea

> but looking around, it looks as
> if this package RPM-Provides all sorts of core libraries, which of
> course it should be doing.
                 ^not^
> Is it possible we can get the customer to run the following commands
> for us?
> 
> rpm -q --provides SYMCpddea
> rpm -q --requires SYMCpddea

Created attachment 1081050 [details]
SYMCpddea.requires

Created attachment 1081051 [details]
SYMCpddea.provides

The SYMCpddea is providing a lot of core packages, eg:

libxml2.so.2()(64bit)
libz.so.1()(64bit)
libcurl.so.4()(64bit)
libssh2.so.1()(64bit)
libssl.so.0.9.8()(64bit)
etc.

That explains why SYMCpddea gets pulled into the appliance, although
not why firefox and the other browser gets pulled in.

So I still don't understand why this bug happens.

Please thank the customer for persisting and providing this extra
detail.  Using the information provided by the customer I was able
to diagnose and reproduce the bug.

The problem is the 'palemoon-25.7.0-1.el7.nux.x86_64' package --
a web browser.  This RPM package 'Provides: libnss3.so()(64bit)'
(and a few other core libraries, but this provides alone is sufficient).
This would normally be provided only by 'nss'.  It is required by
the appliance packages such as openldap, and hence palemoon is
pulled into the appliance by supermin's dependency resolution.

In the next step, palemoon 'Requires: libxul.so()(64bit)', and
since firefox provides this dependency, it too is pulled in to
the appliance.  Since our firefox package is buggy (bug 1269973),
that causes the ultimate supermin failure.  However we should never
have got so far as pulling in palemoon nor firefox in the first
place.

The workaround is to uninstall 'palemoon'.  I am going to add fixes
to supermin upstream and in RHEL 7.3 so that this problem can be
avoided completely.

Created attachment 1082150 [details]
palemoon.spec

This is a reproducer.  The attachment is an empty 'palemoon.spec'
(it's not the web browser).  It just has the same 'Provides' and
'Requires' as the real package.

If you build and install this fake package on a RHEL 7.2 box, then
run 'libguestfs-test-tool', it will fail in the same way as the
customer described:

...
supermin: *** parent directory not found ***
supermin: When reporting this error:
supermin: please include ALL the debugging information below
supermin: AND tell us what system you are running this on.
     src=/usr/lib64/firefox/browser/defaults/preferences
    dest=/usr/lib64/firefox/browser/defaults/preferences
 dirname=/usr/lib64/firefox/browser/defaults
basename=preferences
supermin: ext2fs_namei: parent directory not found: /usr/lib64/firefox/browser/defaults: File not found by ext2_lookup
supermin: failure: ext2fs_namei: parent directory not found
libguestfs: error: /usr/bin/supermin5 exited with error status 1, see debug messages above

(In reply to Richard W.M. Jones from comment #22)
> Created attachment 1082150 [details]
> palemoon.spec
> 
> This is a reproducer.  The attachment is an empty 'palemoon.spec'
> (it's not the web browser).  It just has the same 'Provides' and
> 'Requires' as the real package.
> 
> If you build and install this fake package on a RHEL 7.2 box, then
> run 'libguestfs-test-tool',

I should add that you also need 'firefox' installed to reproduce
the bug.

With fake palemoon + firefox installed, you will see the bug.

If you uninstall palemoon (but leave firefox installed) the bug
will go away and libguestfs will work normally again.

Upstream patch posted:

https://www.redhat.com/archives/libguestfs/2015-October/msg00136.html

However this requires us to rebase to a later supermin,
since we switched from running rpm commands to using librpm
upstream (much faster too).

Upstream commit:
https://github.com/libguestfs/supermin/commit/17b724b5d345f5e9d622bcadf27c57baa102bbba

It still fail with with packages:
libguestfs-1.32.5-6.el7.x86_64
supermin5-5.1.16-2.el7.x86_64

Steps:
1. Assure that you have firefox, then install palemoon or just install a fake one using the attachment provided:
# wget -O palemoon.spec https://bugzilla.redhat.com/attachment.cgi?id=1082150
# rpmbuild -bb palemoon.spec
# rpm -ivh /root/rpmbuild/RPMS/x86_64/palemoon-25.7.0-1.el7.nux.x86_64.rpm 
2. 
# libguestfs-test-tools
... ...
supermin: *** parent directory not found ***
supermin: When reporting this error:
supermin: please include ALL the debugging information below
supermin: AND tell us what system you are running this on.
     src=/usr/lib64/firefox/browser/defaults/preferences
    dest=/usr/lib64/firefox/browser/defaults/preferences
 dirname=/usr/lib64/firefox/browser/defaults
basename=preferences
supermin: ext2fs_namei: parent directory not found: /usr/lib64/firefox/browser/defaults: File not found by ext2_lookup
supermin: failure: ext2fs_namei: parent directory not found
libguestfs: error: /usr/bin/supermin5 exited with error status 1, see debug messages above
libguestfs-test-tool: failed to launch appliance

Yes, I can reproduce the error too.  Doh.

I enabled debugging and found the problem is our "shortest provider"
algorithm doesn't work for the nss-softokn packages, eg:

supermin: rpm: multiple providers: requirement libfreebl3.so()(64bit): providers
: nss-softokn-freebl palemoon
supermin: rpm: multiple providers: picked palemoon

supermin: rpm: multiple providers: requirement libnssdbm3.so()(64bit): providers
: nss-softokn palemoon
supermin: rpm: multiple providers: picked palemoon

Notice that len(palemoon) < len(nss-softokn*).

This is not very simple to fix.  The good news is that the nux repository
has deleted the 'palemoon' package entirely, so that means hopefully
new customers will not hit this particular bug in future.

Since the customer closed the bug and we have a workaround
(either uninstall palemoon or uninstall firefox) I am going to
close the bug as WONTFIX and drop it from the erratum.