Bug 688990

Apologies for the imprecise description, I am not smart enough to entirely understand this bug. :) But I'm doing some weeding for the blocker review meeting. 684742 was reported as an anaconda bug but seems to ultimately trace down to a bug in glibc's dynamic library loading. A workaround is being implemented for the resulting anaconda issue, but we did not want to lose track of the underlying glibc bug. Please see the following stuff from the original bug for more details. Ales can probably provide any extra info necessary.

+++ This bug was initially created as a clone of Bug #684742 +++

Description of problem:
  File "/usr/lib64/python2.7/site-packages/pyanaconda/kickstart.py", line 21, in <module>
    from storage.deviceaction import *
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/__init__.py", line 48, in <module>
    from devicelibs.crypto import generateBackupPassphrase
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/devicelibs/crypto.py", line 24, in <module>
    from pycryptsetup import CryptSetup
  File "/usr/lib64/python2.7/site-packages/pycryptsetup/__init__.py", line 22, in <module>
    import cryptsetup
ImportError: /lib64/libcryptsetup.so.1: undefined symbol: uuid_generate, version UUID_1.0

Version-Release number of selected component (if applicable):
anacoda-15.20.1

How reproducible:
always

Steps to Reproduce:
1. start installation with a kickstart
  
Actual results:
traceback

--- Additional comment from clumens on 2011-03-14 10:51:12 EDT ---

Does libuuid exist on your installation media?  Can you attach the complete traceback?

--- Additional comment from jhrozek on 2011-03-14 11:39:17 EDT ---

(In reply to comment #1)
> Does libuuid exist on your installation media? 

Not sure if I answer your question completely - there's no libuuid package, but it looks like the installation image contains just files. find(1) tells me this:

./lib64/libuuid.so.1.3.0
./lib64/libuuid.so.1
./proc/sys/kernel/random/uuid
./bin/dbus-uuidgen
./sys/devices/virtual/dmi/id/product_uuid
./usr/lib64/python2.7/uuid.py
./usr/lib64/python2.7/uuid.pyc


> Can you attach the complete
> traceback?

What I pasted /is/ the complete traceback, just reproduced with python -c "import cryptsetup" because I couldn't find it in /tmp/anaconda.log

I'm attaching a screenshot of the traceback so we're on the same page.

--- Additional comment from jhrozek on 2011-03-14 11:39:57 EDT ---

Created attachment 484229 [details]
installation screenshot

--- Additional comment from jlaska on 2011-03-14 11:56:39 EDT ---

(In reply to comment #3)
> Created attachment 484229 [details]
> installation screenshot

To help me reproduce your failure, can you provide some information on how you are hitting this issue?

1. What version of anaconda are you using to install: F-15-Alpha, nightly, custom?
2. How are you booting the installer: DVD, netinst.iso, PXEboot?
3. What boot parameters are you adding: any ks= or similar?
4. If using kickstart, can you provide the ks.cfg?
5. If doing manual, can you detail the manual selections you make that contribute to this failure?

--- Additional comment from clumens on 2011-03-14 12:01:11 EDT ---

Additionally, I'm wondering if there's media read errors going on here, since the code path isn't protected at all.  That is, I think just about everyone would be hitting this problem since the import's at the very top of kickstart.py and should always be happening.  Can you grab /tmp/syslog?

--- Additional comment from jhrozek on 2011-03-14 12:27:06 EDT ---

(In reply to comment #4)
> (In reply to comment #3)
> > Created attachment 484229 [details]
> > installation screenshot
> 
> To help me reproduce your failure, can you provide some information on how you
> are hitting this issue?
> 
> 1. What version of anaconda are you using to install: F-15-Alpha, nightly,
> custom?

Whatever is in http://download.fedora.redhat.com/pub/fedora/linux/development/15/x86_64/os/ (or rather our internal mirror of it)

> 2. How are you booting the installer: DVD, netinst.iso, PXEboot?

koan --virt, the tree is available via NFS. I pointed cobbler at vmlinuz and initrd from the mirror, not sure what magic does koan do after that.

> 3. What boot parameters are you adding: any ks= or similar?

yep, a custom kickstart

The full kernel command line looks like:
method=$url ksdevice=link lang= text ks=http://$cobbler_server/cblr/svc/op/ks/pr
ofile/Fedora-15-x86_64-minimal selinux=0 dhcpclass=ipa-lab-vms kssendmac 

> 4. If using kickstart, can you provide the ks.cfg?

OK

--- Additional comment from jhrozek on 2011-03-14 12:27:51 EDT ---

(In reply to comment #6)
> > 2. How are you booting the installer: DVD, netinst.iso, PXEboot?
> 
> koan --virt, the tree is available via NFS. I pointed cobbler at vmlinuz and
> initrd from the mirror, not sure what magic does koan do after that.

Sorry, not NFS, but rather HTTP.

--- Additional comment from jhrozek on 2011-03-14 12:28:35 EDT ---

Created attachment 484248 [details]
sanitized kickstart

--- Additional comment from jhrozek on 2011-03-14 12:29:29 EDT ---

(In reply to comment #5)
> Additionally, I'm wondering if there's media read errors going on here, since
> the code path isn't protected at all.  That is, I think just about everyone
> would be hitting this problem since the import's at the very top of
> kickstart.py and should always be happening.  Can you grab /tmp/syslog?

Since this is a network install, I doubt it, but sure, I'll attach.

Can it be that the mirror is in some kind of odd state?

--- Additional comment from jhrozek on 2011-03-14 12:30:57 EDT ---

Created attachment 484249 [details]
/tmp/syslog

--- Additional comment from clumens on 2011-03-14 13:03:27 EDT ---

*** Bug 684847 has been marked as a duplicate of this bug. ***

--- Additional comment from jlaska on 2011-03-14 13:27:39 EDT ---

Created attachment 484257 [details]
anaconda-logs.tgz

Reproduced with F-15-Alpha (anaconda-15.20.1-1) using the provided ks.cfg.  Attaching log files.

-rw-r--r-- root/root      5872 2011-03-14 13:24 tmp/anaconda.log
-rw-r--r-- root/root         0 2011-03-14 13:24 tmp/ifcfg.log
-rw-r--r-- root/root         0 2011-03-14 13:24 tmp/program.log
-rw-r--r-- root/root      2609 2011-03-14 13:24 tmp/storage.log
-rw-r--r-- root/root     50101 2011-03-14 13:24 tmp/syslog

I was *unable* to reproduce the bug using anaconda-15.22-1 along with the provided ks.cfg.

--- Additional comment from jhrozek on 2011-03-14 18:23:07 EDT ---

(In reply to comment #12)
> I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> provided ks.cfg.

How do I test a new anaconda version? I see that 15.22 was built in koji, do I need to regenerate any of the images?

Thank you for testing, James!

--- Additional comment from jlaska on 2011-03-15 08:14:23 EDT ---

(In reply to comment #13)
> (In reply to comment #12)
> > I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> > provided ks.cfg.
> 
> How do I test a new anaconda version? I see that 15.22 was built in koji, do I
> need to regenerate any of the images?

F-15 installation images are created daily at http://download.fedoraproject.org/pub/fedora/linux/development/15/x86_64/os

However, those images use only content that is in the base F-15 (stable) repo.  anaconda-15.22.1 is not yet in the stable repository.  Until then, I have created a custom boot.iso (and pxeboot images) that you can point virt-install (or koan) to.

http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/

If you are able to boot anaconda-15.22-1, can you please add positive karma feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

--- Additional comment from jhrozek on 2011-03-15 08:50:00 EDT ---

(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> > > provided ks.cfg.
> > 
> > How do I test a new anaconda version? I see that 15.22 was built in koji, do I
> > need to regenerate any of the images?
> 
> F-15 installation images are created daily at
> http://download.fedoraproject.org/pub/fedora/linux/development/15/x86_64/os
> 
> However, those images use only content that is in the base F-15 (stable) repo. 
> anaconda-15.22.1 is not yet in the stable repository.  Until then, I have
> created a custom boot.iso (and pxeboot images) that you can point virt-install
> (or koan) to.
> 
> http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/
> 
> If you are able to boot anaconda-15.22-1, can you please add positive karma
> feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

Works for me, too. Karma++. Thanks for providing the images!

--- Additional comment from clumens on 2011-03-15 16:03:32 EDT ---

Shorter reproducer, from python on tty2:

>>> from pyanaconda.storage import *

This avoids all the path weirdness we do in the anaconda script, so maybe it helps narrow down what's going on.

--- Additional comment from jhrozek on 2011-03-17 05:52:49 EDT ---

Reopening. 

I hit this again with 15.22 now that it got pushed into repos. Exactly the same traceback, reproducible with:

python -c "from pyanaconda.storage import *"

I'm not sure why I didn't hit it with James' custom initrd - maybe during the initrd build he rebuilt the bindings that seem to be failing?

--- Additional comment from twu on 2011-03-17 06:35:32 EDT ---

*** Bug 688524 has been marked as a duplicate of this bug. ***

--- Additional comment from jlaska on 2011-03-17 07:59:49 EDT ---

*** Bug 688514 has been marked as a duplicate of this bug. ***

--- Additional comment from kzak on 2011-03-17 08:19:54 EDT ---

Try:

  $ readelf -a /lib64/libuuid.so.1 | grep uuid_generate

you have to see the symbol in the output, something like:

  63: 0000003e50e02640    41 FUNC    GLOBAL DEFAULT   12 uuid_generate@@UUID_1.0

Do you really have libuuid from util-linux package?

There was libuuid in e2fsprogs package. The newly compiled binaries (with verioned ABI) cannot be executed against the old library.

--- Additional comment from akozumpl on 2011-03-17 10:38:57 EDT ---

A simpler reproducer:

import parted
try:
    import volume_key
except ImportError as e:
    print e
    print 'no volume key'
    volume_key = None

import pycryptsetup

Note the 'import parted' part is necessary. Why?

--- Additional comment from akozumpl on 2011-03-17 13:04:44 EDT ---

Patch adding missing library (presence of which makes the problem go away) was sent for a review: https://www.redhat.com/archives/anaconda-devel-list/2011-March/msg00222.html

Note we still don't know the root cause of the import failures.

--- Additional comment from akozumpl on 2011-03-17 13:39:02 EDT ---

This is not Anaconda specific, I am able to reproduce this on an installed system:
1) use the f15 alpha netinst to install the system, boot it
2) yum update
3) switch for the current f15 devel repo, e.g. http://download.fedora.redhat.com/pub/fedora/linux/development/15/x86_64/os/
4) yum install python-cryptsetup python-volume_key pyparted
5) hide away or remove /usr/lib64/libassuan.*
6) run the code in comment 21

the results are the volume_key complaining about a missing libassuan (that makes sense) but then right after cryptsetup complaining about missing uuid_generate (that makes no sense). 

a strange thing also is pycryptsetup can still be imported if the import of the other two packages is not attempted first. the order also matters. this is very weird and points to either

a) one of the packages playing nasty tricks with the import mechanism

b) more fundamental problem with dynamic libraries load mechanism.

David, have you ever seen anything like that? Who would you talk to (note I doubt it is specifically a python problem)?

--- Additional comment from akozumpl on 2011-03-17 13:40:13 EDT ---

(forgot in the reproducer description: one needs to run ldconfig as root after step 5).

--- Additional comment from clumens on 2011-03-17 13:41:38 EDT ---

*** Bug 688682 has been marked as a duplicate of this bug. ***

--- Additional comment from akozumpl on 2011-03-17 13:50:52 EDT ---

I can also provide access to the virtual machine where everything is set up.

--- Additional comment from akozumpl on 2011-03-18 05:18:25 EDT ---

I tried with an f14 kernel 2.6.35.6-45.fc14.x86_64 with no different result and then with an f14 glibc (glibc-2.12.90-17.x86_64) and the problem is gone (pycryptsetup can still be imported after a failed volume_key import).

Reassigning to glibc. Glibc mainteiner: I'll be happy to share the virtual machine where this is reproducible. Please do not let this sit, it seems like a serious, sneaky issue.

--- Additional comment from akozumpl on 2011-03-18 05:43:44 EDT ---

I doubt this is util-linux-ng problem.

In short: when we import few (unrelated?) python modules (which load dynamic libraries) and one of those modules legitimately fails to load do to a missing .so then pycryptsetup also fails to load (with: ImportError: /lib64/libcryptsetup.so.1: undefined symbol: uuid_generate. note the lib and the symbol ARE present). If the python interpreter is restarted then pycryptsetup can be loaded without a problem. 

And reverting glibc makes this go away.

--- Additional comment from gareth.glaccum on 2011-03-18 07:02:13 EDT ---

(In reply to comment #14)

> However, those images use only content that is in the base F-15 (stable) repo. 
> anaconda-15.22.1 is not yet in the stable repository.  Until then, I have
> created a custom boot.iso (and pxeboot images) that you can point virt-install
> (or koan) to.
> 
> http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/
> 
> If you are able to boot anaconda-15.22-1, can you please add positive karma
> feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

This fix in comment #14 has worked for me too. 
If it helps to narrow down why it is not being seen more, I am updating my local copy infrequently, this bug did not occur in the copy I made on March 10th, and I was experiencing the problem on March 16th (first reported here though 13th, so that means 3 days really).