Bug 684742 - glibc dynamic library loading bug causes installs to fail
glibc dynamic library loading bug causes installs to fail
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: lorax (Show other bugs)
15
Unspecified Unspecified
unspecified Severity urgent
: ---
: ---
Assigned To: Ales Kozumplik
Fedora Extras Quality Assurance
AcceptedBlocker
: Reopened
: 684847 688514 688524 688682 689159 690180 690742 (view as bug list)
Depends On:
Blocks: F15Beta/F15BetaBlocker
  Show dependency treegraph
 
Reported: 2011-03-14 08:08 EDT by Jakub Hrozek
Modified: 2014-09-30 19:39 EDT (History)
28 users (show)

See Also:
Fixed In Version: lorax-0.3.2-1.fc15
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 688990 (view as bug list)
Environment:
Last Closed: 2011-03-23 00:58:52 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
installation screenshot (30.98 KB, image/png)
2011-03-14 11:39 EDT, Jakub Hrozek
no flags Details
sanitized kickstart (2.41 KB, text/plain)
2011-03-14 12:28 EDT, Jakub Hrozek
no flags Details
/tmp/syslog (10.41 KB, application/x-bzip2)
2011-03-14 12:30 EDT, Jakub Hrozek
no flags Details
anaconda-logs.tgz (13.25 KB, application/x-gzip)
2011-03-14 13:27 EDT, James Laska
no flags Details

  None (edit)
Description Jakub Hrozek 2011-03-14 08:08:03 EDT
Description of problem:
  File "/usr/lib64/python2.7/site-packages/pyanaconda/kickstart.py", line 21, in <module>
    from storage.deviceaction import *
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/__init__.py", line 48, in <module>
    from devicelibs.crypto import generateBackupPassphrase
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/devicelibs/crypto.py", line 24, in <module>
    from pycryptsetup import CryptSetup
  File "/usr/lib64/python2.7/site-packages/pycryptsetup/__init__.py", line 22, in <module>
    import cryptsetup
ImportError: /lib64/libcryptsetup.so.1: undefined symbol: uuid_generate, version UUID_1.0

Version-Release number of selected component (if applicable):
anacoda-15.20.1

How reproducible:
always

Steps to Reproduce:
1. start installation with a kickstart
  
Actual results:
traceback
Comment 1 Chris Lumens 2011-03-14 10:51:12 EDT
Does libuuid exist on your installation media?  Can you attach the complete traceback?
Comment 2 Jakub Hrozek 2011-03-14 11:39:17 EDT
(In reply to comment #1)
> Does libuuid exist on your installation media? 

Not sure if I answer your question completely - there's no libuuid package, but it looks like the installation image contains just files. find(1) tells me this:

./lib64/libuuid.so.1.3.0
./lib64/libuuid.so.1
./proc/sys/kernel/random/uuid
./bin/dbus-uuidgen
./sys/devices/virtual/dmi/id/product_uuid
./usr/lib64/python2.7/uuid.py
./usr/lib64/python2.7/uuid.pyc


> Can you attach the complete
> traceback?

What I pasted /is/ the complete traceback, just reproduced with python -c "import cryptsetup" because I couldn't find it in /tmp/anaconda.log

I'm attaching a screenshot of the traceback so we're on the same page.
Comment 3 Jakub Hrozek 2011-03-14 11:39:57 EDT
Created attachment 484229 [details]
installation screenshot
Comment 4 James Laska 2011-03-14 11:56:39 EDT
(In reply to comment #3)
> Created attachment 484229 [details]
> installation screenshot

To help me reproduce your failure, can you provide some information on how you are hitting this issue?

1. What version of anaconda are you using to install: F-15-Alpha, nightly, custom?
2. How are you booting the installer: DVD, netinst.iso, PXEboot?
3. What boot parameters are you adding: any ks= or similar?
4. If using kickstart, can you provide the ks.cfg?
5. If doing manual, can you detail the manual selections you make that contribute to this failure?
Comment 5 Chris Lumens 2011-03-14 12:01:11 EDT
Additionally, I'm wondering if there's media read errors going on here, since the code path isn't protected at all.  That is, I think just about everyone would be hitting this problem since the import's at the very top of kickstart.py and should always be happening.  Can you grab /tmp/syslog?
Comment 6 Jakub Hrozek 2011-03-14 12:27:06 EDT
(In reply to comment #4)
> (In reply to comment #3)
> > Created attachment 484229 [details]
> > installation screenshot
> 
> To help me reproduce your failure, can you provide some information on how you
> are hitting this issue?
> 
> 1. What version of anaconda are you using to install: F-15-Alpha, nightly,
> custom?

Whatever is in http://download.fedora.redhat.com/pub/fedora/linux/development/15/x86_64/os/ (or rather our internal mirror of it)

> 2. How are you booting the installer: DVD, netinst.iso, PXEboot?

koan --virt, the tree is available via NFS. I pointed cobbler at vmlinuz and initrd from the mirror, not sure what magic does koan do after that.

> 3. What boot parameters are you adding: any ks= or similar?

yep, a custom kickstart

The full kernel command line looks like:
method=$url ksdevice=link lang= text ks=http://$cobbler_server/cblr/svc/op/ks/pr
ofile/Fedora-15-x86_64-minimal selinux=0 dhcpclass=ipa-lab-vms kssendmac 

> 4. If using kickstart, can you provide the ks.cfg?

OK
Comment 7 Jakub Hrozek 2011-03-14 12:27:51 EDT
(In reply to comment #6)
> > 2. How are you booting the installer: DVD, netinst.iso, PXEboot?
> 
> koan --virt, the tree is available via NFS. I pointed cobbler at vmlinuz and
> initrd from the mirror, not sure what magic does koan do after that.

Sorry, not NFS, but rather HTTP.
Comment 8 Jakub Hrozek 2011-03-14 12:28:35 EDT
Created attachment 484248 [details]
sanitized kickstart
Comment 9 Jakub Hrozek 2011-03-14 12:29:29 EDT
(In reply to comment #5)
> Additionally, I'm wondering if there's media read errors going on here, since
> the code path isn't protected at all.  That is, I think just about everyone
> would be hitting this problem since the import's at the very top of
> kickstart.py and should always be happening.  Can you grab /tmp/syslog?

Since this is a network install, I doubt it, but sure, I'll attach.

Can it be that the mirror is in some kind of odd state?
Comment 10 Jakub Hrozek 2011-03-14 12:30:57 EDT
Created attachment 484249 [details]
/tmp/syslog
Comment 11 Chris Lumens 2011-03-14 13:03:27 EDT
*** Bug 684847 has been marked as a duplicate of this bug. ***
Comment 12 James Laska 2011-03-14 13:27:39 EDT
Created attachment 484257 [details]
anaconda-logs.tgz

Reproduced with F-15-Alpha (anaconda-15.20.1-1) using the provided ks.cfg.  Attaching log files.

-rw-r--r-- root/root      5872 2011-03-14 13:24 tmp/anaconda.log
-rw-r--r-- root/root         0 2011-03-14 13:24 tmp/ifcfg.log
-rw-r--r-- root/root         0 2011-03-14 13:24 tmp/program.log
-rw-r--r-- root/root      2609 2011-03-14 13:24 tmp/storage.log
-rw-r--r-- root/root     50101 2011-03-14 13:24 tmp/syslog

I was *unable* to reproduce the bug using anaconda-15.22-1 along with the provided ks.cfg.
Comment 13 Jakub Hrozek 2011-03-14 18:23:07 EDT
(In reply to comment #12)
> I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> provided ks.cfg.

How do I test a new anaconda version? I see that 15.22 was built in koji, do I need to regenerate any of the images?

Thank you for testing, James!
Comment 14 James Laska 2011-03-15 08:14:23 EDT
(In reply to comment #13)
> (In reply to comment #12)
> > I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> > provided ks.cfg.
> 
> How do I test a new anaconda version? I see that 15.22 was built in koji, do I
> need to regenerate any of the images?

F-15 installation images are created daily at http://download.fedoraproject.org/pub/fedora/linux/development/15/x86_64/os

However, those images use only content that is in the base F-15 (stable) repo.  anaconda-15.22.1 is not yet in the stable repository.  Until then, I have created a custom boot.iso (and pxeboot images) that you can point virt-install (or koan) to.

http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/

If you are able to boot anaconda-15.22-1, can you please add positive karma feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15
Comment 15 Jakub Hrozek 2011-03-15 08:50:00 EDT
(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> > > provided ks.cfg.
> > 
> > How do I test a new anaconda version? I see that 15.22 was built in koji, do I
> > need to regenerate any of the images?
> 
> F-15 installation images are created daily at
> http://download.fedoraproject.org/pub/fedora/linux/development/15/x86_64/os
> 
> However, those images use only content that is in the base F-15 (stable) repo. 
> anaconda-15.22.1 is not yet in the stable repository.  Until then, I have
> created a custom boot.iso (and pxeboot images) that you can point virt-install
> (or koan) to.
> 
> http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/
> 
> If you are able to boot anaconda-15.22-1, can you please add positive karma
> feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

Works for me, too. Karma++. Thanks for providing the images!
Comment 16 Chris Lumens 2011-03-15 16:03:32 EDT
Shorter reproducer, from python on tty2:

>>> from pyanaconda.storage import *

This avoids all the path weirdness we do in the anaconda script, so maybe it helps narrow down what's going on.
Comment 17 Jakub Hrozek 2011-03-17 05:52:49 EDT
Reopening. 

I hit this again with 15.22 now that it got pushed into repos. Exactly the same traceback, reproducible with:

python -c "from pyanaconda.storage import *"

I'm not sure why I didn't hit it with James' custom initrd - maybe during the initrd build he rebuilt the bindings that seem to be failing?
Comment 18 Tao Wu 2011-03-17 06:35:32 EDT
*** Bug 688524 has been marked as a duplicate of this bug. ***
Comment 19 James Laska 2011-03-17 07:59:49 EDT
*** Bug 688514 has been marked as a duplicate of this bug. ***
Comment 20 Karel Zak 2011-03-17 08:19:54 EDT
Try:

  $ readelf -a /lib64/libuuid.so.1 | grep uuid_generate

you have to see the symbol in the output, something like:

  63: 0000003e50e02640    41 FUNC    GLOBAL DEFAULT   12 uuid_generate@@UUID_1.0

Do you really have libuuid from util-linux package?

There was libuuid in e2fsprogs package. The newly compiled binaries (with verioned ABI) cannot be executed against the old library.
Comment 21 Ales Kozumplik 2011-03-17 10:38:57 EDT
A simpler reproducer:

import parted
try:
    import volume_key
except ImportError as e:
    print e
    print 'no volume key'
    volume_key = None

import pycryptsetup

Note the 'import parted' part is necessary. Why?
Comment 22 Ales Kozumplik 2011-03-17 13:04:44 EDT
Patch adding missing library (presence of which makes the problem go away) was sent for a review: https://www.redhat.com/archives/anaconda-devel-list/2011-March/msg00222.html

Note we still don't know the root cause of the import failures.
Comment 23 Ales Kozumplik 2011-03-17 13:39:02 EDT
This is not Anaconda specific, I am able to reproduce this on an installed system:
1) use the f15 alpha netinst to install the system, boot it
2) yum update
3) switch for the current f15 devel repo, e.g. http://download.fedora.redhat.com/pub/fedora/linux/development/15/x86_64/os/
4) yum install python-cryptsetup python-volume_key pyparted
5) hide away or remove /usr/lib64/libassuan.*
6) run the code in comment 21

the results are the volume_key complaining about a missing libassuan (that makes sense) but then right after cryptsetup complaining about missing uuid_generate (that makes no sense). 

a strange thing also is pycryptsetup can still be imported if the import of the other two packages is not attempted first. the order also matters. this is very weird and points to either

a) one of the packages playing nasty tricks with the import mechanism

b) more fundamental problem with dynamic libraries load mechanism.

David, have you ever seen anything like that? Who would you talk to (note I doubt it is specifically a python problem)?
Comment 24 Ales Kozumplik 2011-03-17 13:40:13 EDT
(forgot in the reproducer description: one needs to run ldconfig as root after step 5).
Comment 25 Chris Lumens 2011-03-17 13:41:38 EDT
*** Bug 688682 has been marked as a duplicate of this bug. ***
Comment 26 Ales Kozumplik 2011-03-17 13:50:52 EDT
I can also provide access to the virtual machine where everything is set up.
Comment 27 Ales Kozumplik 2011-03-18 05:18:25 EDT
I tried with an f14 kernel 2.6.35.6-45.fc14.x86_64 with no different result and then with an f14 glibc (glibc-2.12.90-17.x86_64) and the problem is gone (pycryptsetup can still be imported after a failed volume_key import).

Reassigning to glibc. Glibc mainteiner: I'll be happy to share the virtual machine where this is reproducible. Please do not let this sit, it seems like a serious, sneaky issue.
Comment 28 Ales Kozumplik 2011-03-18 05:43:44 EDT
I doubt this is util-linux-ng problem.

In short: when we import few (unrelated?) python modules (which load dynamic libraries) and one of those modules legitimately fails to load do to a missing .so then pycryptsetup also fails to load (with: ImportError: /lib64/libcryptsetup.so.1: undefined symbol: uuid_generate. note the lib and the symbol ARE present). If the python interpreter is restarted then pycryptsetup can be loaded without a problem. 

And reverting glibc makes this go away.
Comment 29 Gareth 2011-03-18 07:02:13 EDT
(In reply to comment #14)

> However, those images use only content that is in the base F-15 (stable) repo. 
> anaconda-15.22.1 is not yet in the stable repository.  Until then, I have
> created a custom boot.iso (and pxeboot images) that you can point virt-install
> (or koan) to.
> 
> http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/
> 
> If you are able to boot anaconda-15.22-1, can you please add positive karma
> feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

This fix in comment #14 has worked for me too. 
If it helps to narrow down why it is not being seen more, I am updating my local copy infrequently, this bug did not occur in the copy I made on March 10th, and I was experiencing the problem on March 16th (first reported here though 13th, so that means 3 days really).
Comment 30 Adam Williamson 2011-03-18 14:15:35 EDT
Discussed at 2011-03-18 blocker review meeting. We agreed to make this bug specific to the lorax bug which the underlying glibc bug causes, and make it a Beta blocker. We cloned this bug off for the glibc issue, so that on doesn't get lost. This bug can be closed once the workaround is implemented and pushed.
Comment 31 Adam Williamson 2011-03-18 14:16:53 EDT
The clone for the glibc issue is https://bugzilla.redhat.com/show_bug.cgi?id=688990 .
Comment 32 Chris Lumens 2011-03-20 10:11:53 EDT
*** Bug 689159 has been marked as a duplicate of this bug. ***
Comment 33 Ales Kozumplik 2011-03-21 03:52:27 EDT
Fixed by eefd77e1b116a6821b402013983f64a1ae86d23b on master and c0a8d7deff4b5a928873249c5436f453ad4a8341 on f15-branch.
Comment 34 Fedora Update System 2011-03-21 10:01:29 EDT
lorax-0.3.2-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/lorax-0.3.2-1.fc15
Comment 35 James Laska 2011-03-21 13:43:39 EDT
Verified this fix in a private boot.iso build using the updated lorax and anaconda packages.
Comment 36 Tim Flink 2011-03-21 15:56:21 EDT
Verified this fix using a private boot.iso in a QEMU/KVM x86_64 VM.
Comment 37 Fedora Update System 2011-03-23 00:58:30 EDT
lorax-0.3.2-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 38 Ales Kozumplik 2011-03-23 11:36:26 EDT
*** Bug 690180 has been marked as a duplicate of this bug. ***
Comment 39 Ales Kozumplik 2011-03-25 07:36:33 EDT
*** Bug 690742 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.