Bug 684742 - glibc dynamic library loading bug causes installs to fail
Summary: glibc dynamic library loading bug causes installs to fail
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: lorax
Version: 15
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Ales Kozumplik
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Keywords: Reopened
: 684847 688514 688524 688682 689159 690180 690742 (view as bug list)
Depends On:
Blocks: F15Beta, F15BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2011-03-14 12:08 UTC by Jakub Hrozek
Modified: 2014-09-30 23:39 UTC (History)
28 users (show)

(edit)
Clone Of:
: 688990 (view as bug list)
(edit)
Last Closed: 2011-03-23 04:58:52 UTC


Attachments (Terms of Use)
installation screenshot (30.98 KB, image/png)
2011-03-14 15:39 UTC, Jakub Hrozek
no flags Details
sanitized kickstart (2.41 KB, text/plain)
2011-03-14 16:28 UTC, Jakub Hrozek
no flags Details
/tmp/syslog (10.41 KB, application/x-bzip2)
2011-03-14 16:30 UTC, Jakub Hrozek
no flags Details
anaconda-logs.tgz (13.25 KB, application/x-gzip)
2011-03-14 17:27 UTC, James Laska
no flags Details

Description Jakub Hrozek 2011-03-14 12:08:03 UTC
Description of problem:
  File "/usr/lib64/python2.7/site-packages/pyanaconda/kickstart.py", line 21, in <module>
    from storage.deviceaction import *
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/__init__.py", line 48, in <module>
    from devicelibs.crypto import generateBackupPassphrase
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/devicelibs/crypto.py", line 24, in <module>
    from pycryptsetup import CryptSetup
  File "/usr/lib64/python2.7/site-packages/pycryptsetup/__init__.py", line 22, in <module>
    import cryptsetup
ImportError: /lib64/libcryptsetup.so.1: undefined symbol: uuid_generate, version UUID_1.0

Version-Release number of selected component (if applicable):
anacoda-15.20.1

How reproducible:
always

Steps to Reproduce:
1. start installation with a kickstart
  
Actual results:
traceback

Comment 1 Chris Lumens 2011-03-14 14:51:12 UTC
Does libuuid exist on your installation media?  Can you attach the complete traceback?

Comment 2 Jakub Hrozek 2011-03-14 15:39:17 UTC
(In reply to comment #1)
> Does libuuid exist on your installation media? 

Not sure if I answer your question completely - there's no libuuid package, but it looks like the installation image contains just files. find(1) tells me this:

./lib64/libuuid.so.1.3.0
./lib64/libuuid.so.1
./proc/sys/kernel/random/uuid
./bin/dbus-uuidgen
./sys/devices/virtual/dmi/id/product_uuid
./usr/lib64/python2.7/uuid.py
./usr/lib64/python2.7/uuid.pyc


> Can you attach the complete
> traceback?

What I pasted /is/ the complete traceback, just reproduced with python -c "import cryptsetup" because I couldn't find it in /tmp/anaconda.log

I'm attaching a screenshot of the traceback so we're on the same page.

Comment 3 Jakub Hrozek 2011-03-14 15:39:57 UTC
Created attachment 484229 [details]
installation screenshot

Comment 4 James Laska 2011-03-14 15:56:39 UTC
(In reply to comment #3)
> Created attachment 484229 [details]
> installation screenshot

To help me reproduce your failure, can you provide some information on how you are hitting this issue?

1. What version of anaconda are you using to install: F-15-Alpha, nightly, custom?
2. How are you booting the installer: DVD, netinst.iso, PXEboot?
3. What boot parameters are you adding: any ks= or similar?
4. If using kickstart, can you provide the ks.cfg?
5. If doing manual, can you detail the manual selections you make that contribute to this failure?

Comment 5 Chris Lumens 2011-03-14 16:01:11 UTC
Additionally, I'm wondering if there's media read errors going on here, since the code path isn't protected at all.  That is, I think just about everyone would be hitting this problem since the import's at the very top of kickstart.py and should always be happening.  Can you grab /tmp/syslog?

Comment 6 Jakub Hrozek 2011-03-14 16:27:06 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > Created attachment 484229 [details]
> > installation screenshot
> 
> To help me reproduce your failure, can you provide some information on how you
> are hitting this issue?
> 
> 1. What version of anaconda are you using to install: F-15-Alpha, nightly,
> custom?

Whatever is in http://download.fedora.redhat.com/pub/fedora/linux/development/15/x86_64/os/ (or rather our internal mirror of it)

> 2. How are you booting the installer: DVD, netinst.iso, PXEboot?

koan --virt, the tree is available via NFS. I pointed cobbler at vmlinuz and initrd from the mirror, not sure what magic does koan do after that.

> 3. What boot parameters are you adding: any ks= or similar?

yep, a custom kickstart

The full kernel command line looks like:
method=$url ksdevice=link lang= text ks=http://$cobbler_server/cblr/svc/op/ks/pr
ofile/Fedora-15-x86_64-minimal selinux=0 dhcpclass=ipa-lab-vms kssendmac 

> 4. If using kickstart, can you provide the ks.cfg?

OK

Comment 7 Jakub Hrozek 2011-03-14 16:27:51 UTC
(In reply to comment #6)
> > 2. How are you booting the installer: DVD, netinst.iso, PXEboot?
> 
> koan --virt, the tree is available via NFS. I pointed cobbler at vmlinuz and
> initrd from the mirror, not sure what magic does koan do after that.

Sorry, not NFS, but rather HTTP.

Comment 8 Jakub Hrozek 2011-03-14 16:28:35 UTC
Created attachment 484248 [details]
sanitized kickstart

Comment 9 Jakub Hrozek 2011-03-14 16:29:29 UTC
(In reply to comment #5)
> Additionally, I'm wondering if there's media read errors going on here, since
> the code path isn't protected at all.  That is, I think just about everyone
> would be hitting this problem since the import's at the very top of
> kickstart.py and should always be happening.  Can you grab /tmp/syslog?

Since this is a network install, I doubt it, but sure, I'll attach.

Can it be that the mirror is in some kind of odd state?

Comment 10 Jakub Hrozek 2011-03-14 16:30:57 UTC
Created attachment 484249 [details]
/tmp/syslog

Comment 11 Chris Lumens 2011-03-14 17:03:27 UTC
*** Bug 684847 has been marked as a duplicate of this bug. ***

Comment 12 James Laska 2011-03-14 17:27:39 UTC
Created attachment 484257 [details]
anaconda-logs.tgz

Reproduced with F-15-Alpha (anaconda-15.20.1-1) using the provided ks.cfg.  Attaching log files.

-rw-r--r-- root/root      5872 2011-03-14 13:24 tmp/anaconda.log
-rw-r--r-- root/root         0 2011-03-14 13:24 tmp/ifcfg.log
-rw-r--r-- root/root         0 2011-03-14 13:24 tmp/program.log
-rw-r--r-- root/root      2609 2011-03-14 13:24 tmp/storage.log
-rw-r--r-- root/root     50101 2011-03-14 13:24 tmp/syslog

I was *unable* to reproduce the bug using anaconda-15.22-1 along with the provided ks.cfg.

Comment 13 Jakub Hrozek 2011-03-14 22:23:07 UTC
(In reply to comment #12)
> I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> provided ks.cfg.

How do I test a new anaconda version? I see that 15.22 was built in koji, do I need to regenerate any of the images?

Thank you for testing, James!

Comment 14 James Laska 2011-03-15 12:14:23 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> > provided ks.cfg.
> 
> How do I test a new anaconda version? I see that 15.22 was built in koji, do I
> need to regenerate any of the images?

F-15 installation images are created daily at http://download.fedoraproject.org/pub/fedora/linux/development/15/x86_64/os

However, those images use only content that is in the base F-15 (stable) repo.  anaconda-15.22.1 is not yet in the stable repository.  Until then, I have created a custom boot.iso (and pxeboot images) that you can point virt-install (or koan) to.

http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/

If you are able to boot anaconda-15.22-1, can you please add positive karma feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

Comment 15 Jakub Hrozek 2011-03-15 12:50:00 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > I was *unable* to reproduce the bug using anaconda-15.22-1 along with the
> > > provided ks.cfg.
> > 
> > How do I test a new anaconda version? I see that 15.22 was built in koji, do I
> > need to regenerate any of the images?
> 
> F-15 installation images are created daily at
> http://download.fedoraproject.org/pub/fedora/linux/development/15/x86_64/os
> 
> However, those images use only content that is in the base F-15 (stable) repo. 
> anaconda-15.22.1 is not yet in the stable repository.  Until then, I have
> created a custom boot.iso (and pxeboot images) that you can point virt-install
> (or koan) to.
> 
> http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/
> 
> If you are able to boot anaconda-15.22-1, can you please add positive karma
> feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

Works for me, too. Karma++. Thanks for providing the images!

Comment 16 Chris Lumens 2011-03-15 20:03:32 UTC
Shorter reproducer, from python on tty2:

>>> from pyanaconda.storage import *

This avoids all the path weirdness we do in the anaconda script, so maybe it helps narrow down what's going on.

Comment 17 Jakub Hrozek 2011-03-17 09:52:49 UTC
Reopening. 

I hit this again with 15.22 now that it got pushed into repos. Exactly the same traceback, reproducible with:

python -c "from pyanaconda.storage import *"

I'm not sure why I didn't hit it with James' custom initrd - maybe during the initrd build he rebuilt the bindings that seem to be failing?

Comment 18 Tao Wu 2011-03-17 10:35:32 UTC
*** Bug 688524 has been marked as a duplicate of this bug. ***

Comment 19 James Laska 2011-03-17 11:59:49 UTC
*** Bug 688514 has been marked as a duplicate of this bug. ***

Comment 20 Karel Zak 2011-03-17 12:19:54 UTC
Try:

  $ readelf -a /lib64/libuuid.so.1 | grep uuid_generate

you have to see the symbol in the output, something like:

  63: 0000003e50e02640    41 FUNC    GLOBAL DEFAULT   12 uuid_generate@@UUID_1.0

Do you really have libuuid from util-linux package?

There was libuuid in e2fsprogs package. The newly compiled binaries (with verioned ABI) cannot be executed against the old library.

Comment 21 Ales Kozumplik 2011-03-17 14:38:57 UTC
A simpler reproducer:

import parted
try:
    import volume_key
except ImportError as e:
    print e
    print 'no volume key'
    volume_key = None

import pycryptsetup

Note the 'import parted' part is necessary. Why?

Comment 22 Ales Kozumplik 2011-03-17 17:04:44 UTC
Patch adding missing library (presence of which makes the problem go away) was sent for a review: https://www.redhat.com/archives/anaconda-devel-list/2011-March/msg00222.html

Note we still don't know the root cause of the import failures.

Comment 23 Ales Kozumplik 2011-03-17 17:39:02 UTC
This is not Anaconda specific, I am able to reproduce this on an installed system:
1) use the f15 alpha netinst to install the system, boot it
2) yum update
3) switch for the current f15 devel repo, e.g. http://download.fedora.redhat.com/pub/fedora/linux/development/15/x86_64/os/
4) yum install python-cryptsetup python-volume_key pyparted
5) hide away or remove /usr/lib64/libassuan.*
6) run the code in comment 21

the results are the volume_key complaining about a missing libassuan (that makes sense) but then right after cryptsetup complaining about missing uuid_generate (that makes no sense). 

a strange thing also is pycryptsetup can still be imported if the import of the other two packages is not attempted first. the order also matters. this is very weird and points to either

a) one of the packages playing nasty tricks with the import mechanism

b) more fundamental problem with dynamic libraries load mechanism.

David, have you ever seen anything like that? Who would you talk to (note I doubt it is specifically a python problem)?

Comment 24 Ales Kozumplik 2011-03-17 17:40:13 UTC
(forgot in the reproducer description: one needs to run ldconfig as root after step 5).

Comment 25 Chris Lumens 2011-03-17 17:41:38 UTC
*** Bug 688682 has been marked as a duplicate of this bug. ***

Comment 26 Ales Kozumplik 2011-03-17 17:50:52 UTC
I can also provide access to the virtual machine where everything is set up.

Comment 27 Ales Kozumplik 2011-03-18 09:18:25 UTC
I tried with an f14 kernel 2.6.35.6-45.fc14.x86_64 with no different result and then with an f14 glibc (glibc-2.12.90-17.x86_64) and the problem is gone (pycryptsetup can still be imported after a failed volume_key import).

Reassigning to glibc. Glibc mainteiner: I'll be happy to share the virtual machine where this is reproducible. Please do not let this sit, it seems like a serious, sneaky issue.

Comment 28 Ales Kozumplik 2011-03-18 09:43:44 UTC
I doubt this is util-linux-ng problem.

In short: when we import few (unrelated?) python modules (which load dynamic libraries) and one of those modules legitimately fails to load do to a missing .so then pycryptsetup also fails to load (with: ImportError: /lib64/libcryptsetup.so.1: undefined symbol: uuid_generate. note the lib and the symbol ARE present). If the python interpreter is restarted then pycryptsetup can be loaded without a problem. 

And reverting glibc makes this go away.

Comment 29 Gareth 2011-03-18 11:02:13 UTC
(In reply to comment #14)

> However, those images use only content that is in the base F-15 (stable) repo. 
> anaconda-15.22.1 is not yet in the stable repository.  Until then, I have
> created a custom boot.iso (and pxeboot images) that you can point virt-install
> (or koan) to.
> 
> http://jlaska.fedorapeople.org/updates/anaconda-15.22-1/x86_64/os/
> 
> If you are able to boot anaconda-15.22-1, can you please add positive karma
> feedback to https://admin.fedoraproject.org/updates/anaconda-15.22-1.fc15

This fix in comment #14 has worked for me too. 
If it helps to narrow down why it is not being seen more, I am updating my local copy infrequently, this bug did not occur in the copy I made on March 10th, and I was experiencing the problem on March 16th (first reported here though 13th, so that means 3 days really).

Comment 30 Adam Williamson 2011-03-18 18:15:35 UTC
Discussed at 2011-03-18 blocker review meeting. We agreed to make this bug specific to the lorax bug which the underlying glibc bug causes, and make it a Beta blocker. We cloned this bug off for the glibc issue, so that on doesn't get lost. This bug can be closed once the workaround is implemented and pushed.

Comment 31 Adam Williamson 2011-03-18 18:16:53 UTC
The clone for the glibc issue is https://bugzilla.redhat.com/show_bug.cgi?id=688990 .

Comment 32 Chris Lumens 2011-03-20 14:11:53 UTC
*** Bug 689159 has been marked as a duplicate of this bug. ***

Comment 33 Ales Kozumplik 2011-03-21 07:52:27 UTC
Fixed by eefd77e1b116a6821b402013983f64a1ae86d23b on master and c0a8d7deff4b5a928873249c5436f453ad4a8341 on f15-branch.

Comment 34 Fedora Update System 2011-03-21 14:01:29 UTC
lorax-0.3.2-1.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/lorax-0.3.2-1.fc15

Comment 35 James Laska 2011-03-21 17:43:39 UTC
Verified this fix in a private boot.iso build using the updated lorax and anaconda packages.

Comment 36 Tim Flink 2011-03-21 19:56:21 UTC
Verified this fix using a private boot.iso in a QEMU/KVM x86_64 VM.

Comment 37 Fedora Update System 2011-03-23 04:58:30 UTC
lorax-0.3.2-1.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 38 Ales Kozumplik 2011-03-23 15:36:26 UTC
*** Bug 690180 has been marked as a duplicate of this bug. ***

Comment 39 Ales Kozumplik 2011-03-25 11:36:33 UTC
*** Bug 690742 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.