Bug 1201897

Created attachment 1001470 [details]
backtrace

Dang it, missed that the core file was rejected by bugzilla, on account of hugeness. Here it is: https://dshea.fedorapeople.org/1393.core.gz

Hi David,

I can't currently access the core file (hitting 302 Forbidden). Could you change the permissions of the core file so that the file is world readable?

Oops. Fixed.

*** Bug 1204294 has been marked as a duplicate of this bug. ***

I'm seeing this with F22 Beta TC4 - at least when I do a server netinst in openQA or a virt-manager VM, the screen goes black shortly after the hub comes up, and on ctrl-alt-f1 I see a traceback running through libpython, libhawkey, libedit and libpthread.

Two other testers are reporting something that sounds 99% likely to be the same bug (black screen shortly after reaching install hub), so proposing this as a Beta blocker as it seems to commonly violate:

"When using a dedicated installer image, the installer must be able to complete an installation using the text, graphical and VNC installation interfaces. "

https://fedoraproject.org/wiki/Fedora_22_Alpha_Release_Criteria#Installation_interfaces

seen in VirtualBox and boxes but not in dd USB to HD install
https://fedoraproject.org/wiki/Test_Results:Fedora_22_Beta_TC4_Installation?rd=Test_Results:Current_Installation_Test#Default_boot_and_install

This is an incredibly weird issue. Hawkey is not linked against libedit, it does not #include it and libedit does not even export the map_init function as part of its API. A bit of source code reading uncovers that the map_init function is supposed to be included and run from libsolv. This suggests that the symbol resolution have gone wrong for some reason. It also explains why the app got SIGBUS (misaligned memory access) at the place it did.

As I'm still not sure what caused the symbol resolution problems, I can't say for sure what the solution is but one of the suspects here is the new glibc 5.0 version which requires app rebuild in some cases (the new libedit package rebuilt for fc23 should already be available, maybe that will help).

Ah, sorry for the typo, my comment was supposed to refer to the new gcc 5.0 version, not the glibc.

*** Bug 1204507 has been marked as a duplicate of this bug. ***

Discussed at 2015-03-23 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2015-03-23/f22-blocker-review.2015-03-23-16.02.log.txt . Accepted as a blocker per criterion cited in #c6.

Boris, if you think rebuilding libedit (and anything else?) might help, can you do so for F22? Thanks!

@awilliam: I've already done that few days back for libedit (Mar 18th). I do not have commit rights for other repos. I've checked and it is indeed compiled against gcc-5.0. It is the libedit-3.1-10.20141030cvs.fc22 version. I did not test it just yet though.

If that does not help then I'm starting to run out of ideas of what could have caused the libedit call. I've checked the hawkey sources and compiled libraries for the presence of libedit bits but there are none. It would be nice if we could get hawkey and probably even libsolv maintainer(s) involved to take a look then.

As Boris investigated the bug is there probably due to symbol collision. AFAIK hawkey does not have any libedit dependency.

# repoquery --tree-requires hawkey --releasever=22 --disablerepo=* --enablerepo=fedora | grep libedit
# <nothing>

Libsolv's map_init is called from hawkey too. The newest release of hawkey in f22 is built in gcc-5.0.0-0.15.fc22.x86_64, libsolv and libedit in gcc-5.0.0-0.17.fc22.x86_64. I will do the hawkey rebuild.

If it didn't help, please attach small single-threaded reproducer of hawkey/dnf crash.

well, no joy...I tried building a boot.iso with:

libedit-3.1-10.20141030cvs.fc22
librepo-1.7.13-1.fc22
libsolv-0.6.8-3.fc22
hawkey-0.5.3-2.1.fc22 (that's 0.5.3-2, rebuilt with newer gcc)

and anaconda crashes identically. I'm not capable of producing a small single-threaded reproducer, I'm afraid, I'm just the test monkey...

*** Bug 1205071 has been marked as a duplicate of this bug. ***

Boris, would be possible to define "protected" as static? AFAIK these functions are not between includes in devel package. Or make at least map_init static, please.

Functions in libsolv are wrongly without any prefix too but exported in devel and used by other projects. Renaming these would be more difficult.

Jan, the protected map_init function is shared between the separate libedit source files so I can't define it as static. I might be able to rename it (probably with all the other protected map_* functions) but only as a temporary workaround until the real issue is fixed -- why is the function even being called? Maybe, we should involve gcc people here?

The problem is that both libsolv (via dnf and hawkey) and libedit (via the readline module maybe? We're not 100% sure) are being loaded into the same program. That detail is not likely to change. The only means that C has of differentiating between functions is the function name, and these two functions have the same name, and it just happened to work out from the other of things being loaded and called that in this case the wrong function was chosen for this name. It's kind of a crappy name.

The gcc visibility attributes might be able to make this function usable across modules but stay local to the shared object.

So, we have a fairly solid theory here now - it involves llvm-libs .

llvm-libs has a file /usr/lib64/llvm/readline.so , and a /etc/ld.so.conf.d/llvm-x86_64.conf which reads:

/usr/lib64/llvm

Between TC3 and TC4, we restored the file /etc/ld.so.conf to the anaconda environment, which causes those /etc/ld.so.conf.d/*.conf files to actually be loaded. We did that to fix https://bugzilla.redhat.com/show_bug.cgi?id=1204031 .

So in TC4, /usr/lib64/llvm/readline.so will be in the linker cache; prior to TC3 it was not. It seems pretty reasonable to suspect that suspiciously-named lib is gumming up the works here, and indeed davidshea says if he takes away /etc/ld.so.conf.d/llvm-x86_64.conf and re-runs ldconfig, the crash stops happening.

ajax, you're the llvm maintainer - any thoughts on why it has this readline.so lib at all? it seems like anaconda isn't doing anything wrong here, and this all seems to be caused by some unfortunate naming in other libs.

The readline.so thing may have been a red herring. Moving readline.so out of the way but keeping $libdir/llvm configured still crashes. I have no clue how any of this works anymore.

Turns out readline.so isn't the issue, but *something* in llvm seems to be involved:

<davidshea> if I remove the llvm config file from ld.so.conf.d and rerun ldconfig, no crash
<davidshea> hm. maybe it's not that readline.so that's the problem. if I remove that particular file, but leave the rest of them, and rerun ldconfig, it still crashes

Also possibly of use in triaging whatever the crap's going on here: the bug affects only non-live GUI installs. It doesn't affect non-live TUI installs or live installs.

So I think what's going on is the stuff discussed from #c13 to #c18 is the real bug here. llvm's involvement is simply that it's one of very few things in the anaconda env that are linked against libedit. The only other thing I can find is libxatracker.so.2 , which probably isn't actually loaded with anaconda.

So when the llvm libs aren't in the linker cache, libedit never actually gets loaded with anaconda, and the collision between hawkey and libedit doesn't happen.

When the llvm libs *are* in the linker cache, libedit gets loaded with anaconda (probably through the graphics layers; mesa-dri-drivers depends on llvm-libs), and the collision happens.

I was looking at the contents of /etc/lib64/llvm and actually all the libraries there are linked against libedit so that explains why removing the llvm ld.so.conf.d fixes the issue.

I will try to use some gcc magic to not export all the protected symbols outside the libedit.so shared library to fix this.

I've performed a scratch build [1] where I set the visibility of all the libedit protected function symbols to hidden (i.e. they are not placed in dynamic symbol table). The nm -D libedit.so* command does indeed show no map_* symbols. Could anyone please test the build and let me know if it helps so that I can push the patch to the fedora dist-git?

[1] http://koji.fedoraproject.org/koji/taskinfo?taskID=9325045

will do. ajax also suggested a possible fix, with a scratch build of libsolv:

http://koji.fedoraproject.org/koji/taskinfo?taskID=9325563

what he changed was to add this to %make:

export LDFLAGS="-Wl,-Bsymbolic %{?__global_ldflags}"

He also posted a scratch build of llvm, http://koji.fedoraproject.org/koji/taskinfo?taskID=9325345 - though I'm not sure what that does, I think the only practical help llvm could be here is if it stopped linking against libedit entirely.

I'll test building boot.isos with each of the scratch builds, anyway, and see what happens.

ajax's libsolv build does not work for me; a boot.iso built with that still crashes.

However, Boris' libedit build does work! A boot.iso built with libedit-3.1-11.20141030cvs.fc23.x86_64.rpm does not crash for me.

Could you do an official build and update, Boris? Thanks a lot!

Done, built for f22 as well as rawhide, I'll push the fedora update for f22 right away.

libedit-3.1-11.20141030cvs.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/libedit-3.1-11.20141030cvs.fc22

Package libedit-3.1-12.20150325cvs.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libedit-3.1-12.20150325cvs.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-4925/libedit-3.1-12.20150325cvs.fc22
then log in and leave karma (feedback).

libedit-3.1-12.20150325cvs.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.