Description of problem: I'm on a x86_64 Fedora 23 system and was attempting to get the development libraries and headers for 32 bit compilation for legacy C programs that needed the -m32 flag for gcc. These were glibc-2.22-10.fc.23.i686, etc. After installing these rpm's, I lost the ability to run a number of programs, including firefox, thunderbird, rpm, and dnf. After running strace on them, they all seem to get a abort signal after reading from /dev/urandom. The pattern is: open( "/dev/urandom", O_RDONLY ) = 26 read( 26, "....", 110 ) = 110 close( 26 ) sysinfo({uptime=4271, ....}) = 0 uname({sysname="Linux", ...} ) = 0 sysinfo( {uptime=427, ....}) = 0 stat( "/dev/urandom", {stmode=S_IFCHR|0666, st_rdev=makedev(1,9), ..} = 0 open( "/dev/urandom", O_RDONLY ) = 26 read( 26, "....", 1024 ) = 1024 close(26) = 0 rt_sigprocmask( SIG_UNBLOCK, [ABRT], NULL, 8 ) = 0 tgkill(2398, 2398, SIGABRT ) = 0 --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=2398, si_uid=1000} ... A test program to simply read from /dev/urandom doesn't have this behavior. And the pattern just prior to the error for all of the programs is the same, which makes me think that some library call has been affected. I also noticed that the directory /usr/lib/gcc/i686-redhat-linux/5.3.1 is missing, causing a lot of broken symbolic links in /usr/lib/gcc/x86_64-redhat-linux/5.3.1/32 Version-Release number of selected component (if applicable): I'm pretty sure that it was glibc-2.22-10.fc23.i686 and the development rpm's, but since I can no longer run dnf/rpm I'm not entirely sure on the version. How reproducible: I can't reproduce this now as my current machine is only half-functional and I can't run either rpm or dnf to try to undo or redo the problem. I think my only way forward is to back up my data and reinstall Fedora 23. I am unclear if this was just something that got corrupted for me alone or if there's a bug in one of the glibc*.i686 rpms for Fedora 23. (I've never had any problems with any of the hundreds of Fedora rpm's I've installed in any of my Fedora systems previously.) Thus, I'm adding this bug to bugzilla with the hopes that someone can double check the appropriate rpms in case there's a problem with them that might affect someone else (or me again after I reinstall the operating system). Thanks in advance for any help you might be able to provide.
Oh. And to be clear, I was installing the packages via "sudo dnf install xxxxx"
glibc32 is not what you think it is.
I'm running systems with glibc-2.22-10.fc23 both i686 and x86_64 versions without any problems. Nobody else has reported an issue against this released version for F23 and it passed all the bodhi checks and testers. I expect something is wrong with your hardware. I can't really offer any more advice than to try boot a rescue cd and recover your system that way, or reinstall and see what happens then. I'm marking this closed/insufficient data, please reopen when you have more details.
I am also having the same problem. I was trying to install the 32-bit support in connection with bring up an Android development system. I have the DNF log files, if that helps. Here is what was installed last: Installed: glibc.i686 2.22-10.fc23 glibc-devel.i686 2.22-10.fc23 libX11.i686 1.6.3-2.fc23 libX11-devel.i686 1.6.3-2.fc23 libXau.i686 1.0.8-5.fc23 libXext.i686 1.3.3-3.fc23 libXrandr.i686 1.5.0-2.fc23 libXrender.i686 0.9.9-2.fc23 libgcc.i686 5.3.1-2.fc23 libstdc++.i686 5.3.1-2.fc23 libxcb.i686 1.11.1-1.fc23 ncurses-devel.i686 5.9-21.20150214.fc23 ncurses-libs.i686 5.9-21.20150214.fc23 nss-softokn-freebl.i686 3.22.2-1.0.fc23 zlib.i686 1.2.8-9.fc23 zlib-devel.i686 1.2.8-9.fc23 Upgraded: glibc.x86_64 2.22-10.fc23 glibc-common.x86_64 2.22-10.fc23 glibc-devel.x86_64 2.22-10.fc23 glibc-headers.x86_64 2.22-10.fc23 nss-softokn-freebl.x86_64 3.22.2-1.0.fc23 Please do have another look at this. I am not in a hurry to reload the whole system so any crumbs you might like to look at, I can provide.
(In reply to George Anzinger from comment #4) > Please do have another look at this. I am not in a hurry to reload the > whole system so any crumbs you might like to look at, I can provide. Can you reproduce this? We need a backtrace with debuginfo packages installed at least.
I wrote a short script (tcl) that runs strace on 5 programs that end up aborting them selves. The script then collects all the file names and throws any out that don't appear in all 5 'strace's. It then prints the 'mtime' of each remaining file along with it name. Here is the result: (sample size: 5, traced programs: rpm -qf /opt dnf install foo nm-connection-editor firefox gdb /usr/bin/rpm /usr/src/outgoing/Dropbox/Public/George/core/core.rpm Thu Mar 03 15:33:41 MST 2016 /etc/ld.so.cache Fri Feb 19 14:03:28 MST 2016 /usr/lib64/libc-2.22.so Fri Feb 19 14:03:28 MST 2016 /usr/lib64/libpthread-2.22.so Fri Feb 19 14:03:27 MST 2016 /usr/lib64/libdl-2.22.so Fri Feb 19 14:03:27 MST 2016 /usr/lib64/libm-2.22.so The dnf log above was generated on Thu Mar 03 15:33. The prime suspect is thus reduced to /etc/ld.so.cache. I would take a wild guess that this file is produced by a script in one (or more) of the rmps listed in comment 4. Is it possible that /usr/sbin/glibc_post_upgrade might have something to do with this? Let me know if I can help. As to backtraces and debuginfo packages, how can I tell if they are installed? As you can see on the the programs that is aborting is gdb which I found while trying to have it look a core file for rpm. My running system (same Fedora release but using glibc 2.22-7 (not -10) might be able to download the required debug packages (if they are not installed) and then I could manually install them. Still, I hope the /etc/ld.so.cache info is enough.
(In reply to George Anzinger from comment #6) > As to backtraces and debuginfo packages, how can I tell if they are > installed? You can run “rpm -q glibc-debuginfo”. “dnf debuginfo-install glibc-2.22-10.fc23.i686” should be able to install the correct version. You may have to remove a previously installed version of the debuginfo packages, though. Alternatively, you could compress and attach the core file. The one from RPM should not contain any private data.
Created attachment 1135602 [details] xz compression of rpm core file The system seems to catch these on its own, saving them in /var/lib/
(In reply to George Anzinger from comment #8) > Created attachment 1135602 [details] > xz compression of rpm core file > > The system seems to catch these on its own, saving them in /var/lib/ I cannot find complete matching debuginfo for this core file. What is the version of the sqlite-libs package? I saw one odd thing. Can you double-check you do not have any outdated DSOs in the /g/lib directory?
Searching for sqlite in /lib64 gives: /libsqlite3.so.0 ./libsqlite3.so.0.8.6 <-----------------I assume this is what you need ./python2.7/lib-dynload/_sqlite3.so ./python2.7/site-packages/_sqlitecache.so ./python2.7/site-packages/sqlitecachec.py ./python2.7/site-packages/sqlitecachec.pyc ./python2.7/site-packages/sqlitecachec.pyo ./python2.7/sqlite3 ./python3.4/lib-dynload/_sqlite3.cpython-34m.so ./python3.4/sqlite3 ./qt4/plugins/landmarks/libqtlandmarks_sqlite.so ./qt4/plugins/sqldrivers/libqsqlite.so ./redland/librdf_storage_sqlite.so ./thunderbird/libmozsqlite3.so Remember I can not ask rpm for this info..
(In reply to George Anzinger from comment #10) > Searching for sqlite in /lib64 gives: > /libsqlite3.so.0 > ./libsqlite3.so.0.8.6 <-----------------I assume this is what you need What's size and sha1sum of this file? Thanks. Did you check the /g/lib directory? Does it even exist?
Size is 872,216 sha1sum from problem machine and the good one: 152b777641356903d83b2ee9aff63ac54b9b338b /usr/lib64/libsqlite3.so.0.8.6 problem 152b777641356903d83b2ee9aff63ac54b9b338b /usr/lib64/libsqlite3.so.0.8.6 good I can find no /g/lib or /usr/lib64/g directory, so no, it does not exist (on either machine).
(In reply to George Anzinger from comment #12) > Size is 872,216 > > sha1sum from problem machine and the good one: > 152b777641356903d83b2ee9aff63ac54b9b338b /usr/lib64/libsqlite3.so.0.8.6 > problem > 152b777641356903d83b2ee9aff63ac54b9b338b /usr/lib64/libsqlite3.so.0.8.6 good Okay, this is sqlite-3.10.2-1.fc23.x86_64.rpm. I have installed it, but GDB still does not provide sensible output for the core file. I suspect I do not have the correct RPM version. Can you tell me the size and sha1sum of /usr/bin/rpm (the version which generated the coredump)? > I can find no /g/lib or /usr/lib64/g directory, so no, it does not exist (on > either machine). Did you set LD_LIBRARY_PATH? Maybe it was clobbered.
Ah, the mystery of /g/lib: I usually copy .bashrc from system to system. This has been going on since 1997 or so. Somewhere along the way LD_LIBRARY_PATH=/g/lib was added. I don't recall why or when any more.. /bin/dnf is softlinked to dnf-3: -rwxr-xr-x. 1 root root 1897 Jan 25 09:42 /bin/dnf-3* 7e7c4e90d4389a9083574d1c174e009549695f9a /usr/bin/dnf-3
(In reply to George Anzinger from comment #14) > /bin/dnf is softlinked to dnf-3: > > -rwxr-xr-x. 1 root root 1897 Jan 25 09:42 /bin/dnf-3* > > 7e7c4e90d4389a9083574d1c174e009549695f9a /usr/bin/dnf-3 Sorry, but I need this information for /usr/bin/rpm. The core is for rpm, not dnf.
-rwxr-xr-x. 1 root root 19968 Jan 29 09:25 /bin/rpm* [george@hp360 core]$ sha1sum /bin/rpm 70ef74c8de7beb8ea49b0142d86bdcf735b3b362 /bin/rpm
(In reply to George Anzinger from comment #16) > -rwxr-xr-x. 1 root root 19968 Jan 29 09:25 /bin/rpm* > > > [george@hp360 core]$ sha1sum /bin/rpm > 70ef74c8de7beb8ea49b0142d86bdcf735b3b362 /bin/rpm Thanks. Sorry for the slow progress on this. Now I need the same data (ls -l output and sha1 sum) for: /lib64/libfreebl3.so Unfortunately, I don't know how to speed this up, I'll the next file to request only after getting the information for this one.
-rwxr-xr-x. 1 root root 502112 Feb 29 10:48 /lib64/libfreebl3.so* [george@hp360 tools]$ sha1sum /lib64/libfreebl3.so 5b38be073edbac2b003c234bf87538df14d58212 /lib64/libfreebl3.so
Thanks, we are finally getting somewhere! This file comes from nss-softokn-freebl-3.22.2-1.0.fc23.x86_64. With this, I finally get a nice backtrace: #0 0x00007efdfea9ba98 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007efdfea9d69a in __GI_abort () at abort.c:89 #2 0x00007efdfd494ce4 in PR_GetEnvSecure_stub (var=<optimized out>) at stubs.c:473 #3 0x00007efdfd4475f1 in RNG_SystemInfoForRNG () at unix_rand.c:892 #4 0x00007efdfd457a18 in rng_init () at drbg.c:432 #5 0x00007efdfe0074ea in PR_CallOnce (once=0x7efdfd6bb3a0 <coRNGInit>, func=<optimized out>) at ../../../nspr/pr/src/misc/prinit.c:786 #6 0x00007efdfd4572d7 in RNG_RNGInit () at drbg.c:476 #7 0x00007efdfd9a340b in nsc_CommonInitialize (pReserved=0x7ffc27bf78e0, isFIPS=0) at pkcs11.c:2944 #8 0x00007efdfd9a381a in NSC_Finalize (pReserved=0x7efdfdbd1950 <nsc_init>) at pkcs11.c:3148 #9 0x00007efdff6abdcc in secmod_ModuleInit () from /lib64/libnss3.so #10 0x00007efdff6ac42a in secmod_LoadPKCS11Module () from /lib64/libnss3.so #11 0x00007efdff6b897f in SECMOD_LoadModule () from /lib64/libnss3.so #12 0x00007efdff6b8a80 in SECMOD_LoadModule () from /lib64/libnss3.so #13 0x00007efdff68720b in nss_Init () from /lib64/libnss3.so #14 0x00007efdff68790e in NSS_InitWithMerge () from /lib64/libnss3.so The call to the abort function in libfreebl3.so looks like this: #2 0x00007efdfd494ce4 in PR_GetEnvSecure_stub (var=<optimized out>) at stubs.c:473 473 abort(); (gdb) l 468 469 extern char* 470 PR_GetEnvSecure_stub(const char *var) 471 { 472 STUB_SAFE_CALL1(PR_GetEnvSecure, var); 473 abort(); 474 return NULL; 475 } 476 477 I don't know why this doesn't get redirect to PR_GetEnvSecure or secure_getenv. Certainly this was the attempt.
I replaced /usr/lib64/libfreebl3.so from rpm nss-softokn-freebl-3.22.2-1.0.fc23.x86_64 with the same file from rpm nss-softokn-freebl-3.21.0-1.1.fc23.x86_64 Now dnf, nm-connection-editor, firefox, gdb and rpm all seem to be working. Thanks Florian, saved me a lot of work rebuilding the system.
Elio, Something is going wrong with the stub function, probably initialization. The call to freebl is happening from regular NSS and softoken, both of which have dependencies on NSPR, which means STUB_SAFE_CALL1() should call PR_GetEnvSecure and return. Since it's not, it means either FREEBL_InitStubs() is not getting called, or it's being called, but failing to the nsprlibrary (which should already be loaded by the loader. Look at FREEBL_InitStubs in nss/lib/freebl/stubs.c Also make sure FREEBLnsprGlobalLib is not set to zero when we abort. (if it is non-zero, then it's likely freeblInitNSPR is failing. FREEBL_InitStubs should be called from FREEBL_GetVector(). bob
A couple of comments. My fix (using the version from nss-softokn-freebl-3.21.0-1.1.fc23.x86_64) caused login to fail. Reverting allowed login. I then changed back to the prior version and did "dnf distro-sync". This loaded nss-softokn-freebl-3.23.0-1.0.fc23.x86_64. Ldconfig complains that it is not a symbolic link, but otherwise, every thing works again. (Could ldconfig be complaining about the version in /lib ? (which is the 32-bit version).
PR_GetEnvSecure is a new function, which had been added in NSPR 4.12. Did you install the nspr 4.12 update? NSS since version 3.22.1 requires NSPR 4.12. Does the NSS package have a requirement to pull in NSPR 4.12?
I'm afraid I can't provide any more information. I blew away the old Fedora 23 installation (KDE spin) almost a month ago. I couldn't wait because the problem took down my main work computer. I didn't bother to save any logs, etc. as no one gave me any advice on any logs that would have been worth saving. (And with dnf and rpm down, it was very difficult for me to figure exactly what packages were present or not. Unfortunately, I hadn't been writing down what packages I had added to the system.) But the list of dead programs is very similar to what George Anzinger has been reporting, so I suspect that the root cause might be same. If there was a missing package dependency, I easy could have tripped over it as I was relying on dnf/rpm to sort that out for me. I have since reinstalled the Fedora 23 KDE spin on exactly the same hardware from exactly the same burned iso image, and I'm up and running now (and I am currently compiling both programs for both 32 and 64 bits). And I have reinstalled much of the missing packages from the base install that I use for my work (but I might not have gotten to them all by now). Is it possible there was a window of time when the dependencies were wrong, but now they're right?
Bob, this is the implementation of PR_GetEnvSecure_stub, which calls abort(), like many other stubs: extern char* PR_GetEnvSecure_stub(const char *var) { STUB_SAFE_CALL1(PR_GetEnvSecure, var); abort(); return NULL; } Is this a function that must not call abort?
FYI, the new stub was added with this commit: https://hg.mozilla.org/projects/nss/rev/4f727a27da00
If the function is used in the standalone hashing path, then it needs an implementation. If it is only used in the softokn/nss path, then it doesn't. I've checked and we don't do any GetEnvSecure in the nsslowinit.c patch. The stack tracebacks are clearly showing it being called from softokn. One thing that could cause this is if we have an old version of NSPR that doesn't include PR_GetEnvSecure() in it, but I think NSS and softoken explicitly call the NSPR function, so that would force it to be in the library (or the loader would blow up). In any case the fact that it's aborting in a path that clearly has NSS and softoken in the stack means we should already have NSPR loaded, and softokn all ways calls freebl through the freebl vector, which should have forces an FREEB_InitStubs call. bob
I suppose if you have an old version of NSS, softokn and NSPR but a new version of freebl you could conceivably wind up in this state. freebl doesn't have any explicit dependencies on nspr (that's what the stubs are all about). I'm not sure how to fix this other than maybe locking softokn to a specific version of freebl. I think we did that once and wound up with problems at one point.
(In reply to Bob Relyea from comment #28) > I suppose if you have an old version of NSS, softokn and NSPR but a new > version of freebl you could conceivably wind up in this state. freebl > doesn't have any explicit dependencies on nspr (that's what the stubs are > all about). You could still declare such a dependency because it is currently functionally present. I now suspect that the installation of glibc.i686 on a previously x86_64-only system caused an upgrade of freebl on x86_64 (to match the i686 version which was being installed), without pulling in all the needed dependencies. Come to think of it, this is exactly what comment #4 shows. > I'm not sure how to fix this other than maybe locking softokn to > a specific version of freebl. I think we did that once and wound up with > problems at one point. Would it be possible to change the stub to just return NULL instead of aborting the process? For this particular function, this should be a safe choice, and it would at least allow RPM & Co. to continue to work, I think.
(In reply to Kai Engert (:kaie) from comment #23) > PR_GetEnvSecure is a new function, which had been added in NSPR 4.12. > > Did you install the nspr 4.12 update? > > NSS since version 3.22.1 requires NSPR 4.12. > Does the NSS package have a requirement to pull in NSPR 4.12? The dnf that loaded the package summary is above (comment 4). The actual request was to bring in the 32-bit packages to support Android development. I was surprised that it also updated the 64-bit stuff. I have no idea about NSPR.
(In reply to Florian Weimer from comment #29) > (In reply to Bob Relyea from comment #28) > > I suppose if you have an old version of NSS, softokn and NSPR but a new > > version of freebl you could conceivably wind up in this state. freebl > > doesn't have any explicit dependencies on nspr (that's what the stubs are > > all about). > > You could still declare such a dependency because it is currently > functionally present. I now suspect that the installation of glibc.i686 on > a previously x86_64-only system caused an upgrade of freebl on x86_64 (to > match the i686 version which was being installed), without pulling in all > the needed dependencies. Come to think of it, this is exactly what comment > #4 shows. > > > I'm not sure how to fix this other than maybe locking softokn to > > a specific version of freebl. I think we did that once and wound up with > > problems at one point. > > Would it be possible to change the stub to just return NULL instead of > aborting the process? For this particular function, this should be a safe > choice, and it would at least allow RPM & Co. to continue to work, I think. If the dependencies were right, this is a dnf error, if not...
(In reply to George Anzinger from comment #30) > (In reply to Kai Engert (:kaie) from comment #23) > > PR_GetEnvSecure is a new function, which had been added in NSPR 4.12. > > > > Did you install the nspr 4.12 update? > > > > NSS since version 3.22.1 requires NSPR 4.12. > > Does the NSS package have a requirement to pull in NSPR 4.12? > > > The dnf that loaded the package summary is above (comment 4). The actual > request was to bring in the 32-bit packages to support Android development. > I was surprised that it also updated the 64-bit stuff. I have no idea about > NSPR. I examined the dnf logs after the distro-sync (comment 22) and it, at that time, updated to nspr-4.12.0-1 (64bit). It is, I think, save to assume that an earlier version was installed when the fault happened (dnf does not log the prior version).
so it's probably not dnf here, I'm sure we haven't given dnf enough information. libfreebl purposefully does not have a dependency on nspr because glibc (or more exactly libcrypt) has a dependency on libfreebl and we don't want to bring in nspr just because you've loaded glibc. The issue here is when softoken uses libfreebl, libfreebl *does* need nspr, but so does softoken, so libfreebl depends on softoken's dependency. If softoken were locked to a specific version of libfreebl (which is easy to do since libfreebl is in the nss-softokn-freebl package, which is a subpackage of nss-softokn) then in theory dnf would know that updating libfreebl would mean updating softokn, which in turn would bring in the right version of nspr. I think we didn't do the lock because softokn is perfectly happy using a newer version of libfreebl (but we've just hit the case where the newer libfreebl needs a newer nspr to make it's services to softokn happy, so maybe we need to do the lock). Ideally libfreebl should have a dependency that says "if you have nspr, it should be at least this version, but if you don't have nspr, don't add it". That kind of semantic sounds like a nightmare for the package installer people, though (what is you add nspr after you've processed libfreebl). > Would it be possible to change the stub to just return NULL instead of > aborting the process? For this particular function, this should be a safe > choice, and it would at least allow RPM & Co. to continue to work, I think. That would silently turn off environmental control of freebl variables. It could cause problems if, for instance, you need to turn off hardware AES. Providing an actually stub implementation could work in this case, but the next time we add a new NSPR (or nss-util, which freebl also depends on if softoken is loaded). bob
*** Bug 1315811 has been marked as a duplicate of this bug. ***
(In reply to Bob Relyea from comment #33) > so it's probably not dnf here, I'm sure we haven't given dnf enough > information. libfreebl purposefully does not have a dependency on nspr because > glibc (or more exactly libcrypt) has a dependency on libfreebl and we don't > want to bring in nspr just because you've loaded glibc. Possibly in the near future you could use weak dependencies? > Ideally libfreebl should have a dependency that says "if you have nspr, it > should be at least this version, but if you don't have nspr, don't add it". > That kind of semantic sounds like a nightmare for the package installer > people, though (what is you add nspr after you've processed libfreebl). This might use boolean ("rich") dependencies: Requires: (nspr >= 4.12.0 if nspr) I *think* would do it, although there aren't any actual examples with "if" and versions, or for that matter of using the same package in both parts of the expression.
(In reply to Matthew Miller from comment #35) > Requires: (nspr >= 4.12.0 if nspr) Why not simply "Conflicts: nspr < 4.12.0" which works with old rpm, too?
(In reply to Kamil Dudka from comment #36) > (In reply to Matthew Miller from comment #35) > > Requires: (nspr >= 4.12.0 if nspr) > > Why not simply "Conflicts: nspr < 4.12.0" which works with old rpm, too? This seems like a great idea, thanks Kamil!
One Matt agrees with the suggestion, please set needinfo to Elio, to suggest that he implements it.
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
This message is a reminder that Fedora 23 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 23. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '23'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 23 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
Err, sorry, I missed this. The resolution suggested seems fine, if this happens to be still relevant at all.