Description of problem: glibc 2.30.9000-21.fc32 breaks firefox, all tabs error-out pipe error (101): Connection reset by peer: file /builddir/build/BUILD/firefox-71.0/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 358 dnf downgrade glibc-2.30.9000-20.fc32.x86_64 fixes the problem (I destroyed a system attempting a downgrade to fc31 glibc)
Nicolas, Thank you for filling this bug. Upstream recently improved the dynamic loader when it comes to handling dlopen() failures and rolling back state. It may be that Firefox is triggering a corner case we didn't consider. I was able to reproducer this when we were testing the changes before Rawhide deployment, but in my testing Rawhide was so broken already that it was to know if this was just a consequence of the other system brekage. The fact that you double-checked with a downgrade is really helpful. We're looking into this. Florian, I actually noticed this when I was testing the Rawhide upgrade with your scratch build, but since Rawhide is completely busted it wasn't clear to me if this was something that worked before or not. The fact that a downgrade to -20 fixes things means that we might have a case we need to fix in the recent ld.so commit/rollback work. This one is yours to triage, and it may be the same cause as the other issue, perhaps the tab is crashing with an assert because it's a distinct process.
2.30.9000-22.fc32 is still broken. I notice the issue when using icecat. Downgrade fixed things for me. I'm using rawhide exclusively and it works well for me. It might be that I uninstall packages blocking updates, temporarily, so that my system is more up to date.
When doing downgrades, going against recent composes works well. I imagine it's a lot safer than trying to downgrade to a version from a previous release. For example in this case I downgrade versus the November 29th compose. baseurl=https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20191129.n.0/compose/Everything/x86_64/os/ You could also use koji download-build to download recent rpms.
The downgrade issue is tracked as bug 1652867. It requires upstream work, which is currently under review. I was able to reproduce the crash with firefox-71.0-6.npgo.fc32.x86_64 and glibc-2.30.9000-21.fc32.x86_64. It looks like this: Core was generated by `/usr/lib64/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 1 -p'. Program terminated with signal SIGSYS, Bad system call. #0 __GI___open64_nocancel ( file=file@entry=0x7fc7d129d8e0 "/lib64/libsoftokn3.so", oflag=oflag@entry=524288) at ../sysdeps/unix/sysv/linux/open64_nocancel.c:45 #1 0x00007fc7e0e7b8bf in open_verify ( name=0x7fc7d129d8e0 "/lib64/libsoftokn3.so", fbp=fbp@entry=0x7ffe87d14470, loader=<optimized out>, whatcode=whatcode@entry=0, found_other_class=found_other_class@entry=0x7ffe87d1445f, free_name=free_name@entry=true, mode=-1879048190, fd=-1) at dl-load.c:1548 #2 0x00007fc7e0e7d3d6 in _dl_map_object (loader=loader@entry=0x7fc7e0abaed0, name=name@entry=0x7fc7d129d8a0 "/lib64/libsoftokn3.so", type=type@entry=2, trace_mode=trace_mode@entry=0, mode=mode@entry=-1879048190, nsid=<optimized out>) at dl-load.c:2164 #3 0x00007fc7e0e880a9 in dl_open_worker (a=a@entry=0x7ffe87d14a40) at dl-open.c:524 #4 0x00007fc7e0a2d258 in __GI__dl_catch_exception ( exception=exception@entry=0x7ffe87d149c0, operate=operate@entry=0x7fc7e0e88000 <dl_open_worker>, args=args@entry=0x7ffe87d14a40) at dl-error-skeleton.c:208 #5 0x00007fc7e0e87c3b in _dl_open ( file=0x7fc7d129d8a0 "/lib64/libsoftokn3.so", mode=-2147483646, caller_dlopen=0x7fc7e08b899b <PR_LoadLibraryWithFlags+219>, nsid=-2, argc=<optimized out>, argv=<optimized out>, env=0x7ffe87d15c38) at dl-open.c:855 #6 0x00007fc7e0e1939c in dlopen_doit (a=a@entry=0x7ffe87d14ce0) at dlopen.c:66 #7 0x00007fc7e0a2d258 in __GI__dl_catch_exception ( exception=exception@entry=0x7ffe87d14c80, operate=operate@entry=0x7fc7e0e19340 <dlopen_doit>, args=args@entry=0x7ffe87d14ce0) at dl-error-skeleton.c:208 #8 0x00007fc7e0a2d323 in __GI__dl_catch_error ( objname=objname@entry=0x7fc7e0619090, errstring=errstring@entry=0x7fc7e0619098, mallocedp=mallocedp@entry=0x7fc7e0619088, operate=operate@entry=0x7fc7e0e19340 <dlopen_doit>, args=args@entry=0x7ffe87d14ce0) at dl-error-skeleton.c:227 #9 0x00007fc7e0e19b09 in _dlerror_run ( operate=operate@entry=0x7fc7e0e19340 <dlopen_doit>, args=args@entry=0x7ffe87d14ce0) at dlerror.c:170 #10 0x00007fc7e0e1942a in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87 #11 0x00007fc7e08b899b in PR_LoadLibraryWithFlags () from /lib64/libnspr4.so #12 0x00007fc7d6d56b5c in loader_LoadLibInReferenceDir () from /lib64/libnssutil3.so #13 0x00007fc7d6d56bd1 in PORT_LoadLibraryFromOrigin () from /lib64/libnssutil3.so #14 0x00007fc7d6dafd32 in softoken_LoadDSO () from /lib64/libnss3.so #15 0x00007fc7e08bf01e in PR_CallOnce () from /lib64/libnspr4.so #16 0x00007fc7d6db6813 in secmod_LoadPKCS11Module () from /lib64/libnss3.so #17 0x00007fc7d6dc316d in SECMOD_LoadModule () from /lib64/libnss3.so #18 0x00007fc7d6d8fb7b in nss_Init () from /lib64/libnss3.so #19 0x00007fc7d6d903fc in NSS_NoDB_Init () from /lib64/libnss3.so #20 0x00007fc7dad98210 in EnsureNSSInitializedChromeOrContent () at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSComponent.cpp:128 #21 EnsureNSSInitializedChromeOrContent () at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSComponent.cpp:89 #22 0x00007fc7dad9d271 in mozilla::psm::Constructor<nsRandomGenerator, (nsresult (nsRandomGenerator::*)())0, (mozilla::psm::ProcessRestriction)1> ( aResult=0x7ffe87d15200, aIID=..., aOuter=0x0) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSModule.cpp:148 #23 mozilla::psm::NSSConstructor<nsRandomGenerator> (aOuter=aOuter@entry=0x0, aIID=..., aResult=aResult@entry=0x7ffe87d15200) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSModule.cpp:148 #24 0x00007fc7d7909ac2 in mozilla::xpcom::CreateInstanceImpl ( aID=<optimized out>, aIID=..., aResult=0x7ffe87d15200, aOuter=<optimized out>) at StaticComponents.cpp:10168 #25 0x00007fc7d791b5c6 in (anonymous namespace)::EntryWrapper::CreateInstance ( aOuter=0x0, aResult=<optimized out>, aIID=..., this=0x7ffe87d15280) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManager.cpp:224 #26 nsComponentManagerImpl::GetServiceLocked (this=0x7fc7e0658820, aLock=..., aEntry=..., aIID=..., aResult=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManager.cpp:1383 #27 0x00007fc7d791bae3 in nsComponentManagerImpl::GetServiceByContractID ( this=0x7fc7e0658820, aContractID=0x7fc7dcc59060 "@mozilla.org/security/random-generator;1", aIID=..., aResult=0x7ffe87d15300) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/Maybe.h:511 #28 0x00007fc7d791bc56 in nsComponentManagerImpl::GetServiceByContractID ( aResult=0x7ffe87d15300, aIID=..., aContractID=<optimized out>, this=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManager.cpp:1551 #29 CallGetService (aResult=0x7ffe87d15300, aIID=..., aContractID=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManagerUtils.cpp:61 #30 nsGetServiceByContractIDWithError::operator() ( this=this@entry=0x7ffe87d15330, aIID=..., aInstancePtr=aInstancePtr@entry=0x7ffe87d15300) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManagerUtils.cpp:253 #31 0x00007fc7d78b1a5a in nsCOMPtr_base::assign_from_gs_contractid_with_error ( this=this@entry=0x7ffe87d15328, aGS=..., aIID=...) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/base/nsCOMPtr.cpp:91 #32 0x00007fc7daea4c6b in nsCOMPtr<nsIRandomGenerator>::nsCOMPtr (aGS=..., this=0x7ffe87d15328) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/nsCOMPtr.h:425 #33 mozilla::RelativeTimeline::GetRandomTimelineSeed (this=0x7fc7d1165b80) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/toolkit/components/resistfingerprinting/RelativeTimeline.cpp:18 #34 0x00007fc7d9d0946d in mozilla::dom::Performance::Now (this=0x7fc7d11af000) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/dom/performance/Performance.cpp:97 #35 0x00007fc7d89090be in mozilla::dom::Document::HasRecentlyStartedForegroundLoads () at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/dom/base/Document.cpp:16142 #36 0x00007fc7d791f665 in mozilla::MainThreadIdlePeriod::GetIdlePeriodHint ( this=0x7fc7d292d7e0, aIdleDeadline=0x7ffe87d153f0) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/MainThreadIdlePeriod.cpp:62 #37 0x00007fc7d7920830 in mozilla::IdlePeriodState::GetLocalIdleDeadline ( aMutexToUnlock=..., aShuttingDown=@0x7ffe87d15437: false, this=0x7fc7d291be18) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/nsCOMPtr.h:841 #38 mozilla::IdlePeriodState::GetLocalIdleDeadline (this=0x7fc7d291be18, aShuttingDown=@0x7ffe87d15437: false, aMutexToUnlock=...) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/IdlePeriodState.cpp:100 #39 0x00007fc7d7924620 in mozilla::IdlePeriodState::GetIdleDeadlineInternal ( this=this@entry=0x7fc7d291be18, aIsPeek=aIsPeek@entry=true, aMutexToUnlock=...) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/IdlePeriodState.cpp:67 #40 0x00007fc7d7924a44 in mozilla::IdlePeriodState::PeekIdleDeadline ( aMutexToUnlock=..., this=0x7fc7d291be18) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/IdlePeriodState.h:91 #41 mozilla::PrioritizedEventQueue::HasReadyEvent (this=0x7fc7d291bdc0, aProofOfLock=...) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/PrioritizedEventQueue.cpp:274 #42 0x00007fc7d7938a01 in mozilla::ThreadEventQueue<mozilla::PrioritizedEventQueue>::HasPendingEvent (this=0x7fc7e064fc00) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/UniquePtr.h:308 #43 0x00007fc7d792764d in nsThread::HasPendingEvents (aResult=0x7ffe87d154d7, this=0x7fc7e0644b80) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/RefPtr.h:305 #44 nsThread::HasPendingEvents (this=0x7fc7e0644b80, aResult=0x7ffe87d154d7) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThread.cpp:909 #45 0x00007fc7d792dbc8 in hasPendingEvents (aThread=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThreadUtils.cpp:464 #46 NS_HasPendingEvents (aThread=<optimized out>, aThread@entry=0x7fc7e0644b80) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThreadUtils.cpp:464 #47 0x00007fc7d9e91e01 in nsBaseAppShell::OnProcessNextEvent ( this=0x7fc7d29fd7c0, thr=0x7fc7e0644b80, mayWait=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/widget/nsBaseAppShell.cpp:252 #48 0x00007fc7d792b82e in nsThread::ProcessNextEvent (aResult=0x7ffe87d15667, aMayWait=false, this=0x7fc7e0644b80) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThread.cpp:1118 #49 nsThread::ProcessNextEvent (this=0x7fc7e0644b80, aMayWait=<optimized out>, aResult=0x7ffe87d15667) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThread.cpp:1059 #50 0x00007fc7d792dc3c in NS_ProcessNextEvent (aThread=<optimized out>, aThread@entry=0x7fc7e0644b80, aMayWait=aMayWait@entry=false) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThreadUtils.cpp:486 #51 0x00007fc7d7e0c52a in mozilla::ipc::MessagePump::Run (this=0x7fc7e06a01f0, aDelegate=0x7ffe87d15820) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/glue/MessagePump.cpp:88 #52 0x00007fc7d7dd47c9 in MessageLoop::RunInternal (this=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/RefPtr.h:305 #53 MessageLoop::RunHandler (this=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:308 #54 MessageLoop::Run (this=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:290 #55 0x00007fc7d9e8ee4c in nsBaseAppShell::Run (this=0x7fc7d29fd7c0) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/widget/nsBaseAppShell.cpp:137 #56 0x00007fc7daf6db77 in XRE_RunAppShell () at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/toolkit/xre/nsEmbedFunctions.cpp:934 #57 0x00007fc7d7dd47c9 in MessageLoop::RunInternal (this=0x7ffe87d15820) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/RefPtr.h:305 #58 MessageLoop::RunHandler (this=0x7ffe87d15820) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:308 #59 MessageLoop::Run (this=this@entry=0x7ffe87d15820) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:290 #60 0x00007fc7daf6dfaa in XRE_InitChildProcess (aArgc=<optimized out>, aArgv=<optimized out>, aChildData=<optimized out>) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/toolkit/xre/nsEmbedFunctions.cpp:769 #61 0x000055b8fdf6fd4b in content_process_main (bootstrap=0x7fc7e0621640, argc=18, argv=0x7ffe87d15b98) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/browser/app/../../ipc/contentproc/plugin-container.cpp:56 #62 0x000055b8fdf6f5f6 in main (argc=19, argv=0x7ffe87d15b98, envp=0x7ffe87d15c38) at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/UniquePtr.h:308 SIGSYS suggests that the sandbox is overly aggressive. This is something that needs to be fixed inside firefox.
https://bugzilla.redhat.com/show_bug.cgi?id=1778559 suggests that https://github.com/void-linux/void-packages/blob/master/srcpkgs/firefox/patches/fix-sandbox-membarrier.patch may fix this.
*** Bug 1778559 has been marked as a duplicate of this bug. ***
Proposed as an F32 Beta blocker via https://bugzilla.redhat.com/show_bug.cgi?id=1778559#c11 .
Disabling multiple process to work around this ``` $ MOZ_FORCE_DISABLE_E10S=1 firefox ```
(In reply to Ritesh Khadgaray from comment #8) > Disabling multiple process to work around this Yes, sure, since it breaks inter-process communication
Not fixed in glibc 2.30.9000-22.fc32 (https://koji.fedoraproject.org/koji/buildinfo?buildID=1418753).
*** Bug 1778510 has been marked as a duplicate of this bug. ***
*** Bug 1779540 has been marked as a duplicate of this bug. ***
*** Bug 1780060 has been marked as a duplicate of this bug. ***
FWIW, a better fix other than disabling e10s is setting `MOZ_DISABLE_CONTENT_SANDBOX=1` instead. If you use Nightly, you can also set the security.sandbox.content.level preference to 0 in about:config. That doesn't work in release builds, but I think setting it to 1 may work there.
glibc-2.30.9000-24.fc32 contains the workaround I have submitted upstream. This does not fix the broken sandbox, although this particular crash is gone.
glibc-2.30.9000-24.fc32 makes icecat usable again.
As firefox does appear to work normally now (openQA tests of today's compose confirm this), I'm gonna close this as resolved as the bug reported here is basically "the bug that Firefox tabs keep crashing". If there's more work to be done it should probably go under a new bug report.
Firefox still has this issue in its sandbox and will likely break again in a future glibc update until this is fixed. I assume that we won't be doing any development in Fedora to fix this in Firefox, so tracking the issue upstream should be sufficient.
Yeah, :jld is working on re-working the sandbox on that upstream bug, IIUC.
“is working on” is a bit of a stretch — I have a general idea of what needs to happen, but it won't be a small project (and definitely not in time for the next glibc release). I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1603307 for that work, and I'm keeping https://bugzilla.mozilla.org/show_bug.cgi?id=1600574 scoped to this specific problem with dlopen. I'm assuming that glibc (the other upstream here) will eventually merge the patches in https://sourceware.org/ml/libc-alpha/2019-12/msg00175.html and I won't need to revisit using seccomp-bpf to prevent the signal blocking.
(In reply to Jed Davis from comment #20) > “is working on” is a bit of a stretch — I have a general idea of what needs > to happen, but it won't be a small project (and definitely not in time for > the next glibc release). I filed > https://bugzilla.mozilla.org/show_bug.cgi?id=1603307 for that work, and I'm > keeping https://bugzilla.mozilla.org/show_bug.cgi?id=1600574 scoped to this > specific problem with dlopen. > > I'm assuming that glibc (the other upstream here) will eventually merge the > patches in https://sourceware.org/ml/libc-alpha/2019-12/msg00175.html and I > won't need to revisit using seccomp-bpf to prevent the signal blocking. Correct, we will review and hopefully merge a pragmatic solution that undoes the signal blocking. Firefox still needs to be changed, and we want you to have the time to do that correctly.