Bug 1778366 - firefox: Sandbox crashes on openat syscall with SIGSYS blocked
Summary: firefox: Sandbox crashes on openat syscall with SIGSYS blocked
Alias: None
Product: Fedora
Classification: Fedora
Component: firefox
Version: rawhide
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Gecko Maintainer
QA Contact: Fedora Extras Quality Assurance
Whiteboard: openqa
: 1778510 1778559 1779540 1780060 (view as bug list)
Depends On:
Blocks: 1395758 F32BetaBlocker
TreeView+ depends on / blocked
Reported: 2019-11-30 10:36 UTC by Nicolas Mailhot
Modified: 2019-12-12 17:54 UTC (History)
34 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-12-07 01:35:39 UTC
Type: Bug

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Mozilla Foundation 1600574 0 P1 NEW Sandboxed processes crash due to glibc 2.31 dlopen() blocking SIGSYS while opening files 2019-12-12 17:52:08 UTC

Description Nicolas Mailhot 2019-11-30 10:36:43 UTC
Description of problem:
glibc 2.30.9000-21.fc32 breaks firefox, all tabs error-out

pipe error (101): Connection reset by peer: file /builddir/build/BUILD/firefox-71.0/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 358

dnf downgrade glibc-2.30.9000-20.fc32.x86_64 fixes the problem

(I destroyed a system attempting a downgrade to fc31 glibc)

Comment 1 Carlos O'Donell 2019-11-30 15:02:19 UTC

Thank you for filling this bug. Upstream recently improved the dynamic loader when it comes to handling dlopen() failures and rolling back state. It may be that Firefox is triggering a corner case we didn't consider. I was able to reproducer this when we were testing the changes before Rawhide deployment, but in my testing Rawhide was so broken already that it was to know if this was just a consequence of the other system brekage. The fact that you double-checked with a downgrade is really helpful.

We're looking into this.


I actually noticed this when I was testing the Rawhide upgrade with your scratch build, but since Rawhide is completely busted it wasn't clear to me if this was something that worked before or not.

The fact that a downgrade to -20 fixes things means that we might have a case we need to fix in the recent ld.so commit/rollback work.

This one is yours to triage, and it may be the same cause as the other issue, perhaps the tab is crashing with an assert because it's a distinct process.

Comment 2 Bruno Wolff III 2019-12-02 13:18:36 UTC
2.30.9000-22.fc32 is still broken. I notice the issue when using icecat. Downgrade fixed things for me.
I'm using rawhide exclusively and it works well for me. It might be that I uninstall packages blocking updates, temporarily, so that my system is more up to date.

Comment 3 Bruno Wolff III 2019-12-02 13:23:10 UTC
When doing downgrades, going against recent composes works well. I imagine it's a lot safer than trying to downgrade to a version from a previous release. For example in this case I downgrade versus the November 29th compose.
You could also use koji download-build to download recent rpms.

Comment 4 Florian Weimer 2019-12-02 13:39:53 UTC
The downgrade issue is tracked as bug 1652867. It requires upstream work, which is currently under review.

I was able to reproduce the crash with firefox-71.0-6.npgo.fc32.x86_64 and glibc-2.30.9000-21.fc32.x86_64. It looks like this:

Core was generated by `/usr/lib64/firefox/firefox -contentproc -childID 1 -isForBrowser -prefsLen 1 -p'.
Program terminated with signal SIGSYS, Bad system call.

#0  __GI___open64_nocancel (
    file=file@entry=0x7fc7d129d8e0 "/lib64/libsoftokn3.so", 
    at ../sysdeps/unix/sysv/linux/open64_nocancel.c:45
#1  0x00007fc7e0e7b8bf in open_verify (
    name=0x7fc7d129d8e0 "/lib64/libsoftokn3.so", fbp=fbp@entry=0x7ffe87d14470, 
    loader=<optimized out>, whatcode=whatcode@entry=0, 
    free_name=free_name@entry=true, mode=-1879048190, fd=-1) at dl-load.c:1548
#2  0x00007fc7e0e7d3d6 in _dl_map_object (loader=loader@entry=0x7fc7e0abaed0, 
    name=name@entry=0x7fc7d129d8a0 "/lib64/libsoftokn3.so", type=type@entry=2, 
    trace_mode=trace_mode@entry=0, mode=mode@entry=-1879048190, 
    nsid=<optimized out>) at dl-load.c:2164
#3  0x00007fc7e0e880a9 in dl_open_worker (a=a@entry=0x7ffe87d14a40)
    at dl-open.c:524
#4  0x00007fc7e0a2d258 in __GI__dl_catch_exception (
    operate=operate@entry=0x7fc7e0e88000 <dl_open_worker>, 
    args=args@entry=0x7ffe87d14a40) at dl-error-skeleton.c:208
#5  0x00007fc7e0e87c3b in _dl_open (
    file=0x7fc7d129d8a0 "/lib64/libsoftokn3.so", mode=-2147483646, 
    caller_dlopen=0x7fc7e08b899b <PR_LoadLibraryWithFlags+219>, nsid=-2, 
    argc=<optimized out>, argv=<optimized out>, env=0x7ffe87d15c38)
    at dl-open.c:855
#6  0x00007fc7e0e1939c in dlopen_doit (a=a@entry=0x7ffe87d14ce0) at dlopen.c:66
#7  0x00007fc7e0a2d258 in __GI__dl_catch_exception (
    operate=operate@entry=0x7fc7e0e19340 <dlopen_doit>, 
    args=args@entry=0x7ffe87d14ce0) at dl-error-skeleton.c:208
#8  0x00007fc7e0a2d323 in __GI__dl_catch_error (
    operate=operate@entry=0x7fc7e0e19340 <dlopen_doit>, 
    args=args@entry=0x7ffe87d14ce0) at dl-error-skeleton.c:227
#9  0x00007fc7e0e19b09 in _dlerror_run (
    operate=operate@entry=0x7fc7e0e19340 <dlopen_doit>, 
    args=args@entry=0x7ffe87d14ce0) at dlerror.c:170
#10 0x00007fc7e0e1942a in __dlopen (file=<optimized out>, mode=<optimized out>)
    at dlopen.c:87
#11 0x00007fc7e08b899b in PR_LoadLibraryWithFlags () from /lib64/libnspr4.so
#12 0x00007fc7d6d56b5c in loader_LoadLibInReferenceDir ()
   from /lib64/libnssutil3.so
#13 0x00007fc7d6d56bd1 in PORT_LoadLibraryFromOrigin ()
   from /lib64/libnssutil3.so
#14 0x00007fc7d6dafd32 in softoken_LoadDSO () from /lib64/libnss3.so
#15 0x00007fc7e08bf01e in PR_CallOnce () from /lib64/libnspr4.so
#16 0x00007fc7d6db6813 in secmod_LoadPKCS11Module () from /lib64/libnss3.so
#17 0x00007fc7d6dc316d in SECMOD_LoadModule () from /lib64/libnss3.so
#18 0x00007fc7d6d8fb7b in nss_Init () from /lib64/libnss3.so
#19 0x00007fc7d6d903fc in NSS_NoDB_Init () from /lib64/libnss3.so
#20 0x00007fc7dad98210 in EnsureNSSInitializedChromeOrContent ()
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSComponent.cpp:128
#21 EnsureNSSInitializedChromeOrContent ()
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSComponent.cpp:89
#22 0x00007fc7dad9d271 in mozilla::psm::Constructor<nsRandomGenerator, (nsresult (nsRandomGenerator::*)())0, (mozilla::psm::ProcessRestriction)1> (
    aResult=0x7ffe87d15200, aIID=..., aOuter=0x0)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSModule.cpp:148
#23 mozilla::psm::NSSConstructor<nsRandomGenerator> (aOuter=aOuter@entry=0x0, 
    aIID=..., aResult=aResult@entry=0x7ffe87d15200)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/security/manager/ssl/nsNSSModule.cpp:148
#24 0x00007fc7d7909ac2 in mozilla::xpcom::CreateInstanceImpl (
    aID=<optimized out>, aIID=..., aResult=0x7ffe87d15200, 
    aOuter=<optimized out>) at StaticComponents.cpp:10168
#25 0x00007fc7d791b5c6 in (anonymous namespace)::EntryWrapper::CreateInstance (
    aOuter=0x0, aResult=<optimized out>, aIID=..., this=0x7ffe87d15280)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManager.cpp:224
#26 nsComponentManagerImpl::GetServiceLocked (this=0x7fc7e0658820, aLock=..., 
    aEntry=..., aIID=..., aResult=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManager.cpp:1383
#27 0x00007fc7d791bae3 in nsComponentManagerImpl::GetServiceByContractID (
    aContractID=0x7fc7dcc59060 "@mozilla.org/security/random-generator;1", 
    aIID=..., aResult=0x7ffe87d15300)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/Maybe.h:511
#28 0x00007fc7d791bc56 in nsComponentManagerImpl::GetServiceByContractID (
    aResult=0x7ffe87d15300, aIID=..., aContractID=<optimized out>, 
    this=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManager.cpp:1551
#29 CallGetService (aResult=0x7ffe87d15300, aIID=..., 
    aContractID=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManagerUtils.cpp:61
#30 nsGetServiceByContractIDWithError::operator() (
    this=this@entry=0x7ffe87d15330, aIID=..., 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/components/nsComponentManagerUtils.cpp:253
#31 0x00007fc7d78b1a5a in nsCOMPtr_base::assign_from_gs_contractid_with_error (
    this=this@entry=0x7ffe87d15328, aGS=..., aIID=...)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/base/nsCOMPtr.cpp:91
#32 0x00007fc7daea4c6b in nsCOMPtr<nsIRandomGenerator>::nsCOMPtr (aGS=..., 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/nsCOMPtr.h:425
#33 mozilla::RelativeTimeline::GetRandomTimelineSeed (this=0x7fc7d1165b80)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/toolkit/components/resistfingerprinting/RelativeTimeline.cpp:18
#34 0x00007fc7d9d0946d in mozilla::dom::Performance::Now (this=0x7fc7d11af000)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/dom/performance/Performance.cpp:97
#35 0x00007fc7d89090be in mozilla::dom::Document::HasRecentlyStartedForegroundLoads ()
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/dom/base/Document.cpp:16142
#36 0x00007fc7d791f665 in mozilla::MainThreadIdlePeriod::GetIdlePeriodHint (
    this=0x7fc7d292d7e0, aIdleDeadline=0x7ffe87d153f0)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/MainThreadIdlePeriod.cpp:62
#37 0x00007fc7d7920830 in mozilla::IdlePeriodState::GetLocalIdleDeadline (
    aMutexToUnlock=..., aShuttingDown=@0x7ffe87d15437: false, 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/nsCOMPtr.h:841
#38 mozilla::IdlePeriodState::GetLocalIdleDeadline (this=0x7fc7d291be18, 
    aShuttingDown=@0x7ffe87d15437: false, aMutexToUnlock=...)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/IdlePeriodState.cpp:100
#39 0x00007fc7d7924620 in mozilla::IdlePeriodState::GetIdleDeadlineInternal (
    this=this@entry=0x7fc7d291be18, aIsPeek=aIsPeek@entry=true, 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/IdlePeriodState.cpp:67
#40 0x00007fc7d7924a44 in mozilla::IdlePeriodState::PeekIdleDeadline (
    aMutexToUnlock=..., this=0x7fc7d291be18)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/IdlePeriodState.h:91
#41 mozilla::PrioritizedEventQueue::HasReadyEvent (this=0x7fc7d291bdc0, 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/PrioritizedEventQueue.cpp:274
#42 0x00007fc7d7938a01 in mozilla::ThreadEventQueue<mozilla::PrioritizedEventQueue>::HasPendingEvent (this=0x7fc7e064fc00)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/UniquePtr.h:308
#43 0x00007fc7d792764d in nsThread::HasPendingEvents (aResult=0x7ffe87d154d7, 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/RefPtr.h:305
#44 nsThread::HasPendingEvents (this=0x7fc7e0644b80, aResult=0x7ffe87d154d7)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThread.cpp:909
#45 0x00007fc7d792dbc8 in hasPendingEvents (aThread=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThreadUtils.cpp:464
#46 NS_HasPendingEvents (aThread=<optimized out>, aThread@entry=0x7fc7e0644b80)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThreadUtils.cpp:464
#47 0x00007fc7d9e91e01 in nsBaseAppShell::OnProcessNextEvent (
    this=0x7fc7d29fd7c0, thr=0x7fc7e0644b80, mayWait=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/widget/nsBaseAppShell.cpp:252
#48 0x00007fc7d792b82e in nsThread::ProcessNextEvent (aResult=0x7ffe87d15667, 
    aMayWait=false, this=0x7fc7e0644b80)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThread.cpp:1118
#49 nsThread::ProcessNextEvent (this=0x7fc7e0644b80, aMayWait=<optimized out>, 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThread.cpp:1059
#50 0x00007fc7d792dc3c in NS_ProcessNextEvent (aThread=<optimized out>, 
    aThread@entry=0x7fc7e0644b80, aMayWait=aMayWait@entry=false)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/xpcom/threads/nsThreadUtils.cpp:486
#51 0x00007fc7d7e0c52a in mozilla::ipc::MessagePump::Run (this=0x7fc7e06a01f0, 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/glue/MessagePump.cpp:88
#52 0x00007fc7d7dd47c9 in MessageLoop::RunInternal (this=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/RefPtr.h:305
#53 MessageLoop::RunHandler (this=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:308
#54 MessageLoop::Run (this=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:290
#55 0x00007fc7d9e8ee4c in nsBaseAppShell::Run (this=0x7fc7d29fd7c0)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/widget/nsBaseAppShell.cpp:137
#56 0x00007fc7daf6db77 in XRE_RunAppShell ()
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/toolkit/xre/nsEmbedFunctions.cpp:934
#57 0x00007fc7d7dd47c9 in MessageLoop::RunInternal (this=0x7ffe87d15820)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/RefPtr.h:305
#58 MessageLoop::RunHandler (this=0x7ffe87d15820)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:308
#59 MessageLoop::Run (this=this@entry=0x7ffe87d15820)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/ipc/chromium/src/base/message_loop.cc:290
#60 0x00007fc7daf6dfaa in XRE_InitChildProcess (aArgc=<optimized out>, 
    aArgv=<optimized out>, aChildData=<optimized out>)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/toolkit/xre/nsEmbedFunctions.cpp:769
#61 0x000055b8fdf6fd4b in content_process_main (bootstrap=0x7fc7e0621640, 
    argc=18, argv=0x7ffe87d15b98)
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/browser/app/../../ipc/contentproc/plugin-container.cpp:56
#62 0x000055b8fdf6f5f6 in main (argc=19, argv=0x7ffe87d15b98, 
    at /usr/src/debug/firefox-71.0-6.npgo.fc32.x86_64/objdir/dist/include/mozilla/UniquePtr.h:308

SIGSYS suggests that the sandbox is overly aggressive. This is something that needs to be fixed inside firefox.

Comment 6 Adam Williamson 2019-12-02 18:11:50 UTC
*** Bug 1778559 has been marked as a duplicate of this bug. ***

Comment 7 Adam Williamson 2019-12-02 18:12:38 UTC
Proposed as an F32 Beta blocker via https://bugzilla.redhat.com/show_bug.cgi?id=1778559#c11 .

Comment 8 Ritesh Khadgaray 2019-12-02 20:53:09 UTC
Disabling multiple process to work around this 

$ MOZ_FORCE_DISABLE_E10S=1 firefox

Comment 9 Nicolas Mailhot 2019-12-02 21:42:24 UTC
(In reply to Ritesh Khadgaray from comment #8)
> Disabling multiple process to work around this 

Yes, sure, since it breaks inter-process communication

Comment 10 Anders Kaseorg 2019-12-03 02:41:39 UTC
Not fixed in glibc 2.30.9000-22.fc32 (https://koji.fedoraproject.org/koji/buildinfo?buildID=1418753).

Comment 11 Florian Weimer 2019-12-03 12:44:34 UTC
*** Bug 1778510 has been marked as a duplicate of this bug. ***

Comment 12 Martin Stransky 2019-12-04 08:08:20 UTC
*** Bug 1779540 has been marked as a duplicate of this bug. ***

Comment 13 Martin Stransky 2019-12-05 11:39:21 UTC
*** Bug 1780060 has been marked as a duplicate of this bug. ***

Comment 14 Emilio Cobos Álvarez (:emilio) 2019-12-05 11:58:56 UTC
FWIW, a better fix other than disabling e10s is setting `MOZ_DISABLE_CONTENT_SANDBOX=1` instead.

If you use Nightly, you can also set the security.sandbox.content.level preference to 0 in about:config. That doesn't work in release builds, but I think setting it to 1 may work there.

Comment 15 Florian Weimer 2019-12-05 22:04:58 UTC
glibc-2.30.9000-24.fc32 contains the workaround I have submitted upstream. This does not fix the broken sandbox, although this particular crash is gone.

Comment 16 Bruno Wolff III 2019-12-06 16:46:33 UTC
glibc-2.30.9000-24.fc32 makes icecat usable again.

Comment 17 Adam Williamson 2019-12-07 01:35:39 UTC
As firefox does appear to work normally now (openQA tests of today's compose confirm this), I'm gonna close this as resolved as the bug reported here is basically "the bug that Firefox tabs keep crashing". If there's more work to be done it should probably go under a new bug report.

Comment 18 Florian Weimer 2019-12-09 12:16:24 UTC
Firefox still has this issue in its sandbox and will likely break again in a future glibc update until this is fixed. I assume that we won't be doing any development in Fedora to fix this in Firefox, so tracking the issue upstream should be sufficient.

Comment 19 Emilio Cobos Álvarez (:emilio) 2019-12-09 12:18:20 UTC
Yeah, :jld is working on re-working the sandbox on that upstream bug, IIUC.

Comment 20 Jed Davis 2019-12-12 02:07:55 UTC
“is working on” is a bit of a stretch — I have a general idea of what needs to happen, but it won't be a small project (and definitely not in time for the next glibc release).  I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1603307 for that work, and I'm keeping https://bugzilla.mozilla.org/show_bug.cgi?id=1600574 scoped to this specific problem with dlopen.  

I'm assuming that glibc (the other upstream here) will eventually merge the patches in https://sourceware.org/ml/libc-alpha/2019-12/msg00175.html and I won't need to revisit using seccomp-bpf to prevent the signal blocking.

Comment 21 Carlos O'Donell 2019-12-12 17:54:50 UTC
(In reply to Jed Davis from comment #20)
> “is working on” is a bit of a stretch — I have a general idea of what needs
> to happen, but it won't be a small project (and definitely not in time for
> the next glibc release).  I filed
> https://bugzilla.mozilla.org/show_bug.cgi?id=1603307 for that work, and I'm
> keeping https://bugzilla.mozilla.org/show_bug.cgi?id=1600574 scoped to this
> specific problem with dlopen.  
> I'm assuming that glibc (the other upstream here) will eventually merge the
> patches in https://sourceware.org/ml/libc-alpha/2019-12/msg00175.html and I
> won't need to revisit using seccomp-bpf to prevent the signal blocking.

Correct, we will review and hopefully merge a pragmatic solution that undoes the signal blocking.

Firefox still needs to be changed, and we want you to have the time to do that correctly.

Note You need to log in before you can comment on or make changes to this bug.