Bug 1640158
Summary: | Chromium segfaults when attempting to handle GSSAPI requests to fedoraproject.org | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Stephen Gallagher <sgallagh> |
Component: | chromium | Assignee: | Tom "spot" Callaway <spotrh> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | 32 | CC: | akarshan.biswas, demiobenour, jhrozek, mhroncok, puiterwijk, rharwood, tpopela, yaneti |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | chromium-83.0.4103.116-3.fc32.x86_64 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-01-26 19:10:26 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Stephen Gallagher
2018-10-17 12:52:57 UTC
I should also note that Google Chrome closed-source (google-chrome-stable-70.0.3538.67-1.x86_64) does not experience this issue. Thanks for the link, Miro. It seems that Chromium may not be able to handle multiple Kerberos caches properly. Better backtrace - where we can see that it's crashing when initializing boringssl.. __GI___pthread_rwlock_wrlock (rwlock=0x0) at pthread_rwlock_wrlock.c:27 27 int result = __pthread_rwlock_wrlock_full (rwlock, NULL); Missing separate debuginfos, use: dnf debuginfo-install PackageKit-gtk3-module-1.1.11-1.fc29.x86_64 libXScrnSaver-1.2.3-2.fc29.x86_64 libXxf86vm-1.1.4-10.fc29.x86_64 libdbusmenu-devel-16.04.0-8.fc29.x86_64 nss-mdns-0.14.1-2.fc29.x86_64 pciutils-libs-3.6.2-1.fc29.x86_64 (gdb) bt #0 0x00007fffee2e9586 in __GI___pthread_rwlock_wrlock (rwlock=0x0) at pthread_rwlock_wrlock.c:27 #1 0x00007ffff67fa979 in CRYPTO_STATIC_MUTEX_lock_write () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libboringssl.so #2 0x00007ffff679aed3 in CRYPTO_get_ex_new_index () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libboringssl.so #3 0x00007fffc5a65de6 in init_openssl () at openssl.c:57 #4 0x00007fffc5a65de6 in init_openssl__aux () at openssl.c:49 #5 0x00007fffee2ec057 in __pthread_once_slow (once_control=0x7fffc5a69010 <init_openssl.once>, init_routine=0x7fffc5a65d90 <init_openssl__aux>) at pthread_once.c:116 #6 0x00007fffee2ec115 in __GI___pthread_once (once_control=once_control@entry=0x7fffc5a69010 <init_openssl.once>, init_routine=<optimized out>) at pthread_once.c:143 #7 0x00007fffeb147a49 in k5_once (once=once@entry=0x7fffc5a69010 <init_openssl.once>, fn=<optimized out>) at threads.c:562 #8 0x00007fffc5a6621e in setup (context=0x3f2ca5c3b200, fd=525, servername=0x3f2ca6338280 "id.fedoraproject.org", anchors=0x0, handle_out=0x3f2ca64642e0) at openssl.c:443 #9 0x00007fffebaaa24e in setup_tls (realm=<optimized out>, realm=<optimized out>, selstate=0x3f2ca77f7000, conn=0x3f2ca6464180, context=0x3f2ca5c3b200) at sendto_kdc.c:1225 #10 0x00007fffebaaa24e in service_https_write (context=0x3f2ca5c3b200, realm=<optimized out>, conn=0x3f2ca6464180, selstate=0x3f2ca77f7000) at sendto_kdc.c:1247 #11 0x00007fffebaaa571 in service_tcp_connect (context=0x3f2ca5c3b200, realm=0x7fffd8d16200, conn=0x3f2ca6464180, selstate=0x3f2ca77f7000) at sendto_kdc.c:1093 #12 0x00007fffebaa98e1 in service_dispatch (ssflags=<optimized out>, selstate=0x3f2ca77f7000, conn=0x3f2ca6464180, realm=0x7fffd8d16200, context=0x3f2ca5c3b200) at sendto_kdc.c:1067 #13 0x00007fffebaa98e1 in service_fds (context=0x3f2ca5c3b200, selstate=0x3f2ca77f7000, interval=<optimized out>, conns=<optimized out>, seltemp=0x3f2ca77f9004, realm=0x7fffd8d16200, msg_handler=0x7fffebaa8a70 <check_for_svc_unavailable>, msg_handler_data=0x7fffd8d16138, winner_out=0x7fffd8d16038) at sendto_kdc.c:1423 #14 0x00007fffebaaa9bd in k5_sendto (context=context@entry=0x3f2ca5c3b200, message=message@entry=0x7fffd8d161e0, realm=realm@entry=0x7fffd8d16200, servers=servers@entry=0x7fffd8d16150, strategy=UDP_FIRST, callback_info=callback_info@entry=0x0, reply=0x7fffd8d16160, remoteaddr=0x0, remoteaddrlen=0x0, server_used=0x7fffd8d1613c, msg_handler= 0x7fffebaa8a70 <check_for_svc_unavailable>, msg_handler_data=0x7fffd8d16138) at sendto_kdc.c:1518 #15 0x00007fffebaab305 in krb5_sendto_kdc (context=context@entry=0x3f2ca5c3b200, message=message@entry=0x7fffd8d161e0, realm=realm@entry=0x7fffd8d16200, reply_out=reply_out@entry=0x7fffd8d161f0, use_master=use_master@entry=0x7fffd8d161dc, no_udp=no_udp@entry=0) at sendto_kdc.c:507 #16 0x00007fffeba725dd in krb5_tkt_creds_get (context=context@entry=0x3f2ca5c3b200, ctx=0x3f2ca52e7420) at get_creds.c:1194 #17 0x00007fffeba7274d in krb5_get_credentials (context=context@entry=0x3f2ca5c3b200, options=0, ccache=0x3f2ca79191c0, in_creds=in_creds@entry=0x7fffd8d16360, out_creds=out_creds@entry=0x7fffd8d16358) at get_creds.c:1272 #18 0x00007fffebb3d08e in get_credentials (server=<optimized out>, out_creds=<synthetic pointer>, endtime=<optimized out>, now=<optimized out>, cred=0x3f2ca4e27f30, context=<optimized out>) at init_sec_context.c:170 #19 0x00007fffebb3d08e in kg_new_connection (output_token=0x7fffd8d168e0, exts=<optimized out>, context=<optimized out>, time_rec=<optimized out>, ret_flags=<optimized out>, actual_mech_type=<optimized out>, input_token=<optimized out>, input_chan_bindings=<optimized out>, time_req=<optimized out>, req_flags=32, mech_type=0x7fffebb5eac0 <krb5_gss_oid_array>, target_name=<optimized out>, context_handle=0x3f2ca64117f0, cred=0x3f2ca4e27f30, minor_status=<optimized out>) at init_sec_context.c:587 #20 0x00007fffebb3d08e in krb5_gss_init_sec_context_ext (minor_status=minor_status@entry=0x7fffd8d16bf4, claimant_cred_handle=0x3f2ca4e27f30, claimant_cred_handle@entry=0x0, context_handle=context_handle@entry=0x3f2ca64117f0, target_name=<optimized out>, mech_type=0x7fffebb5eac0 <krb5_gss_oid_array>, req_flags=<optimized out>, req_flags@entry=32, time_req=<optimized out>, input_chan_bindings=<optimized out>, input_token=<optimized out>, actual_mech_type=<optimized out>, output_token=<optimized out>, ret_flags=<optimized out>, time_rec=<optimized out>, exts=<optimized out>) at init_sec_context.c:987 #21 0x00007fffebb3db06 in krb5_gss_init_sec_context (minor_status=minor_status@entry=0x7fffd8d16bf4, claimant_cred_handle=claimant_cred_handle@entry=0x0, context_handle=context_handle@entry=0x3f2ca64117f0, target_name=<optimized out>, mech_type=<optimized out>, req_flags=req_flags@entry=32, time_req=4294967295, input_chan_bindings=0x0, input_token=0x0, actual_mech_type=0x3f2ca622b140, output_token=0x7fffd8d168e0, ret_flags=0x3f2ca622b130, time_rec=0x0) at init_sec_context.c:1089 #22 0x00007fffebb270bb in gss_init_sec_context (minor_status=minor_status@entry=0x7fffd8d16bf4, claimant_cred_handle=<optimized out>, context_handle=context_handle@entry=0x3f2ca622b108, target_name=target_name@entry=0x3f2ca7b17390, req_mech_type=<optimized out>, req_flags=32, time_req=4294967295, input_chan_bindings=0x0, input_token=0x0, actual_mech_type=0x3f2ca622b140, output_token=0x7fffd8d168e0, ret_flags=0x3f2ca622b130, time_rec=0x0) at g_init_sec_context.c:215 #23 0x00007fffebb4e6e0 in init_ctx_call_init (minor_status=minor_status@entry=0x7fffd8d16bf4, sc=0x3f2ca622b0e0, spcred=spcred@entry=0x0, target_name=target_name@entry=0x3f2ca7b17390, req_flags=req_flags@entry=0, time_req=time_req@entry=4294967295, mechtok_in=0x0, actual_mech=0x0, mechtok_out=0x7fffd8d168e0, ret_flags=0x0, time_rec=0x0, negState=0x7fffd8d168a8, send_token=0x7fffd8d168a4) at spnego_mech.c:922 #24 0x00007fffebb4ed9f in spnego_gss_init_sec_context (minor_status=minor_status@entry=0x7fffd8d16bf4, claimant_cred_handle=claimant_cred_handle@entry=0x0, context_handle=context_handle@entry=0x3f2ca7505e70, target_name=0x3f2ca7b17390, mech_type=<optimized out>, req_flags=req_flags@entry=0, time_req=4294967295, input_chan_bindings=0x0, input_token=0x7fffd8d16d90, actual_mech=0x0, output_token=0x7fffd8d16da0, ret_flags=0x0, time_rec=0x0) at spnego_mech.c:1089 #25 0x00007fffebb270bb in gss_init_sec_context (minor_status=0x7fffd8d16bf4, claimant_cred_handle=<optimized out>, context_handle=0x3f2ca7688128, target_name=0x3f2ca7bff240, req_mech_type=<optimized out>, req_flags=0, time_req=4294967295, input_chan_bindings=0x0, input_token=0x7fffd8d16d90, actual_mech_type=0x0, output_token=0x7fffd8d16da0, ret_flags=0x0, time_rec=0x0) at g_init_sec_context.c:215 #26 0x00007ffff6f023c5 in net::GSSAPISharedLibrary::init_sec_context(unsigned int*, gss_cred_id_struct*, gss_ctx_id_struct**, gss_name_struct*, gss_OID_desc_struct*, unsigned int, unsigned int, gss_channel_bindings_struct*, gss_buffer_desc_struct*, gss_OID_desc_struct**, gss_buffer_desc_struct*, unsigned int*, unsigned int*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #27 0x00007ffff6f02b89 in net::HttpAuthGSSAPI::GetNextSecurityToken(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, gss_buffer_desc_struct*, gss_buffer_desc_struct*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #28 0x00007ffff6f02773 in net::HttpAuthGSSAPI::GenerateAuthToken(net::AuthCredentials const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, base::OnceCallback<void (int)>) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #29 0x00007ffff6d495b7 in net::HttpAuthHandlerNegotiate::DoLoop(int) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #30 0x00007ffff6d49631 in net::HttpAuthHandlerNegotiate::OnIOComplete(int) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #31 0x00007ffff6f43c0c in net::HostResolverImpl::LegacyRequestImpl::LegacyApiCallback(int) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #32 0x00007ffff6f4434c in net::HostResolverImpl::Job::CompleteRequests(net::HostCache::Entry const&, base::TimeDelta, bool) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #33 0x00007ffff6f45003 in net::HostResolverImpl::Job::OnProcTaskComplete(base::TimeTicks, int, net::AddressList const&) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #34 0x00007ffff6f45b67 in net::HostResolverImpl::ProcTask::OnLookupComplete(net::AddressList const&, base::TimeTicks const&, unsigned int, int, int) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #35 0x00007ffff6f45704 in net::HostResolverImpl::ProcTask::OnLookupAttemptComplete(base::WeakPtr<net::HostResolverImpl::ProcTask>, base::TimeTicks const&, unsigned int, base::TickClock const*, net::AddressList const&, int, int) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #36 0x00007ffff6f45885 in base::internal::Invoker<base::internal::BindState<void (*)(base::WeakPtr<net::HostResolverImpl::ProcTask>, base::TimeTicks const&, unsigned int, base::TickClock const*, net::AddressList const&, int, int), base::WeakPtr<net::HostResolverImpl::ProcTask>, base::TimeTicks, unsigned int, base::TickClock const*>, void (net::AddressList const&, int, int)>::RunOnce(base::internal::BindStateBase*, net::AddressList const&, int, int) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #37 0x00007ffff6f45ff8 in base::internal::Invoker<base::internal::BindState<base::OnceCallback<void (net::AddressList const&, int, int)>, net::AddressList, int, int>, void ()>::RunOnce(base::internal::BindStateBase*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libnet.so #38 0x00007ffff7e96c10 in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libbase.so #39 0x00007ffff7ebf82f in base::MessageLoop::RunTask(base::PendingTask*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libbase.so #40 0x00007ffff7ebfdb2 in base::MessageLoop::DoWork() () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libbase.so #41 0x00007ffff7f83e69 in base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libbase.so #42 0x00007ffff7eead05 in base::RunLoop::Run() () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libbase.so #43 0x00007ffff56f7174 in content::BrowserProcessSubThread::IOThreadRun(base::RunLoop*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libcontent.so #44 0x00007ffff7f375ee in base::Thread::ThreadMain() () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libbase.so #45 0x00007ffff7f86edd in base::(anonymous namespace)::ThreadFunc(void*) () at /home/tpopela/dev/upstream/chromium/src/out/Release/./libbase.so #46 0x00007fffee2e458e in start_thread (arg=<optimized out>) at pthread_create.c:486 #47 0x00007fffed983513 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 And it's https://bugs.launchpad.net/oxide/+bug/1607411 - a clash between openssl and boringssl. The reason why it doesn't crash in the official Chrome build is that it doesn't use the component build, but it's statically linked. Hey spot, you've mentioned on other tickets that boringssl can't be built statically into chromium on Fedora. Do you happen to have a bug report (or log) for the failure? Do you mean the failure that occurs when I try to build it statically? Chromium really only has a "shared everything" or "static everything" switch, and we can't flip it to "static everything" for several reasons. It _might_ (and that's a "nothing is impossible, right" sort of "might) be possible to manually hack just boringssl into linking statically, but I tried and failed a long time ago and I doubt it has gotten easier. If someone was brave enough to try to make a patch to build boringssl static and everything else shared... I'd carry it. I was curious about the failure you mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1448649#c11 which I think is the final possibility you mention here. I tried to just tell chromium to build boringssl as a static component, but it did not work. This was a while ago and chromium has changed a lot since then, but I do not believe any of the mechanisms for the build tooling have changed to allow for a single component to be built static and the others shared. It's an all or nothing thing, and untangling it is not something I was successful at before. This message is a reminder that Fedora 29 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '29'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 29 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. (In reply to Tom "spot" Callaway from comment #8) > I tried to just tell chromium to build boringssl as a static component, but > it did not work. This was a while ago and chromium has changed a lot since > then, but I do not believe any of the mechanisms for the build tooling have > changed to allow for a single component to be built static and the others > shared. It's an all or nothing thing, and untangling it is not something I > was successful at before. Can we build chromium statically? If not, why? We use component build (shared) so that it is possible to: * minimize the footprint for chromium-headless * minimize the footprint for chrome-remote-desktop ... but most importantly * to make it possible for chromium-libs-media-freeworld to exist (and replace chromium-libs-media) (In reply to Tom "spot" Callaway from comment #11) > We use component build (shared) so that it is possible to: > > * minimize the footprint for chromium-headless > * minimize the footprint for chrome-remote-desktop > > ... but most importantly > > * to make it possible for chromium-libs-media-freeworld to exist (and > replace chromium-libs-media) What about using BoringSSL’s symbol renaming feature? We could automatically rename all BoringSSL symbols as well as all references to them. (In reply to Demi Marie Obenour from comment #12) > (In reply to Tom "spot" Callaway from comment #11) > > We use component build (shared) so that it is possible to: > > > > * minimize the footprint for chromium-headless > > * minimize the footprint for chrome-remote-desktop > > > > ... but most importantly > > > > * to make it possible for chromium-libs-media-freeworld to exist (and > > replace chromium-libs-media) > > What about using BoringSSL’s symbol renaming feature? We could > automatically rename all BoringSSL symbols as well as all references to them. Happy to look at a patch that did this. When I last tried (admittedly, a while ago), it spiraled into madness. (In reply to Tom "spot" Callaway from comment #13) > (In reply to Demi Marie Obenour from comment #12) > > What about using BoringSSL’s symbol renaming feature? We could > > automatically rename all BoringSSL symbols as well as all references to them. > > Happy to look at a patch that did this. When I last tried (admittedly, a > while ago), it spiraled into madness. Another option is to modify the binaries *after* they have already been built, which means that we do not need to mess with the Chromium build system. (In reply to Demi Marie Obenour from comment #14) > (In reply to Tom "spot" Callaway from comment #13) > > (In reply to Demi Marie Obenour from comment #12) > > > What about using BoringSSL’s symbol renaming feature? We could > > > automatically rename all BoringSSL symbols as well as all references to them. > > > > Happy to look at a patch that did this. When I last tried (admittedly, a > > while ago), it spiraled into madness. > > Another option is to modify the binaries *after* they have already been > built, which means that we do not need to mess with the Chromium build > system. That seems like a bad idea to me. This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle. Changing version to 32. I _think_ current versions of Chromium in Fedora should no longer have any issues here. Please test and let me know. Confirmed, chromium-83.0.4103.116-3.fc32.x86_64 does not experience this crash any longer. I will update the fedora-chromium-config package to enable GSSAPI for fedoraproject.org now. This is fixed. Can we close it? |