Fedora Account System
Red Hat Associate
Red Hat Customer
Multiple people - myself, Kevin Fenzi, Carlos Garnacho and Igor Raits - noted that, after a recent Rawhide upgrade, our systems boot to GNOME-on-X11 not GNOME-on-Wayland. The journal generally shows some kind of memory allocation-ish error - I've seen various, e.g.: Jun 21 17:54:24 xps13k.happyassassin.net gnome-shell[1663]: g_object_new_with_properties: assertion 'G_TYPE_IS_OBJECT (object_type)' failed Jun 21 17:54:24 xps13k.happyassassin.net audit[1663]: ANOM_ABEND auid=4294967295 uid=42 gid=42 ses=4294967295 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 pid=1663 comm="gnome-shell" exe="/usr/bin/gnome-shell" sig=11 res=1 Jun 21 17:54:24 xps13k.happyassassin.net kernel: gnome-shell[1663]: segfault at 20 ip 00007f77ae6268a3 sp 00007ffe35226280 error 6 in libmutter-clutter-8.so.0.0.0[7f77ae5e7000+9a000] Jun 21 17:54:24 xps13k.happyassassin.net kernel: Code: 00 f3 0f 11 44 24 0c 0f 86 a2 00 00 00 e8 65 b3 fc ff 31 f6 48 89 c7 31 c0 e8 b9 8d fc ff be 68 00 00 00 48 8d 3d 5d a8 0a 00 <4c> 89 68 20 49 89 c4 48 89 68 28 e8 5d 88 fc ff 48 89 c5 4c 89 e6 Jun 21 17:59:06 xps13k.happyassassin.net kernel: gnome-shell[2230]: segfault at 7f4df2377035 ip 00007f4df2377035 sp 00007ffc362ae318 error 15 in iris_dri.so[7f4df2179000+497000] Jun 21 17:59:06 xps13k.happyassassin.net kernel: Code: 61 63 68 00 6c 7a 64 00 66 62 68 00 66 62 6c 00 63 62 69 74 00 61 64 64 63 00 73 75 62 62 00 73 61 64 32 00 73 61 64 61 32 00 <64> 70 34 00 64 70 33 00 64 70 32 00 70 6c 6e 00 6d 61 64 6d 00 6e Jun 21 18:00:34 xps13k.happyassassin.net org.gnome.Shell.desktop[1754]: double free or corruption (out) Jun 21 18:08:27 xps13k.happyassassin.net org.gnome.Shell.desktop[1391]: double free or corruption (out) Jun 22 10:17:51 xps13k.happyassassin.net org.gnome.Shell.desktop[1443]: malloc(): unaligned tcache chunk detected Jun 22 10:36:08 xps13k.happyassassin.net org.gnome.Shell.desktop[1387]: double free or corruption (out) Jun 22 11:27:28 xps13k.happyassassin.net org.gnome.Shell.desktop[1421]: double free or corruption (out) etc. etc. Then gnome-shell crashes; the backtrace isn't always exactly the same, but always runs through the iris dri driver. As discovered by Carlos, setting the environment variable MESA_LOADER_DRIVER_OVERRIDE=i965 (we're setting it in /etc/environment) works around the problem; with that set, we get GNOME-on-Wayland again. So the issue definitely seems to be to do with the iris driver. We each tried downgrading various packages that seemed possibly related, including the kernel, mesa, glib2, systemd, gdm, gnome-shell mutter and clutter. None of these downgrades helped. The only realistic remaining suspect is glibc. I noticed that in my last update it went from glibc-2.33.9000- 2.fc35.x86_64 to glibc-2.33.9000-23.fc35.x86_64 , which is a significant jump. Kevin confirmed the same for him. Unfortunately, downgrading to -2 is both tricky (I broke my system badly enough I needed a live USB to recover) and won't prove much because nearly every package that got upgraded at the same time also has to be downgraded, apparently the ABI changed between those two versions. It does seem very likely that glibc is the cause, though. There's just nothing else plausibly in the affected path that we haven't tried downgrading yet. My hardware is a Dell XPS 13 "Kaby Lake" (9360). Graphics adapter is: 00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 620 [8086:5916] (rev 02)
(In reply to Adam Williamson from comment #0) > Multiple people - myself, Kevin Fenzi, Carlos Garnacho and Igor Raits - > noted that, after a recent Rawhide upgrade, our systems boot to GNOME-on-X11 > not GNOME-on-Wayland. The journal generally shows some kind of memory > allocation-ish error - I've seen various, e.g.: Sorry to hear that! DJ Delorie can reproduce this with matching laptop hardware, thanks for the report. We did do extensive side-tag testing, and VM testing before merging into Rawhide. One of the reasons for the jump from -2 to -23 is that we were waiting for the upstream merge of libpthread functionality into libc. The new glibc 2.33.9000 is effectively glibc 2.34 and it has merged all the threading code into the core library to improve in-place upgrades, and improve thread support. > Jun 21 17:54:24 xps13k.happyassassin.net kernel: gnome-shell[1663]: segfault > at 20 ip 00007f77ae6268a3 sp 00007ffe35226280 error 6 in > libmutter-clutter-8.so.0.0.0[7f77ae5e7000+9a000] This is mutter faulting. > Jun 21 17:59:06 xps13k.happyassassin.net kernel: gnome-shell[2230]: segfault > at 7f4df2377035 ip 00007f4df2377035 sp 00007ffc362ae318 error 15 in > iris_dri.so[7f4df2179000+497000] Possibly because the iris dri driver is faulting. > Jun 21 18:00:34 xps13k.happyassassin.net org.gnome.Shell.desktop[1754]: > double free or corruption (out) > > Jun 21 18:08:27 xps13k.happyassassin.net org.gnome.Shell.desktop[1391]: > double free or corruption (out) > > Jun 22 10:17:51 xps13k.happyassassin.net org.gnome.Shell.desktop[1443]: > malloc(): unaligned tcache chunk detected This is tcache corruption, the chunk is not aligned and that means you generally passed an invalid pointer to free. > Jun 22 10:36:08 xps13k.happyassassin.net org.gnome.Shell.desktop[1387]: > double free or corruption (out) > > Jun 22 11:27:28 xps13k.happyassassin.net org.gnome.Shell.desktop[1421]: > double free or corruption (out) > > etc. etc. Then gnome-shell crashes; the backtrace isn't always exactly the > same, but always runs through the iris dri driver. OK. > As discovered by Carlos, setting the environment variable > MESA_LOADER_DRIVER_OVERRIDE=i965 (we're setting it in /etc/environment) > works around the problem; with that set, we get GNOME-on-Wayland again. So > the issue definitely seems to be to do with the iris driver. OK. > The only realistic remaining suspect is glibc. I noticed that in my last > update it went from glibc-2.33.9000- > 2.fc35.x86_64 to glibc-2.33.9000-23.fc35.x86_64 , which is a significant > jump. Kevin confirmed the same for him. Unfortunately, downgrading to -2 is > both tricky (I broke my system badly enough I needed a live USB to recover) > and won't prove much because nearly every package that got upgraded at the > same time also has to be downgraded, apparently the ABI changed between > those two versions. The ABI did change. > It does seem very likely that glibc is the cause, though. There's just > nothing else plausibly in the affected path that we haven't tried > downgrading yet. We need to look into this. Any help on your end is appreciated.
I was able to capture a traceback that involves the iris driver and a malloc fault. As usual, all the interesting values are <optimized out> :-P
Created attachment 1793315 [details] stacktrace for iris->malloc error
Happy to help if I can, but if DJ can reproduce it I don't think there's anything I can do that he can't! I'm no expert at debugging at this level.
Is this issue visible prior to login? That is, is it expected to reproduce without user interaction? That would mean I can provision a lab machine to see if it happens there as well. Thanks.
Created attachment 1793471 [details] Valgrind output of GNOME Shell Here's the output of a valgrind run of GNOME Shell, taken with recent (~yesterday) mesa/glibc/mutter/etc. Hopefully the invalid reads/writes help spot where it's breaking.
(In reply to Florian Weimer from comment #5) > Is this issue visible prior to login? That is, is it expected to reproduce > without user interaction? That would mean I can provision a lab machine to > see if it happens there as well. Thanks. It is, it might be sneaky to see (GDM falls back to X11 after it failed trying Wayland), but it will leave traces in journalctl/coredumpctl.
(In reply to Carlos Garnacho from comment #6) > Created attachment 1793471 [details] > Valgrind output of GNOME Shell > > Here's the output of a valgrind run of GNOME Shell, taken with recent > (~yesterday) mesa/glibc/mutter/etc. Hopefully the invalid reads/writes help > spot where it's breaking. Thanks, much appreciated! I do not think there is much we can do with this data on the glibc side. The logs appear to confirm that there is a use-after-free bug in application code.
So here's something interesting: other things are crashing since my upgrade. Looking at coredumpctl list, aside from gnome-shell, I have a bunch of coredumps from boltd. I also just tried to run a VM to test something and qemu crashed twice. The qemu crash is too big, but the boltd one got captured. I'll try and trace it. Neither boltd nor qemu was crashing before the last update. qemu was updated in that same transaction (from 6.0.0-5.fc35 to 6.0.0-6.fc35), but bolt was not.
Created attachment 1793623 [details] bolt traceback #1 Here's one of the bolt tracebacks, I have lots more if you like :D They seem all similar to me, and it's more memory allocation stuff: Thread 1 (Thread 0x7f25bba83cc0 (LWP 1121)): #0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:74 No locals. #1 0x00007f25bc521688 in g_strdup (str=0x2a <error: Cannot access memory at address 0x2a>) at ../glib/gstrfuncs.c:363 new_str = <optimized out> length = <optimized out> #2 0x00007f25bc2b3685 in value_collect_string (value=0x7ffe79713bc0, n_collect_values=<optimized out>, collect_values=<optimized out>, collect_flags=<optimized out>) at ../gobject/gvaluetypes.c:293 No locals. etc. Again, this is a thing (boltd) that wasn't updated, indeed hasn't been touched since January. It just started crashing after the update that included the new glibc.
As another data point in the same direction, building mesa from git (main or 21.1.3, doesn't matter) in my dev tree along with Mutter and Xwayland does not seem to result in memory corruption, nor crashes. At the same time, running /usr/bin/mutter with LD_LIBRARY_PATH pointing to that prefix still triggers both. I did ensure that the built mesa library was used in both cases, and specifically the iris driver paths that seemed to cause the memory corruption.
(In reply to Carlos Garnacho from comment #11) > As another data point in the same direction, building mesa from git (main or > 21.1.3, doesn't matter) in my dev tree along with Mutter and Xwayland does > not seem to result in memory corruption, nor crashes. At the same time, > running /usr/bin/mutter with LD_LIBRARY_PATH pointing to that prefix still > triggers both. > > I did ensure that the built mesa library was used in both cases, and > specifically the iris driver paths that seemed to cause the memory > corruption. Thanks for working on this. Both DJ (T480) and myself (T590) can reproduce the crashes in gnome-sell on F35. I admit this really looks like a glibc bug or toolchain issue. I can see crashes with mesa 20 on f33 and mesa 21 on f34 in the same way with the new glibc from f35. The crash is always in the first and main thread in my case, but DJ sees some variation in the crash locations. All threads have their own malloc arena when I see a crash, so they don't actually conflict in the chunks they would be using. We double checked that the "single threaded" optimizations weren't being used in this case (eliding locks). This is really quite odd.
I'm having problems running gnome-shell under gdb, so if you have more concrete magic instructions for debugging that would help.
(In reply to Carlos Garnacho from comment #6) > Created attachment 1793471 [details] > Valgrind output of GNOME Shell > > Here's the output of a valgrind run of GNOME Shell, taken with recent > (~yesterday) mesa/glibc/mutter/etc. Hopefully the invalid reads/writes help > spot where it's breaking. Can you provide the exact details for doing this? I had problems doing this myself.
(In reply to Carlos O'Donell from comment #14) > (In reply to Carlos Garnacho from comment #6) > > Created attachment 1793471 [details] > > Valgrind output of GNOME Shell > > > > Here's the output of a valgrind run of GNOME Shell, taken with recent > > (~yesterday) mesa/glibc/mutter/etc. Hopefully the invalid reads/writes help > > spot where it's breaking. > > Can you provide the exact details for doing this? I had problems doing this > myself. I just booted, without logging in at GDM went to a tty, and launched: valgrind --leak-check=full --num-callers=30 gnome-shell --wayland --display-server Perhaps doing s/gnome-shell/mutter/ in that command helps, since the latter is enough to reproduce the crashes and it's simpler in terms of dependencies.
Oh, but now that I remember I had to do: setcap -r /usr/bin/gnome-shell The CAP_SYS_NICE there impedes tracing the process AFAIR.
(In reply to Carlos Garnacho from comment #15) > (In reply to Carlos O'Donell from comment #14) > > (In reply to Carlos Garnacho from comment #6) > > > Created attachment 1793471 [details] > > > Valgrind output of GNOME Shell > > > > > > Here's the output of a valgrind run of GNOME Shell, taken with recent > > > (~yesterday) mesa/glibc/mutter/etc. Hopefully the invalid reads/writes help > > > spot where it's breaking. > > > > Can you provide the exact details for doing this? I had problems doing this > > myself. > > I just booted, without logging in at GDM went to a tty, and launched: > > valgrind --leak-check=full --num-callers=30 gnome-shell --wayland > --display-server > > Perhaps doing s/gnome-shell/mutter/ in that command helps, since the latter > is enough to reproduce the crashes and it's simpler in terms of dependencies. Thanks, this looks really helpful. I'm bisecting the glibc history now.
git bisect identified this commit as the source of the regression (also double-checked manually that is the right one): commit 572bd547d57a39b6cf0ea072545dc4048921f4c3 Author: Szabolcs Nagy <szabolcs.nagy> Date: Thu Dec 31 13:59:38 2020 +0000 elf: Fix DTV gap reuse logic [BZ #27135] For some reason only dlopen failure caused dtv gaps to be reused. It is possible that the intent was to never reuse modids for a different module, but after dlopen failure all gaps are reused not just the ones caused by the unfinished dlopened. So the code has to handle reused modids already which seems to work, however the data races at thread creation and tls access (see bug 19329 and bug 27111) may be more severe if slots are reused so this is scheduled after those fixes. I think fixing the races are not simpler if reuse is disallowed and reuse has other benefits, so set GL(dl_tls_dtv_gaps) whenever entries are removed from the middle of the slotinfo list. The value does not have to be correct: incorrect true value causes the next modid query to do a slotinfo walk, incorrect false will leave gaps and new entries are added at the end. Fixes bug 27135. Reviewed-by: Adhemerval Zanella <adhemerval.zanella>
I'm scratch building a new Fedora rawhide glibc with the DTV gap reuse reverted. The failure case is interesting, you have to have a dlclose to create a gap, followed by subsequent dlopen of an object with more deps than the gap size, and they must use TLS. I think the rarity of that specific failure mode meant that in our VM testing we didn't see it, but it certainly exists as a failure mode, but other threads can fill the gap 1-at-a-time and so you might not hit the failure. Lets see if this fixes the issue.
Revert fixes the issue completely. I have a working Wayland gnome-shell system running F35 again. I'm testing qemu next.
Confirmed f35 qemu works (after working around the selinux bug 1964317), and was able to boot f34 live ISO. Firefox also works, with various video codecs operating as expected and audio (pipewire) works. Audacity also works, recording and playback. Pipewire seems fully operational also for audio and processing. I think this means we caught the bug and a clean revert gets us operational again. We'll discuss the proper fix upstream.
Thanks a lot Carlos! I'm assuming you'll do a build for Rawhide with the revert? Would be good to be able to use qemu again. :D
(In reply to Adam Williamson from comment #22) > Thanks a lot Carlos! I'm assuming you'll do a build for Rawhide with the > revert? Would be good to be able to use qemu again. :D Fix pushed. Rawhide build in progress: https://koji.fedoraproject.org/koji/taskinfo?taskID=70753610
(In reply to Carlos O'Donell from comment #23) > (In reply to Adam Williamson from comment #22) > > Thanks a lot Carlos! I'm assuming you'll do a build for Rawhide with the > > revert? Would be good to be able to use qemu again. :D > > Fix pushed. Rawhide build in progress: > https://koji.fedoraproject.org/koji/taskinfo?taskID=70753610 All tests clean on the Rawhide build for all arches. No regressions. This looks good. Thank you to everyone here for your patience while we worked out the regression.
Thanks. Yes, seems to be working here too, though I was a bit confused by running into https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/4417 at first...