Does the corruption only happen during the glibc update, or does it persist after the update?

(In reply to Florian Weimer from comment #1)
> Does the corruption only happen during the glibc update, or does it persist
> after the update?

Just during the update (until you reboot or maybe fix it by switching VTs). After reboot, everything is fine.

I reproduced this on X11 as well, so this is not a wayland problem. Also, with X11 (with my limited 1 attempt), I was not able to get rid of the corruption by switching VTs.

(In reply to Kamil Páral from comment #2)
> (In reply to Florian Weimer from comment #1)
> > Does the corruption only happen during the glibc update, or does it persist
> > after the update?
> 
> Just during the update (until you reboot or maybe fix it by switching VTs).
> After reboot, everything is fine.
> 
> I reproduced this on X11 as well, so this is not a wayland problem. Also,
> with X11 (with my limited 1 attempt), I was not able to get rid of the
> corruption by switching VTs.

The display stack is not restarted while the glibc update is running, and it keeps using the previous glibc version/  So the glibc update itself can hardly be the cause of this issue.  It just happens to trigger it.

I expect that this is some sort of temporary interference from evelated CPU or PCI bus load.  Would you please reassign this bug to some package in the display stack?  Thanks.

So, this seems to affect all machines, bare metal and VMs, but in a different way:

* on our two machines with AMD GPUs, we see the screen corruption as described. Their graphics cards are:
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Turks PRO [Radeon HD 6570/7570/8550] [1002:6759]
00:01.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Kaveri [Radeon R7 Graphics] [1002:130f] (rev d4)

* on our machine with Intel GPU, the screen goes black for half a second during installation, but then is restored with no corruption. The graphics card is:
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0152] (rev 09)

* in VMs (kvm,spice,qxl) the screen goes black for a split second and is restored mostly intact, but there's a "VT-like" cursor blinking in the top left corner. Whatever you type on keyboard is printed on this "overlay" and immediately erased, and the cursor keeps moving forward with every typed letter (see screenshots).
00:02.0 VGA compatible controller [0300]: Red Hat, Inc. QXL paravirtual graphic card [1b36:0100] (rev 04)


Offline upgrade doesn't seem to be affected in any way, only when the update is running in a live session.

Created attachment 1194005 [details]
VT cursor visible in graphics (VM)

This is how VM screen looks like after glibc update. Notice the blinking cursor in top left corner.

Created attachment 1194010 [details]
VT cursor moving in graphics (VM)

And on this screen, you can see the cursor moved (it has a black background). It moves with every character typed on the keyboard.

Created attachment 1194035 [details]
KDE corruption in VM

This also affects KDE, but much worse. On AMD cards, the corruption is similar (and I was not able to get rid of it). In VMs, this switches you to a different VT (I guess - in any case, the desktop disappears). You can switch back, but you again see the blinking cursor, and this time, the text typed stays on the screen. That also has the "benefit" of showing your passwords, if you happen to type one. See the video.

This also affects the KDE graphical update manager, it also triggers this bug (i.e. this is not specific to DNF).

Also, this doesn't seem to be caused by the newer glibc. I have triggered the same problem when downgrading to an older glibc. You just need to reboot between your attempts, and (it seems) any glibc version change will trigger this.

> Would you please reassign this bug to some package in the
> display stack?  Thanks.

I have no idea where to assign this. It affects bare metal and VM, all graphics drivers (in a different way). So my blind guesses are systemd, mesa, kernel. Because there is some weirdness with framebuffer/switching VTs at the very same time that systemd prints it's reloading, trying systemd.

This is probably at least a conditional (if not full) violation of:
"The installed system must be able to download and install updates with the default console package manager. "
https://fedoraproject.org/wiki/Fedora_25_Alpha_Release_Criteria#Updates

Proposing for discussion.

To summarize, this is what we learned so far:
* GNOME + Radeon -> game stopper (display unreadable)
* GNOME + Intel -> cosmetic issue (screen blinking)
* GNOME + VM -> minor annoyance (screen blinking, VT cursor visible)
* KDE + Radeon -> game stopper (display unreadable)
* KDE + VM -> major annoyance (VT switch, typed text permanently visible)

KDE + Intel wasn't tested (yet), because we've hit an unrelated issue which prevented us from trying it.

Another symptom was found by nirik. Once the bug happens, regardless of desktop environment or hardware/VM, you can then use alt+left/right to switch VTs, *even when you're in graphics*. That is usually impossible, alt+left/right only works in text VTs, but not in graphics. But after glibc update glitch, you can do it. It almost seems like the glibc update switched you to a text VT mode, while the graphics is still displaying "in the background". Systemd or kernel folk will know more, hope this helps. Trivially reproducible in VMs.

meh, this is why we have offline update. ;) not sure if I'd call it a blocker.

(In reply to Kamil Páral from comment #9)
> To summarize, this is what we learned so far:
> * GNOME + Radeon -> game stopper (display unreadable)
> * GNOME + Intel -> cosmetic issue (screen blinking)
> * GNOME + VM -> minor annoyance (screen blinking, VT cursor visible)
> * KDE + Radeon -> game stopper (display unreadable)
> * KDE + VM -> major annoyance (VT switch, typed text permanently visible)
> 
> KDE + Intel wasn't tested (yet), because we've hit an unrelated issue which
> prevented us from trying it.

Please note that we have a very limited hardware to test with (2 Radeon cards, 1 Intel). We should not draw any definitive conclusions from that (applying it to all cards from that vendor). Also, we have no nvidia card to test with.

On my test box, with an Intel adapter (8086:0126), the system seems to flip to a real live VT for a second or two in glibc %post, then switches back to the desktop, with no graphical corruption evident.

I also have a system with an AMD adapter I'll test soonish. For the record, this can be tested from a live image, you don't need to install (though the live image will start behaving a bit strangely in other ways after glibc is updated).

So, glibc calls glibc_post_upgrade in post. In that it has: 

  /* Check if telinit is available and either SysVInit fifo,
     or upstart telinit.  */
  if (access ("/sbin/telinit", X_OK)
      || ((!!access ("/dev/initctl", F_OK))
          ^ !access ("/sbin/initctl", X_OK)))
    _exit (0);

  /* Check if we are not inside of some chroot, because we'd just
     timeout and leave /etc/initrunlvl.

     On more modern systems this test is not sufficient to detect
     if we're in a chroot.  */
  if (readlink ("/proc/1/exe", initpath, 256) <= 0 ||
      readlink ("/proc/1/root", initpath, 256) <= 0)
    _exit (0);

  /* Here's another well known way to detect chroot, at least on an
     ext and xfs filesystems and assuming nothing mounted on the chroot's
     root. */
  if (stat ("/", &statbuf) != 0
      || (statbuf.st_ino != 2
          && statbuf.st_ino != 128))
    _exit (0);

  if (check_elf ("/proc/1/exe"))
    verbose_exec (116, "/sbin/telinit", "/sbin/telinit", "u");

The 'telinit' man page says: 

"
       u, U
           Serialize state, reexecute daemon and deserialize state again. This is equivalent to systemctl daemon-reexec."

But this doesn't seem to be the case. 
Manually running: 

sudo /sbin/telinit u

here causes this exact bug. 

Doing: 

sudo /sbin/systemctl daemon-reexec 

doesn't. 

So, there's some systemd behavior change here. Either it needs to fix 'telinit u' or glibc needs to change to call 'systemctl daemon-reexec' or something.

We think Kevin's wrong there; it's not that direct systemctl call is OK and telinit -u is bad, it's simply that the bug only happens on the *first* re-exec. He didn't see the bug when he ran 'systemctl daemon-reexec' directly because he'd already done `telinit -u` on that boot.

Discussed today at Go/NoGo meeting. Voted as RejectedBlocker - this is obviously annoying if you hit it, but it's not exactly fatal (you can just reboot), it doesn't affect all systems (many show no significant symptoms), and it's probably workaroundable by doing the update from a VT (or offline, for GNOME).

I have tested "sudo systemctl daemon-reexec" on both GNOME and KDE in a VM and the symptoms are exactly the same as described in this bug. So this is the real issue, daemon-reexec causes this. It's trivial to reproduce. If you do, please note that the corruption/issues are only visible on your first invocation of daemon-reexec. If you want to trigger it again, reboot the system first.

(In reply to Adam Williamson from comment #15)
> We think Kevin's wrong there; it's not that direct systemctl call is OK and
> telinit -u is bad, it's simply that the bug only happens on the *first*
> re-exec. He didn't see the bug when he ran 'systemctl daemon-reexec'
> directly because he'd already done `telinit -u` on that boot.

Yeah, my bad. I didn't reboot between tests. It happens with daemon-reexec too.

This is fixed upstream [https://github.com/systemd/systemd/commit/bd64d82c1c0e3fe2a5f9b3dd9132d62834f50b2d, https://github.com/systemd/systemd/commit/158fbf7661912adf0f42c93155499119811dde82].

I'd mark this as duplicate of #1367766 (in fact the fix above was developed for #1367766), but now I checked the version numbers and see that #1367766 must have a different reason, since the patch which was the underlying cause and was reverted in the fix was not yet present in systemd-229 which is in F24.

Proposing as a Final blocker, just to make sure this doesn't slip through the cracks.

Discussed during the 2016-08-29 blocker review meeting: [1]

The decision to delay classification of this as a bug was made because there is a likely possibility that this bug will be fixed before Final.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2016-08-29/f25-blocker-review.2016-08-29-16.00.txt

For the record, I just experienced a system update in VM where the screen went completely black with only a blinking cursor in top left corner (so it looked like switching to a different TTY, but there was no login prompt there). That was in a *VM*, where I never experienced this issue before (the screen usually just flashed). This leads to me a conclusion that this is likely a race condition, and different drivers often behave consistently, but it doesn't have to be a rule. It's very possible that a screen corruption or some other behavior will happen even with drivers for whose it's not usual (in our testing), just because the timing will be slightly different.

When this bug occurs, the negative effects also include:
* Alt+left/right arrow switching your VTs, instead of say navigating back and forward in your browser/nautilus/etc.
* Ctrl+Alt+Del rebooting your computer immediately (no questions asked) instead of displaying a shutdown/reboot/cancel dialog (with possible warnings about unsaved work or other people logged in).

Today I've twice seen my display go berserk during live update. glibc was not in the package set, but glib2 and systemd was. The monitor started flashing, showing a sequence of actual picture and a black screen, in fast succession. Impossible to work with that. I could switch to VT, but only once of the two times. I assume this might be the same or a related problem. Intel graphics.

*** Bug 1362711 has been marked as a duplicate of this bug. ***

systemd-231-4.fc25 claims to have fixed this:

- Fix issue with daemon-reload messing up graphics (#1367766)

can anyone reproduce after that version of systemd is installed and the system rebooted?

Discussed during the 2016-09-12 blocker review meeting: [1]

The decision to classify this bug as an accepted blocker was made as the bug violates the following criteria:

"The installed system must be able to download and install updates with the default console package manager"

"All known bugs that can cause corruption of user data must be fixed or documented at Common F25 bugs"

This bug can cause the system to reboot before the update completes, breaking the system and risking data loss.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2016-09-12/f25-blocker-review.2016-09-12-16.01.txt

(In reply to Adam Williamson from comment #26)
> systemd-231-4.fc25 claims to have fixed this:
> 
> - Fix issue with daemon-reload messing up graphics (#1367766)

I tested this by downgrading to glibc-2.24-1.fc25.x86_64 and back to -3, in VM and bare metal (AMD graphics), on GNOME and KDE. No issues. Fixed.