Bug 1394755

Summary: screen recording freezes GNOME under Wayland when gstreamer cache is updated
Product: [Fedora] Fedora Reporter: Kamil Páral <kparal>
Component: gnome-shellAssignee: Owen Taylor <otaylor>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 28CC: ahsaleem, angrypenguinpoland, bdpepple, buhrt, fmuellner, fspolti, goreyshi, ITProf13, jan.public, jkurik, jsedlak, luan.cestari, martial.pageau, nphilipp, otaylor, pablodav, pdavydov108, pirado, rcyriac, rds, simon, tobias47n9e, uraeus, wtaymans, zing
Target Milestone: ---Keywords: CommonBugs, Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: https://fedoraproject.org/wiki/Common_F25_bugs#wayland-frozen-login-after-upgrade
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-28 23:52:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1277927    
Attachments:
Description Flags
journal after upgrade while trying to log in
none
rpm -qa after upgrade
none
broken registry.x86_64.bin
none
fixed registry.x86_64.bin
none
gst-inspect-1.0 output none

Description Kamil Páral 2016-11-14 12:09:01 UTC
Description of problem:
I don't have good amount of information to debug this issue, but I'm filing it anyway in order to document it and hopefully we'll discover more helpful details with time.

Sometimes when you upgrade from F24 to F25, you can no longer log in into GNOME. The display stays gray, cursor is frozen, and no keyboard shortcuts work (not even Ctrl+Alt+Fx). This is a permanent error, even if you hard reboot, you still can't log in. But it occurs only for a Wayland session (the default for F25). If you switch to X11 session, you can log in. And what is even more interesting, once you do this, the whole problem goes away and you can log in to Wayland session as well, any time.

I've seen this on my laptop a long time ago when I upgraded to pre-release F25, but didn't debug it. Now we've seen this a few days back on Jan Sedlak's laptop (cced). The best thing I managed to find in the journal was this:

lis 11 15:06:45 dhcp-28-124.brq.redhat.com org.gnome.Shell.desktop[2294]: ERROR: Caught a segmentation fault while loading plugin file:
lis 11 15:06:45 dhcp-28-124.brq.redhat.com org.gnome.Shell.desktop[2294]: /usr/lib64/gstreamer-1.0/libgstpythonplugin.cpython-35m-x86_64-linux-gnu.so
lis 11 15:06:45 dhcp-28-124.brq.redhat.com org.gnome.Shell.desktop[2294]: Please either:
lis 11 15:06:45 dhcp-28-124.brq.redhat.com org.gnome.Shell.desktop[2294]: - remove it and restart.
lis 11 15:06:45 dhcp-28-124.brq.redhat.com org.gnome.Shell.desktop[2294]: - run with --gst-disable-segtrap --gst-disable-registry-fork and debug.


So it seems gnome-shell crashed, because a gstreamer plugin crashed. (But it is very interesting that this only happens until you log into X11, and then stops happening - maybe related to user configuration files?). That file comes from python3-gstreamer1 RPM, and was most probably present because of pitivi.

So, we've seen this on 2 bare metal machines, with real-world content. When I tried to reproduce this in VMs, I never managed to reproduce the crash. Even if I install pitivi + python3-gstreamer1, there's no problem logging in after upgrade. I don't know whether this only affects bare metal machines (driver issues) or it is somehow related to existing user configuration files.

The current workaround is:
a) In the session picker in login screen pick X11 and log in once. Then it should be possible to log into Wayland just OK.
b) If your user doesn't have a password set, so can't access to session picker. Switch to different TTY (Ctrl+Alt+F2) and set a password ("passwd username"). Switch back to login screen (Ctrl+Alt+F1). Or try c).
c) If you have autologin set, the easiest way is probably to edit kernel command line in grub ('e' to edit) and add 'nomodeset' to the 'linux' line, then boot. That should force basic graphics mode, which should automatically pick X11 session (not tested). 


Version-Release number of selected component (if applicable):
python3-gstreamer1-1.9.2-1.fc25.x86_64
gnome-shell-3.22.1-2.fc25.x86_64

How reproducible:
100% on affected machines, but not clear what differentiates affected from non-affected machines

Steps to Reproduce:
1. upgrade from F24 to F25
2. try to log in after upgrade, see only gray frozen screen
3. after reboot and logging in into X11, Wayland session starts working well

Comment 1 Kamil Páral 2016-11-14 12:10:28 UTC
Created attachment 1220368 [details]
journal after upgrade while trying to log in

These logs come from Jan's laptop. Unfortunately we've already logged in using X11, so we can't reproduce the Wayland login failure no more.

Comment 2 Kamil Páral 2016-11-14 12:10:46 UTC
Created attachment 1220369 [details]
rpm -qa after upgrade

Comment 3 romu 2016-11-24 21:07:22 UTC
My problem is similar but a bit different.

Computer : Dell Latitude E6400 - Core 2 Duo, old Intel Graphics

Problem: we can't login to the Wayland session, as described here. Same results. But this happens always. And we can always login using the Xorg session.

Comment 4 Kamil Páral 2016-11-25 09:03:24 UTC
romu, that's likely a different bug. Can you please create a new bug, attach system journal of a failed login attempt in there (`journalctl -b -1` shows the journal of the previous boot) and link it here? Thanks.

Comment 5 Yaacov Zamir 2016-11-25 17:40:19 UTC
It happened to me too,
hw: thinkpad t450s

I have the package python3-gstreamer1 1.10.0
I used the workaround to solve it.

Comment 6 Spolti 2016-11-26 17:35:11 UTC
Happened here too.

hw: thinkpad T460s

Comment 7 Matej Dusik 2016-11-26 20:09:42 UTC
Same issue here (after upgrading from 24 to 25) but the workaround in https://fedoraproject.org/wiki/Common_F25_bugs#Frozen_gray_screen_during_logging_in_after_upgrade

worked like a charm. Thank you :-)

Comment 8 Jeff Buhrt 2016-11-26 22:08:46 UTC
I might suggest a message on the login screen during the first login after an upgrade to Wayland that suggests how to help debug the issue and a (temporary) way to shutdown Wayland like X11's Control-Alt-Backspace and how to work around the problem.

This repeated as described on a Toshiba L755 when going from F24 to F25. [Ironically I didn't try switching to another terminal (Ctrl+Alt+F2) given the keyboard seemed to be frozen.] Now like described, Wayland is working fine on the laptop display. 

Any log info you would like to help analyze the problem to help everyone else that will be experiencing this?

I have yet to try the external HDMI attached monitor, but would love to find an external display 'card' to drive a 3rd monitor (2nd+ external monitor). I haven't yet been successful on X11 having tried 3-4+ USB->HDMI adapters. One I got so far the test C code drove a monitor, but I didn't get it working under X11.

Comment 9 Pablo José Almeida da Guia 2016-11-27 03:45:44 UTC
The workaround was worked for me.

But...
...after I logged in as root, the problem returns (for my normal user, not root).
Then, i cannot more apply the workaround (xorg and classic stopped to work).
Login as root still working, but my normal user not.

(Sony VAIO FW160AE, upgrade from fedora 24 to 25)

Comment 10 Pablo José Almeida da Guia 2016-11-27 04:33:35 UTC
(In reply to Pablo José Almeida da Guia from comment #9)
> The workaround was worked for me.
> 
> But...
> ...after I logged in as root, the problem returns (for my normal user, not
> root).
> Then, i cannot more apply the workaround (xorg and classic stopped to work).
> Login as root still working, but my normal user not.
> 
> (Sony VAIO FW160AE, upgrade from fedora 24 to 25)

My fail, I have changed a configuration, after rever worked.
Please Disregard..

Comment 11 romu 2016-11-28 08:25:55 UTC
To me, it was not the same bug, but something related to a kernel which didn't get updated. A fresh install fixed the issue.

Comment 12 Kamil Páral 2016-11-28 15:24:36 UTC
I finally reproduced this on my home PC, and I debugged the issue to be caused by ~/.cache/gstreamer-1.0/registry.x86_64.bin file. Either keeping the file or even removing it (!!) freezes wayland login. However, logging with X11 modifies the file and subsequent wayland logins are possible. I can reproduce this 100%.

Comment 13 Kamil Páral 2016-11-28 15:29:40 UTC
Created attachment 1225318 [details]
broken registry.x86_64.bin

This is the file that causes login freeze when present.

Comment 14 Kamil Páral 2016-11-28 15:32:38 UTC
Created attachment 1225319 [details]
fixed registry.x86_64.bin

This is the same file, but modified after one X11 login. Now wayland login works as well with this.

Comment 15 Kamil Páral 2016-11-28 15:51:21 UTC
Proposing as PrioritizedBug. This is a very unpleasant bug which wasn't caught before release, and seems to affect non-trivial part of our user base. Should be solved with priority.

Comment 16 Wim Taymans 2016-11-28 16:53:26 UTC
on an affected machine, can you:

1) log in (using one of the workarounds)

2) rm -rf ~/.cache/gstreamer-1.0

3) run gst-inspect and paste the output in this bug

I suspect 3) will also crash and maybe we can get more detailed info about it.

Comment 17 Kamil Páral 2016-11-28 17:00:37 UTC
Created attachment 1225392 [details]
gst-inspect-1.0 output

Comment 18 Wim Taymans 2016-11-28 17:59:26 UTC
what does 

 gst-inspect-1.0 -b 

say?

Comment 19 Kamil Páral 2016-11-29 19:32:22 UTC
$ gst-inspect-1.0 -b 
Blacklisted files:
  libgstpythonplugin.so

Total count: 1 blacklisted file

Comment 20 Jan Kurik 2016-11-30 16:17:06 UTC
This bug has been accepted to the list of Prioritized bugs as it badly affects user experience. As such we have accepted this bug to the "Prioritized bugs list".

Comment 21 Pavel Davydov 2016-12-05 13:40:24 UTC
I had this issue after 24 -> 25 upgrade. Unfortunelly, non of the available workarounds worked for me: in my case X11 failed to boot (this is a different issue of course). I haven't used X for a long time, I switched to wayland as soon as it was available, so I didn't want to waste my time on fixing it. I just took fixed registry.x86_64.bin from Kamil's attachment and copied it to my fedora machine to ~/.cache/gstreamer-1.0 directory. With this file my wayland session started just fine. Hope this will help users who have issues with X11 on there machines.

Comment 22 Kamil Páral 2016-12-05 15:46:33 UTC
Pavel, do you perhaps have EasyScreenCast extension installed? [1]

I reproduced this issue with clean F25 in a VM:
1. boot F25 with Wayland
2. go to [1] and install it (an icon should appear in top right corner)
3. rm ~/.cache/gstreamer-1.0/registry.x86_64.bin
4. reboot and try to log in again -> frozen screen

Also reproduced with clean F24->F25 upgrade in a VM:
1. boot F24 with X11
2. go to [1] and install it (an icon should appear in top right corner)
3. upgrade to F25
4. try to log in using Wayland -> frozen screen

This seem to be tightly related to bug 1373217.

[1] https://extensions.gnome.org/extension/690/easyscreencast/

Comment 23 Pavel Davydov 2016-12-05 16:09:37 UTC
Kamil, no I don't have and even never had it installed. It's the first time when I hear about it. I can try to help you somehow, though I'm not sure if freezes will reproduce or not in case I remove the registry.x86_64.bin file..

Comment 24 Kamil Páral 2016-12-06 09:41:56 UTC
Hmm, I was almost sure I discovered the issue. Jan Sedlak, who I mentioned in comment 0, also had EasyScreenCast installed. Pavel, do you have/had any other extensions installed? You can see their list with:
$ ls ~/.local/share/gnome-shell/extensions/

It's possible that there are multiple extensions which touch/initialize gstreamer when started.

Comment 25 Pavel Davydov 2016-12-06 10:04:20 UTC
Here is a list of my extensions in this directory:
$ ls -l ~/.local/share/gnome-shell/extensions/
drwxrwxr-x. 6 pablo pablo 4096 May 30  2016 display-switcher.gmail.com
drwxr-xr-x. 2 pablo pablo 4096 Oct  5 12:51 drop-down-terminal.org
drwxr-xr-x. 5 pablo pablo 4096 Oct  5 12:51 SkypeNotification.com
drwxr-xr-x. 4 pablo pablo 4096 Oct  5 12:51 text_translator.com
drwxrwxr-x. 3 pablo pablo 4096 Apr 27  2016 workspace-grid.gmail.com

Comment 26 Kamil Páral 2016-12-06 11:28:48 UTC
We already identified text_translator as an extension that crashes GNOME in bug #1399616, but it might not be relevant for this bug.

Comment 27 Kamil Páral 2016-12-14 08:46:22 UTC
This problem has been identified as a gnome-shell architecture issue in https://bugzilla.gnome.org/show_bug.cgi?id=776041 .

Comment 28 Kamil Páral 2016-12-14 08:49:48 UTC
*** Bug 1373217 has been marked as a duplicate of this bug. ***

Comment 29 Kamil Páral 2016-12-14 13:50:58 UTC
Another workaround (instead of uninstalling EasyScreenCast) is to remove clutter-gst2 package (but that might remove some dependent packages, like rhythmbox).

Comment 30 Kamil Páral 2016-12-14 14:35:25 UTC
This also happens for integrated screen recorder in GNOME (Ctrl+Alt+Shift+R shortcut), so it's not specific to EasyScreenCast. If you have clutter-gst2 installed (the default) and the gstreamer registry file changes (after some plugin update, or you remove it, etc), then your session hangs if you try to start screen recording. Ctrl+Alt+Fx hotkey doesn't work, only SysRq (or you need to ssh in and kill the session).

Reproducer:
1. boot F25 with Wayland
2. rm ~/.cache/gstreamer-1.0/registry.x86_64.bin (this simulates updating the cache e.g. when some new plugin version changes it)
3. reboot, log in, hit Ctrl+Alt+Shift+R to start recording
4. hang

From the upstream bug report:
"The only sane way forward is to stop using gstreamer in the compositor which is in the plans."

Comment 31 Jan Kurik 2016-12-15 10:52:26 UTC
Bug 1394755 is de-prioritized, since further investigation has revealed that the scope of affected users is much smaller than originally believed.

Comment 32 Angry Penguin 2017-04-07 17:04:26 UTC
Hi.

I still have this issue and can't fix it by any workaround.
I make clean F25 install and trying on Wayland record desktop via CTRL+ALT+SHIFT+R but system freeze. Also I see that new app Green Recorder working on Wayland an it works on Wayland from Fedora 25, but no, on my case, pressing record button on this app, also freezeing my system.

Trying workaround:
Loging into X-session and record desktop CTRL+ALT+SHIFT+R, next reboot and trying Wayland but again, system freeze.

I don't have EasyScreenCast, so trying another.

Remove this file (~/.cache/gstreamer-1.0/registry.x86_64.bin), loging reboot, loging into X-server session, reboot, login into Wayland session, trying recording and again freeze.

Next, remove clutter-gst2 (this remove also few other app) but still Wayland session freeze when trying recording.

Any suggestion? Any hope for fix it?

(Sorry for poor language)

Comment 33 Kamil Páral 2017-04-10 15:44:49 UTC
You can ask in the upstream bug (https://bugzilla.gnome.org/show_bug.cgi?id=776041) but I'm afraid there's no workaround currently. If you need to record your screen, use X11 session.

Comment 34 Tobias Schönberg 2017-07-08 08:34:07 UTC
A possible solution to the issue is in this merge request. Sanity checks and testing would be greatly appreciated: https://github.com/EasyScreenCast/EasyScreenCast/pull/148

Comment 35 Arif Saleem 2017-07-16 11:11:40 UTC
1. I had a working F25 installation, which was a fresh install. Wayland was working fine. I have not installed the EasyScreenCast extension. PC has nvidia graphics with the nouveau driver (GeForce 960).
2. I upgraded yesterday to F26, upgrade was fine.
3. On logging in I get the frozen grey screen, and I can't shutdown even over SSH from another machine. Had to hard reset.
4. I changed to Xorg session and that works fine.
5. Tried to change back to Wayland after one successful login on Xorg, but same freeze.

So at the moment I can only use Xorg sessions. Wayland does not allow login at all on F26, although before on F25 it was working.

?

Comment 36 Roman 2017-07-16 20:15:20 UTC
In step 5 I tried go to https://extensions.gnome.org/extension/690/easyscreencast/ and reinstall the extension.
6. I changed Xorg session to Wayland and logged with no problems.
Hope this helps.

Comment 37 Martial Pageau 2017-07-17 23:38:22 UTC
To complete Arif Saleem,

1. I also had a working F25 installation on a computer with a GeForce 960.

2. I upgraded to F26.

3. I also get the frozen grey screen after login.

4. I changed to Xorg session and that works fine.

5. I tried to change back to Wayland after one successful login on Xorg, but same freeze.

6. I tried to fresh install F26, but still get the frozen grey screen after login. 

7. I tried to remove clutter-gst2, but still get the frozen grey screen after login.

8. I tried to install, reinstall and remove the easyscreecast extension, but still get the frozen grey screen after login.

Comment 38 Martial Pageau 2017-07-29 02:56:48 UTC
I solved the problem by installing the NVIDIA proprietary drivers from this guide: https://www.if-not-true-then-false.com/2015/fedora-nvidia-guide/

So, it seems the problem is the `nouveau` NVIDIA driver.

Comment 39 Nils Philippsen 2017-07-30 11:50:08 UTC
(In reply to Martial Pageau from comment #38) 
> So, it seems the problem is the `nouveau` NVIDIA driver.

This is unlikely to be the (sole?) problem, I have Intel graphics hardware.

Comment 40 Vafa 2017-11-15 16:31:42 UTC
Same Problem on my laptop.
On first boot after upgrading FC26 to FC27, Fedora freeze right after trying to gnome user login.(gray screen with mouse pointer in center)

My emergency solution was:

Ctrl+F6
login to terminal with affected user
rm -f -r ~/.local/share/gnome-shell/extensions/EasyScreenCast.com/

And I have my FC back.

I don't have Easy Screen Cast anymore and Ctrl+Shift+Alt+R is recording without problem.

Comment 41 Fedora End Of Life 2017-11-16 19:26:41 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 42 Arif Saleem 2017-11-22 12:32:23 UTC
(In reply to Arif Saleem from comment #35)
> 1. I had a working F25 installation, which was a fresh install. Wayland was
> working fine. I have not installed the EasyScreenCast extension. PC has
> nvidia graphics with the nouveau driver (GeForce 960).
> 2. I upgraded yesterday to F26, upgrade was fine.
> 3. On logging in I get the frozen grey screen, and I can't shutdown even
> over SSH from another machine. Had to hard reset.
> 4. I changed to Xorg session and that works fine.
> 5. Tried to change back to Wayland after one successful login on Xorg, but
> same freeze.
> 
> So at the moment I can only use Xorg sessions. Wayland does not allow login
> at all on F26, although before on F25 it was working.
> 
> ?

Update: Upgraded to F27 from F26, and now I can login normally with a standard GNOME on Wayland session, rather than Xorg. So my problem has been resolved.

Comment 43 Fedora End Of Life 2017-12-12 10:33:04 UTC
Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 44 Luan Cestari 2018-05-04 11:18:34 UTC
Hi,

Is it possible to re-open this issue? I got the same problem during fedora upgrade 27->28 and it still wayland and gnome extensions. 

Thanks

Comment 45 Kamil Páral 2018-05-04 12:17:39 UTC
Sure, reopening. It's better to follow the upstream bug, though.

Comment 46 Daibhidh 2018-05-11 12:56:30 UTC
Same problem for me following the Fedora upgrade 27 -> 28, viz. I just got a frozen grey screen when trying to log in under Gnome with Wayland, and had to do a hard reset.

I had to log in under Gnome with Xorg first, then I could log in under Gnome with Wayland.

Silly question - how do I locate the upstream bug that ought to be followed?

Comment 47 Kamil Páral 2018-05-11 14:05:10 UTC
(In reply to Daibhidh from comment #46)
> Silly question - how do I locate the upstream bug that ought to be followed?

See "External Trackers" area.

Comment 48 Ben Cotton 2019-05-02 19:43:37 UTC
This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 49 Ben Cotton 2019-05-28 23:52:26 UTC
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.