Bug 1438891 - Black output on ppc64 and s390x
Summary: Black output on ppc64 and s390x
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: mesa
Version: 7.4
Hardware: ppc64
OS: Linux
high
urgent
Target Milestone: rc
: 7.4
Assignee: Ben Crocker
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks: 1446209 1395724 1445423 1456040
TreeView+ depends on / blocked
 
Reported: 2017-04-04 16:56 UTC by Jiri Koten
Modified: 2018-07-03 05:18 UTC (History)
11 users (show)

Fixed In Version: mesa-17.0.1-6.20170307.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1445423 1456040 (view as bug list)
Environment:
Last Closed: 2017-08-01 16:03:11 UTC
Target Upstream Version:


Attachments (Terms of Use)
journal log (122.13 KB, text/plain)
2017-04-04 17:02 UTC, Jiri Koten
no flags Details
syslog (830.36 KB, text/plain)
2017-04-04 17:04 UTC, Jiri Koten
no flags Details
xorg.conf (6.73 KB, text/plain)
2017-04-04 20:13 UTC, Tomas Pelka
no flags Details
10-dummylibvnc.conf (780 bytes, text/plain)
2017-04-04 20:14 UTC, Tomas Pelka
no flags Details
better, but not adequate (882.01 KB, image/png)
2017-04-10 13:20 UTC, Ray Strode [halfline]
no flags Details
Vertex shader IR from before Mesa commit e827d917.... (49.57 KB, text/plain)
2017-04-20 14:58 UTC, Ben Crocker
no flags Details
Vertex shader IR from after Mesa commit e827d917... (49.33 KB, text/plain)
2017-04-20 15:57 UTC, Ben Crocker
no flags Details
PPC64LE assembly code from vs-pre-patch.ll (21.21 KB, text/plain)
2017-04-20 21:14 UTC, Ben Crocker
no flags Details
PPC64LE assembly code from vs-post-patch.ll (20.05 KB, text/plain)
2017-04-20 21:15 UTC, Ben Crocker
no flags Details


Links
System ID Priority Status Summary Last Updated
FreeDesktop.org 100613 None None None 2017-04-10 12:34:46 UTC
Red Hat Product Errata RHSA-2017:1865 normal SHIPPED_LIVE Moderate: X.org X11 libraries security, bug fix and enhancement update 2017-08-01 17:50:43 UTC

Description Jiri Koten 2017-04-04 16:56:56 UTC
Description of problem:
Running gnome-shell in vnc session on ppc64le results in black screen. If I run other WM like metacity the output is fine.

Version-Release number of selected component (if applicable):
gnome-shell-3.22.3-8.el7.ppc64
mutter-3.22.3-4.el7.ppc64

How reproducible:
100%

Steps to Reproduce:
1.vncserver
2. connect to vnc session
3.

Actual results:
Nothing but black screen.

Expected results:
Session is displayed properly.

Additional info:

Comment 3 Jiri Koten 2017-04-04 17:02:25 UTC
Created attachment 1268721 [details]
journal log

journal log for gnome-shell

Comment 4 Jiri Koten 2017-04-04 17:04:46 UTC
Created attachment 1268722 [details]
syslog

Comment 5 Ray Strode [halfline] 2017-04-04 19:26:21 UTC
probably a mesa problem? does glxgears work ? can you attach your xorg.conf ?

Comment 6 Tomas Pelka 2017-04-04 20:09:55 UTC
Not ppc64le specific I can reproduce on ppc64 too (see https://imagebin.ca/v/3HwJdajL8rcA ), it is caused by random VNC session we are using for all our automation for secondary arches

I can confirm what Jirka observed:

Apr 04 16:03:45 ibm-p740-01-lp1.rhts.eng.bos.redhat.com gnome-shell[9146]: JS ERROR: Exception in callback for signal: desktop-changed: TypeError: this._applicationsButtons.forEach is not a function
                                                                           ApplicationsButton<._init/<@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:430
                                                                           _emit@resource:///org/gnome/gjs/modules/signals.js:124
                                                                           DesktopTarget<._setDesktop@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:324
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           DesktopTarget<._onWindowAdded@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:312
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178

Btw glxgears is running, I mean no crash and it reports fps. I can even see wheel spinning when gnome-shell is not running.

Comment 7 Tomas Pelka 2017-04-04 20:11:05 UTC
Extra js output appeared after a while:

Apr 04 16:08:41 ibm-p740-01-lp1.rhts.eng.bos.redhat.com gnome-shell[9146]: JS ERROR: Exception in callback for signal: desktop-changed: TypeError: this._applicationsButtons.forEach is not a function
                                                                           ApplicationsButton<._init/<@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:430
                                                                           _emit@resource:///org/gnome/gjs/modules/signals.js:124
                                                                           DesktopTarget<._setDesktop@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:324
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           DesktopTarget<.destroy@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:345
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           ApplicationsButton<._onDestroy@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:480
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           Button<.destroy@resource:///org/gnome/shell/ui/panelMenu.js:186
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           _parent@resource:///org/gnome/gjs/modules/lang.js:131
                                                                           ApplicationsButton<.destroy@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:740
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           disable@/usr/share/gnome-shell/extensions/apps-menu@gnome-shell-extensions.gcampax.github.com/extension.js:756
                                                                           disableExtension@resource:///org/gnome/shell/ui/extensionSystem.js:82
                                                                           disableAllExtensions/<@resource:///org/gnome/shell/ui/extensionSystem.js:357
                                                                           disableAllExtensions@resource:///org/gnome/shell/ui/extensionSystem.js:356
                                                                           _sessionUpdated@resource:///org/gnome/shell/ui/extensionSystem.js:374
                                                                           _emit@resource:///org/gnome/gjs/modules/signals.js:124
                                                                           SessionMode<._sync@resource:///org/gnome/shell/ui/sessionMode.js:205
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           SessionMode<.pushMode@resource:///org/gnome/shell/ui/sessionMode.js:167
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           ScreenShield<.activate@resource:///org/gnome/shell/ui/screenShield.js:1280
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           ScreenShield<._onLongLightboxShown@resource:///org/gnome/shell/ui/screenShield.js:898
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           _emit@resource:///org/gnome/gjs/modules/signals.js:124
                                                                           Lightbox<.show/<.onComplete<@resource:///org/gnome/shell/ui/lightbox.js:186
                                                                           _addHandler/params[name]@resource:///org/gnome/shell/ui/tweener.js:91
                                                                           _callOnFunction@resource:///org/gnome/gjs/modules/tweener/tweener.js:203
                                                                           _updateTweenByIndex@resource:///org/gnome/gjs/modules/tweener/tweener.js:333
                                                                           _updateTweens@resource:///org/gnome/gjs/modules/tweener/tweener.js:345
                                                                           _onEnterFrame@resource:///org/gnome/gjs/modules/tweener/tweener.js:360
                                                                           _emit@resource:///org/gnome/gjs/modules/signals.js:124
                                                                           ClutterFrameTicker<._onNewFrame@resource:///org/gnome/shell/ui/tweener.js:208
                                                                           wrapper@resource:///org/gnome/gjs/modules/lang.js:178
                                                                           ClutterFrameTicker<._init/<@resource:///org/gnome/shell/ui/tweener.js:183

Comment 8 Tomas Pelka 2017-04-04 20:13:42 UTC
Created attachment 1268742 [details]
xorg.conf

Comment 9 Tomas Pelka 2017-04-04 20:14:10 UTC
Created attachment 1268743 [details]
10-dummylibvnc.conf

Comment 10 Tomas Pelka 2017-04-04 20:38:14 UTC
just FYI metacity works

Comment 11 Tomas Pelka 2017-04-04 21:03:38 UTC
New (rhel7.4 one) mesa:
rpm -qa mesa\*
mesa-libGL-17.0.1-2.20170307.el7.ppc64
mesa-libOSMesa-17.0.1-2.20170307.el7.ppc64
mesa-demos-8.2.0-3.el7.ppc64
mesa-private-llvm-debuginfo-3.9.1-2.el7.ppc64
mesa-libglapi-17.0.1-2.20170307.el7.ppc64
mesa-libgbm-17.0.1-2.20170307.el7.ppc64
mesa-libGLw-8.0.0-4.el7.ppc64
mesa-libGLw-devel-8.0.0-4.el7.ppc64
mesa-libEGL-devel-17.0.1-2.20170307.el7.ppc64
mesa-private-llvm-devel-3.9.1-2.el7.ppc64
mesa-demos-debuginfo-8.2.0-3.el7.ppc64
mesa-libEGL-17.0.1-2.20170307.el7.ppc64
mesa-filesystem-17.0.1-2.20170307.el7.ppc64
mesa-libGL-devel-17.0.1-2.20170307.el7.ppc64
mesa-libGLU-devel-9.0.0-4.el7.ppc64
mesa-libGLES-devel-17.0.1-2.20170307.el7.ppc64
mesa-libGLw-debuginfo-8.0.0-4.el7.ppc64
mesa-private-llvm-3.9.1-2.el7.ppc64
mesa-libOSMesa-devel-17.0.1-2.20170307.el7.ppc64
mesa-libgbm-devel-17.0.1-2.20170307.el7.ppc64
mesa-debuginfo-17.0.1-2.20170307.el7.ppc64
mesa-libGLU-9.0.0-4.el7.ppc64
mesa-libGLES-17.0.1-2.20170307.el7.ppc64
mesa-dri-drivers-17.0.1-2.20170307.el7.ppc64
mesa-libGLU-debuginfo-9.0.0-4.el7.ppc64

Old mesa:
# rpm -qa mesa\*
mesa-demos-8.2.0-3.el7.ppc64
mesa-private-llvm-debuginfo-3.9.1-2.el7.ppc64
mesa-libGL-11.2.2-2.20160614.el7.ppc64
mesa-libGLES-devel-11.2.2-2.20160614.el7.ppc64
mesa-libGLw-8.0.0-4.el7.ppc64
mesa-libGLw-devel-8.0.0-4.el7.ppc64
mesa-demos-debuginfo-8.2.0-3.el7.ppc64
mesa-libEGL-11.2.2-2.20160614.el7.ppc64
mesa-filesystem-11.2.2-2.20160614.el7.ppc64
mesa-libOSMesa-devel-11.2.2-2.20160614.el7.ppc64
mesa-private-llvm-devel-3.8.1-1.el7.ppc64
mesa-libGLU-devel-9.0.0-4.el7.ppc64
mesa-libGLw-debuginfo-8.0.0-4.el7.ppc64
mesa-private-llvm-3.8.1-1.el7.ppc64
mesa-libOSMesa-11.2.2-2.20160614.el7.ppc64
mesa-dri-drivers-11.2.2-2.20160614.el7.ppc64
mesa-libEGL-devel-11.2.2-2.20160614.el7.ppc64
mesa-debuginfo-17.0.1-2.20170307.el7.ppc64
mesa-libgbm-11.2.2-2.20160614.el7.ppc64
mesa-libGLES-11.2.2-2.20160614.el7.ppc64
mesa-libgbm-devel-11.2.2-2.20160614.el7.ppc64
mesa-libGL-devel-11.2.2-2.20160614.el7.ppc64
mesa-libGLU-9.0.0-4.el7.ppc64
mesa-libGLU-debuginfo-9.0.0-4.el7.ppc64
mesa-libglapi-11.2.2-2.20160614.el7.ppc64

All works.

Comment 12 Tomas Pelka 2017-04-04 21:04:39 UTC
By old I mean of course 7.3 one.

Comment 13 Florian Müllner 2017-04-04 21:27:52 UTC
(In reply to Tomas Pelka from comment #6)
> Apr 04 16:03:45 ibm-p740-01-lp1.rhts.eng.bos.redhat.com gnome-shell[9146]:
> JS ERROR: Exception in callback for signal: desktop-changed: TypeError:
> this._applicationsButtons.forEach is not a function

That's my fault - when backporting the patch in bug 1236601, I didn't double-check that I wasn't relying on newer mozjs functionality (Map.prototype.forEach was added in 25.0, gjs in RHEL uses 24.0). I'll use the original bug to fix this.

Comment 14 Vitezslav Humpa 2017-04-05 08:27:36 UTC
NOTE: This issue is ppc64 specific, output is normal on ppc64le. Will check s390x, as that is also big-endian arch.

Comment 15 Vitezslav Humpa 2017-04-06 09:06:36 UTC
We've just fixed our testbed on s390x and reproduced the same there. Thus this happens exclusively on big-endian arches and not on little-endians. So the endianism could relate to the issue.

Comment 16 Ray Strode [halfline] 2017-04-06 14:43:01 UTC
So I assume the forEach problem is a separate issue and unrelated to the core problem, the black screen.

if you do mutter --replace it's still black right?

Comment 17 Florian Müllner 2017-04-06 14:51:13 UTC
(In reply to Ray Strode [halfline] from comment #16)
> if you do mutter --replace it's still black right?

Or log into the regular "GNOME" session instead of "GNOME Classic".

Comment 18 Jiri Koten 2017-04-06 15:06:18 UTC
(In reply to Ray Strode [halfline] from comment #16)
> So I assume the forEach problem is a separate issue and unrelated to the
> core problem, the black screen.
> 
> if you do mutter --replace it's still black right?

That is correct, still black.

Comment 19 Ray Strode [halfline] 2017-04-10 13:20:17 UTC
Created attachment 1270490 [details]
better, but not adequate

So I did a scratch build with the reversion mentioned in freedesktop.org 100613 and it helped quite a bit (no more black screen!), but still produced some misrendering. see above screenshot.

Comment 20 Adam Jackson 2017-04-11 18:36:30 UTC
(In reply to Ray Strode [halfline] from comment #19)
> Created attachment 1270490 [details]
> better, but not adequate
> 
> So I did a scratch build with the reversion mentioned in freedesktop.org
> 100613 and it helped quite a bit (no more black screen!), but still produced
> some misrendering. see above screenshot.

This looks like channel swizzling confusion, since for 32-bit argb data if you're byte-swapped in the wrong place A and B get interchanged.

Comment 21 Ray Strode [halfline] 2017-04-14 13:41:02 UTC
so i did some more investigation (bisecting) on this the last few days.  It seems like comment 19 is an llvm regression caused by this commit:

https://github.com/llvm-mirror/llvm/commit/76b12c4bf0bbc5c70def7b5d083a8a70547ea4e3

Comment 22 Ray Strode [halfline] 2017-04-14 21:19:30 UTC
so worse comes to worse if we need a "right now" solution we can just ship the two reverts.  I've tested that and it works okay.

I'll let tstellar chime in on a potentially more right fix for llvm and Ben chime in on a potentially more right fix for mesa, but regardless we obviously can't ship 7.4 like this and we have potential path forward so devack+

Comment 23 Tom Stellard 2017-04-20 13:51:26 UTC
Can you post the assembly/llvm IR output generated by the last good commit and also the commit that introduced the bug.  The environment variable GALLIVM_DEBUG=ir,asm will dump this output for you.

You will need to make sure these environment variables are set before you start gnome-shell.

Comment 24 Ben Crocker 2017-04-20 14:58:48 UTC
Created attachment 1272993 [details]
Vertex shader IR from before Mesa commit e827d917....

Comment 25 Ben Crocker 2017-04-20 15:57:17 UTC
Created attachment 1273025 [details]
Vertex shader IR from after Mesa commit e827d917...

I typically set

export MESA_DEBUG=verbose
export MESA_GLSL="dump log uniform useprog errors"
export GALLIVM_DEBUG="tgsi ir asm dumpbc"

in my .bashrc.

This attachment, vs-post-patch.ll, and its predecessor vs-pre-patch.ll
are annotated with what I think are the significant differences.

Comment 26 Ben Crocker 2017-04-20 21:14:07 UTC
Created attachment 1273086 [details]
PPC64LE assembly code from vs-pre-patch.ll

I've had a request to post the actual assembly code for the vertex
shader programs, so here it is.

First, the pre-patch assembly code from vs-pre-patch.ll,
generated via

% llc -mcpu=pwr8 -mattr=+altivec,+vsx vs-pre-patch.ll

+altivec turns on FP vector support, while +vsx turns
on the Vector-Scalar facility support.

Comment 27 Ben Crocker 2017-04-20 21:15:29 UTC
Created attachment 1273087 [details]
PPC64LE assembly code from vs-post-patch.ll

As promised....

% llc -mcpu=pwr8 -mattr=+altivec,+vsx

Comment 29 Tom Stellard 2017-04-24 09:35:41 UTC
Ben, are you able to reproduce the problem with newer versions of LLVM?

Comment 39 errata-xmlrpc 2017-08-01 16:03:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1865


Note You need to log in before you can comment on or make changes to this bug.