Bug 1438891
| Summary: | Black output on ppc64 and s390x | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Jiri Koten <jkoten> | ||||||||||||||||||||
| Component: | mesa | Assignee: | Ben Crocker <bcrocker> | ||||||||||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Desktop QE <desktop-qa-list> | ||||||||||||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||||||||||||
| Priority: | high | ||||||||||||||||||||||
| Version: | 7.4 | CC: | bcrocker, bmilar, fmuellner, jstodola, lmiksik, mkrajnak, rstrode, salmy, tpelka, tstellar, vhumpa | ||||||||||||||||||||
| Target Milestone: | rc | Keywords: | Regression, TestBlocker | ||||||||||||||||||||
| Target Release: | 7.4 | ||||||||||||||||||||||
| Hardware: | ppc64 | ||||||||||||||||||||||
| OS: | Linux | ||||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||||
| Fixed In Version: | mesa-17.0.1-6.20170307.el7 | Doc Type: | If docs needed, set a value | ||||||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||||||
| Clone Of: | |||||||||||||||||||||||
| : | 1445423 1456040 (view as bug list) | Environment: | |||||||||||||||||||||
| Last Closed: | 2017-08-01 16:03:11 UTC | Type: | Bug | ||||||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||||
| Embargoed: | |||||||||||||||||||||||
| Bug Depends On: | |||||||||||||||||||||||
| Bug Blocks: | 1395724, 1445423, 1446209, 1456040 | ||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||
|
Description
Jiri Koten
2017-04-04 16:56:56 UTC
Created attachment 1268721 [details]
journal log
journal log for gnome-shell
Created attachment 1268722 [details]
syslog
probably a mesa problem? does glxgears work ? can you attach your xorg.conf ? Not ppc64le specific I can reproduce on ppc64 too (see https://imagebin.ca/v/3HwJdajL8rcA ), it is caused by random VNC session we are using for all our automation for secondary arches I can confirm what Jirka observed: Apr 04 16:03:45 ibm-p740-01-lp1.rhts.eng.bos.redhat.com gnome-shell[9146]: JS ERROR: Exception in callback for signal: desktop-changed: TypeError: this._applicationsButtons.forEach is not a function ApplicationsButton<._init/<@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:430 _emit@resource:///org/gnome/gjs/modules/signals.js:124 DesktopTarget<._setDesktop@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:324 wrapper@resource:///org/gnome/gjs/modules/lang.js:178 DesktopTarget<._onWindowAdded@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:312 wrapper@resource:///org/gnome/gjs/modules/lang.js:178 Btw glxgears is running, I mean no crash and it reports fps. I can even see wheel spinning when gnome-shell is not running. Extra js output appeared after a while:
Apr 04 16:08:41 ibm-p740-01-lp1.rhts.eng.bos.redhat.com gnome-shell[9146]: JS ERROR: Exception in callback for signal: desktop-changed: TypeError: this._applicationsButtons.forEach is not a function
ApplicationsButton<._init/<@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:430
_emit@resource:///org/gnome/gjs/modules/signals.js:124
DesktopTarget<._setDesktop@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:324
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
DesktopTarget<.destroy@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:345
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
ApplicationsButton<._onDestroy@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:480
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
Button<.destroy@resource:///org/gnome/shell/ui/panelMenu.js:186
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
_parent@resource:///org/gnome/gjs/modules/lang.js:131
ApplicationsButton<.destroy@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:740
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
disable@/usr/share/gnome-shell/extensions/apps-menu.github.com/extension.js:756
disableExtension@resource:///org/gnome/shell/ui/extensionSystem.js:82
disableAllExtensions/<@resource:///org/gnome/shell/ui/extensionSystem.js:357
disableAllExtensions@resource:///org/gnome/shell/ui/extensionSystem.js:356
_sessionUpdated@resource:///org/gnome/shell/ui/extensionSystem.js:374
_emit@resource:///org/gnome/gjs/modules/signals.js:124
SessionMode<._sync@resource:///org/gnome/shell/ui/sessionMode.js:205
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
SessionMode<.pushMode@resource:///org/gnome/shell/ui/sessionMode.js:167
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
ScreenShield<.activate@resource:///org/gnome/shell/ui/screenShield.js:1280
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
ScreenShield<._onLongLightboxShown@resource:///org/gnome/shell/ui/screenShield.js:898
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
_emit@resource:///org/gnome/gjs/modules/signals.js:124
Lightbox<.show/<.onComplete<@resource:///org/gnome/shell/ui/lightbox.js:186
_addHandler/params[name]@resource:///org/gnome/shell/ui/tweener.js:91
_callOnFunction@resource:///org/gnome/gjs/modules/tweener/tweener.js:203
_updateTweenByIndex@resource:///org/gnome/gjs/modules/tweener/tweener.js:333
_updateTweens@resource:///org/gnome/gjs/modules/tweener/tweener.js:345
_onEnterFrame@resource:///org/gnome/gjs/modules/tweener/tweener.js:360
_emit@resource:///org/gnome/gjs/modules/signals.js:124
ClutterFrameTicker<._onNewFrame@resource:///org/gnome/shell/ui/tweener.js:208
wrapper@resource:///org/gnome/gjs/modules/lang.js:178
ClutterFrameTicker<._init/<@resource:///org/gnome/shell/ui/tweener.js:183
Created attachment 1268742 [details]
xorg.conf
Created attachment 1268743 [details]
10-dummylibvnc.conf
just FYI metacity works New (rhel7.4 one) mesa: rpm -qa mesa\* mesa-libGL-17.0.1-2.20170307.el7.ppc64 mesa-libOSMesa-17.0.1-2.20170307.el7.ppc64 mesa-demos-8.2.0-3.el7.ppc64 mesa-private-llvm-debuginfo-3.9.1-2.el7.ppc64 mesa-libglapi-17.0.1-2.20170307.el7.ppc64 mesa-libgbm-17.0.1-2.20170307.el7.ppc64 mesa-libGLw-8.0.0-4.el7.ppc64 mesa-libGLw-devel-8.0.0-4.el7.ppc64 mesa-libEGL-devel-17.0.1-2.20170307.el7.ppc64 mesa-private-llvm-devel-3.9.1-2.el7.ppc64 mesa-demos-debuginfo-8.2.0-3.el7.ppc64 mesa-libEGL-17.0.1-2.20170307.el7.ppc64 mesa-filesystem-17.0.1-2.20170307.el7.ppc64 mesa-libGL-devel-17.0.1-2.20170307.el7.ppc64 mesa-libGLU-devel-9.0.0-4.el7.ppc64 mesa-libGLES-devel-17.0.1-2.20170307.el7.ppc64 mesa-libGLw-debuginfo-8.0.0-4.el7.ppc64 mesa-private-llvm-3.9.1-2.el7.ppc64 mesa-libOSMesa-devel-17.0.1-2.20170307.el7.ppc64 mesa-libgbm-devel-17.0.1-2.20170307.el7.ppc64 mesa-debuginfo-17.0.1-2.20170307.el7.ppc64 mesa-libGLU-9.0.0-4.el7.ppc64 mesa-libGLES-17.0.1-2.20170307.el7.ppc64 mesa-dri-drivers-17.0.1-2.20170307.el7.ppc64 mesa-libGLU-debuginfo-9.0.0-4.el7.ppc64 Old mesa: # rpm -qa mesa\* mesa-demos-8.2.0-3.el7.ppc64 mesa-private-llvm-debuginfo-3.9.1-2.el7.ppc64 mesa-libGL-11.2.2-2.20160614.el7.ppc64 mesa-libGLES-devel-11.2.2-2.20160614.el7.ppc64 mesa-libGLw-8.0.0-4.el7.ppc64 mesa-libGLw-devel-8.0.0-4.el7.ppc64 mesa-demos-debuginfo-8.2.0-3.el7.ppc64 mesa-libEGL-11.2.2-2.20160614.el7.ppc64 mesa-filesystem-11.2.2-2.20160614.el7.ppc64 mesa-libOSMesa-devel-11.2.2-2.20160614.el7.ppc64 mesa-private-llvm-devel-3.8.1-1.el7.ppc64 mesa-libGLU-devel-9.0.0-4.el7.ppc64 mesa-libGLw-debuginfo-8.0.0-4.el7.ppc64 mesa-private-llvm-3.8.1-1.el7.ppc64 mesa-libOSMesa-11.2.2-2.20160614.el7.ppc64 mesa-dri-drivers-11.2.2-2.20160614.el7.ppc64 mesa-libEGL-devel-11.2.2-2.20160614.el7.ppc64 mesa-debuginfo-17.0.1-2.20170307.el7.ppc64 mesa-libgbm-11.2.2-2.20160614.el7.ppc64 mesa-libGLES-11.2.2-2.20160614.el7.ppc64 mesa-libgbm-devel-11.2.2-2.20160614.el7.ppc64 mesa-libGL-devel-11.2.2-2.20160614.el7.ppc64 mesa-libGLU-9.0.0-4.el7.ppc64 mesa-libGLU-debuginfo-9.0.0-4.el7.ppc64 mesa-libglapi-11.2.2-2.20160614.el7.ppc64 All works. By old I mean of course 7.3 one. (In reply to Tomas Pelka from comment #6) > Apr 04 16:03:45 ibm-p740-01-lp1.rhts.eng.bos.redhat.com gnome-shell[9146]: > JS ERROR: Exception in callback for signal: desktop-changed: TypeError: > this._applicationsButtons.forEach is not a function That's my fault - when backporting the patch in bug 1236601, I didn't double-check that I wasn't relying on newer mozjs functionality (Map.prototype.forEach was added in 25.0, gjs in RHEL uses 24.0). I'll use the original bug to fix this. NOTE: This issue is ppc64 specific, output is normal on ppc64le. Will check s390x, as that is also big-endian arch. We've just fixed our testbed on s390x and reproduced the same there. Thus this happens exclusively on big-endian arches and not on little-endians. So the endianism could relate to the issue. So I assume the forEach problem is a separate issue and unrelated to the core problem, the black screen. if you do mutter --replace it's still black right? (In reply to Ray Strode [halfline] from comment #16) > if you do mutter --replace it's still black right? Or log into the regular "GNOME" session instead of "GNOME Classic". (In reply to Ray Strode [halfline] from comment #16) > So I assume the forEach problem is a separate issue and unrelated to the > core problem, the black screen. > > if you do mutter --replace it's still black right? That is correct, still black. Created attachment 1270490 [details]
better, but not adequate
So I did a scratch build with the reversion mentioned in freedesktop.org 100613 and it helped quite a bit (no more black screen!), but still produced some misrendering. see above screenshot.
(In reply to Ray Strode [halfline] from comment #19) > Created attachment 1270490 [details] > better, but not adequate > > So I did a scratch build with the reversion mentioned in freedesktop.org > 100613 and it helped quite a bit (no more black screen!), but still produced > some misrendering. see above screenshot. This looks like channel swizzling confusion, since for 32-bit argb data if you're byte-swapped in the wrong place A and B get interchanged. so i did some more investigation (bisecting) on this the last few days. It seems like comment 19 is an llvm regression caused by this commit: https://github.com/llvm-mirror/llvm/commit/76b12c4bf0bbc5c70def7b5d083a8a70547ea4e3 so worse comes to worse if we need a "right now" solution we can just ship the two reverts. I've tested that and it works okay. I'll let tstellar chime in on a potentially more right fix for llvm and Ben chime in on a potentially more right fix for mesa, but regardless we obviously can't ship 7.4 like this and we have potential path forward so devack+ Can you post the assembly/llvm IR output generated by the last good commit and also the commit that introduced the bug. The environment variable GALLIVM_DEBUG=ir,asm will dump this output for you. You will need to make sure these environment variables are set before you start gnome-shell. Created attachment 1272993 [details]
Vertex shader IR from before Mesa commit e827d917....
Created attachment 1273025 [details]
Vertex shader IR from after Mesa commit e827d917...
I typically set
export MESA_DEBUG=verbose
export MESA_GLSL="dump log uniform useprog errors"
export GALLIVM_DEBUG="tgsi ir asm dumpbc"
in my .bashrc.
This attachment, vs-post-patch.ll, and its predecessor vs-pre-patch.ll
are annotated with what I think are the significant differences.
Created attachment 1273086 [details]
PPC64LE assembly code from vs-pre-patch.ll
I've had a request to post the actual assembly code for the vertex
shader programs, so here it is.
First, the pre-patch assembly code from vs-pre-patch.ll,
generated via
% llc -mcpu=pwr8 -mattr=+altivec,+vsx vs-pre-patch.ll
+altivec turns on FP vector support, while +vsx turns
on the Vector-Scalar facility support.
Created attachment 1273087 [details]
PPC64LE assembly code from vs-post-patch.ll
As promised....
% llc -mcpu=pwr8 -mattr=+altivec,+vsx
Ben, are you able to reproduce the problem with newer versions of LLVM? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:1865 |