Bug 1610227

There is a workaround:

export WEBKIT_DISABLE_COMPOSITING_MODE=1

with it it works as expected, so it's not WebKitGTK+'s fault, but video drivers issue. I will reassign this to mesa for now to further investigate it there. Martin can you please try to find out what graphics adapter is on that machine and what drivers are used?

[test@ibm-z-35 yelp]$ glxinfo | grep renderer
    GLX_MESA_multithread_makecurrent, GLX_MESA_query_renderer, 
    GLX_MESA_multithread_makecurrent, GLX_MESA_query_renderer, 
Extended renderer info (GLX_MESA_query_renderer):
OpenGL renderer string: llvmpipe (LLVM 6.0, 128 bits)

xorg-x11-proto-devel-2018.4-1.el7.noarch
xorg-x11-server-common-1.20.0-1.el7.s390x
xorg-x11-server-utils-7.7-20.el7.s390x
xorg-x11-xinit-1.3.4-2.el7.s390x
xorg-x11-utils-7.5-23.el7.s390x
xorg-x11-font-utils-7.5-21.el7.s390x
xorg-x11-xkb-utils-7.7-14.el7.s390x
xorg-x11-xauth-1.0.9-1.el7.s390x
xorg-x11-xtrans-devel-1.3.5-1.el7.noarch
xorg-x11-util-macros-1.19.0-3.el7.noarch
xorg-x11-server-devel-1.20.0-1.el7.s390x
xorg-x11-server-Xvfb-1.20.0-1.el7.s390x
xorg-x11-server-Xorg-1.20.0-1.el7.s390x
abrt-addon-xorg-2.1.11-51.el7.s390x
xorg-x11-drv-dummy-0.3.7-1.el7.1.s390x
xorg-x11-drv-dummy-debuginfo-0.3.7-1.el7.1.s390x
mesa-libGLU-9.0.0-4.el7.s390x
mesa-libEGL-devel-18.0.5-1.el7.s390x
mesa-libgbm-devel-18.0.5-1.el7.s390x
mesa-libglapi-18.0.5-1.el7.s390x
mesa-libGL-devel-18.0.5-1.el7.s390x
mesa-libGL-18.0.5-1.el7.s390x
mesa-filesystem-18.0.5-1.el7.s390x
mesa-dri-drivers-18.0.5-1.el7.s390x
mesa-libEGL-18.0.5-1.el7.s390x
mesa-libgbm-18.0.5-1.el7.s390x

Reproduced on 7.7
[test@ibm-z-13 ~]$ glxinfo | grep renderer
    GLX_MESA_multithread_makecurrent, GLX_MESA_query_renderer, 
    GLX_EXT_visual_rating, GLX_MESA_copy_sub_buffer, GLX_MESA_query_renderer, 
Extended renderer info (GLX_MESA_query_renderer):
OpenGL renderer string: llvmpipe (LLVM 7.0, 128 bits)

xorg-x11-font-utils-7.5-21.el7.s390x
xorg-x11-server-devel-1.20.4-3.el7.s390x
xorg-x11-xtrans-devel-1.3.5-1.el7.noarch
xorg-x11-server-Xorg-1.20.4-3.el7.s390x
xorg-x11-server-common-1.20.4-3.el7.s390x
xorg-x11-drv-dummy-0.3.7-1.el7.1.s390x
xorg-x11-xkb-utils-7.7-14.el7.s390x
xorg-x11-drv-dummy-debuginfo-0.3.7-1.el7.1.s390x
xorg-x11-proto-devel-2018.4-1.el7.noarch
xorg-x11-xauth-1.0.9-1.el7.s390x
xorg-x11-utils-7.5-23.el7.s390x
xorg-x11-server-utils-7.7-20.el7.s390x
abrt-addon-xorg-2.1.11-55.el7.s390x
xorg-x11-xinit-1.3.4-2.el7.s390x
xorg-x11-util-macros-1.19.0-3.el7.noarch
xorg-x11-server-Xvfb-1.20.4-3.el7.s390x

mesa-libGL-18.3.4-5.el7.s390x
mesa-libgbm-18.3.4-5.el7.s390x
mesa-libEGL-devel-18.3.4-5.el7.s390x
mesa-libGLU-9.0.0-4.el7.s390x
mesa-libEGL-18.3.4-5.el7.s390x
mesa-libGL-devel-18.3.4-5.el7.s390x
mesa-libgbm-devel-18.3.4-5.el7.s390x
mesa-libglapi-18.3.4-5.el7.s390x
mesa-filesystem-18.3.4-5.el7.s390x
mesa-dri-drivers-18.3.4-5.el7.s390x
mesa-khr-devel-18.3.4-5.el7.s390x

workaround WEBKIT_DISABLE_COMPOSITING_MODE=1 yelp
still valid

Michael, should we permanently disable the compositing mode on s390x (downstream and upstream?)?

Ummm, well it certainly might make sense to do that downstream if we aren't going to be able to fix it. The simplest way would be to disable OpenGL support when building WebKit for s390x.

I wouldn't do this upstream though.

It's most likely an endianness issue, and that doesn't imply it's definitely the mesa drivers fault, endian color issues can exist on either side of the API I expect.

Does this happen on RHEL 8? since we have done some upstream work on regression tests on s390 in this area.

(In reply to Dave Airlie from comment #7)
> It's most likely an endianness issue, and that doesn't imply it's definitely
> the mesa drivers fault, endian color issues can exist on either side of the
> API I expect.

It could easily be WebKit's fault. None of WebKit's graphics stack is tested on big endian platforms.

passing this to Ben to see if he can reproduce and get us a setup to track it down (Ben try on RHEL8 maybe as well).

(In reply to Dave Airlie from comment #7)
> Does this happen on RHEL 8? since we have done some upstream work on
> regression tests on s390 in this area.

I asked yesterday the QA for RHEL 8 testing, but didn't hear anything back from them. Michale, did you get any results?

I am still trying to get a usable s390x from beaker, so far with no success I was not able to get gdm running on this architecture. Still investigating.

So this is fun.  The BZ features webkit, so I figured it might be a
good idea to build webkitgtk4.  I did this on two separate machines:
my 160-processor Power8 machine and the Power8 in RDU that I've been
using.

Interestingly, webkitgtk4 appears to be obsolete on Fedora as of
Fedora 28 & later; it appears to have been superseded by webkit2gtk3?
and/or webkitgtk-sharp?

I am running Fedora 31 on my 160-processor machine, on which I (also) have gcc
9.3.1-2.  I'm already running RHEL 8 (4.18.0-193.6.1) on the RDU
machine, with gcc 8.3.1-5.

I did
'rhpkg co -a webkitgtk4 ; rhpkg switch-branch rhel-8.0 ; rhpkg prep ; rhpkg local' 
on both machines and, after installing missing packages indicated by
rhpkg prep, I was able to get builds going.  (Sorry, tried 'rhpkg
mockbuild' early on; failed miserably.)

The build logs are enormous, as one might expect from a source code
base of more than 2.5 million lines.

Both builds fail in the same place:

/home/rhel8-packages/webkitgtk4/webkitgtk-2.18.1/Source/WebKit/UIProcess/gtk/WaylandCompositor.cpp:133:116:
error: 'EGL_WAYLAND_BUFFER_WL' was not declared in this scope

EGL_WAYLAND_BUFFER_WL is #defined in /usr/include/EGL/eglmesaext.h, so
SOMEBODY appears not to be #including eglmesaext.h (or, more likely,
/usr/include/eglext.h, which #includes eglmesaext.h).

Looking in WaylandCompositor.cpp, I see that <EGL/eglext.h> is (only)
#included #if PLATFORM(WAYLAND) && USE(EGL).

However, it seems that PLATFORM(WAYLAND) and USE(EGL) should both be
true, as both WTF_PLATFORM_WAYLAND and USE_EGL are #defined as 1 in 
.../ppc64le-redhat-linux-gnu/cmakeconfig.h.

Investigation continues.

Also built webkitgtk3 on 160-processor Power8 system.  Webkitgtk3
support stopped with RHEL 7.9; build errors: invalid conversion from
'const JSChar*' to 'const UChar*' and from 'const UChar*' to 'const
JSChar*' during compilation of JSStringRef.cpp

(In reply to Ben Crocker from comment #12)
> So this is fun.  The BZ features webkit, so I figured it might be a
> good idea to build webkitgtk4.  I did this on two separate machines:
> my 160-processor Power8 machine and the Power8 in RDU that I've been
> using.
> 
> Interestingly, webkitgtk4 appears to be obsolete on Fedora as of
> Fedora 28 & later; it appears to have been superseded by webkit2gtk3?

It's the same thing, the package was just renamed.

> and/or webkitgtk-sharp?

These are obsolete .NET bindings.

> Both builds fail in the same place:
> 
> /home/rhel8-packages/webkitgtk4/webkitgtk-2.18.1/Source/WebKit/UIProcess/gtk/
> WaylandCompositor.cpp:133:116:
> error: 'EGL_WAYLAND_BUFFER_WL' was not declared in this scope

OK, 2.18.1 is really old. The current stable version is 2.28.2, which we have in RHEL 8.3 branch. There's no point in debugging anything older than this, so I would avoid debugging the RHEL 8.0 branch.

> Also built webkitgtk3 on 160-processor Power8 system.  Webkitgtk3
> support stopped with RHEL 7.9; build errors: invalid conversion from
> 'const JSChar*' to 'const UChar*' and from 'const UChar*' to 'const
> JSChar*' during compilation of JSStringRef.cpp

webkitgtk3 is an old branch of WebKit from March 2014, the last version to support a single-process API. It only exists in RHEL 7 because we are not able to retire packages from RHEL 7.

As you suggested, I downloaded and built webkit2gtk3; no problems this
time.

I tried MiniBrowser as suggested above; it may be a good start.

I think we need an even smaller program to do something like this:
1. open a video file, e.g. myvideo.webm
2. read a frame;
3. display the frame;
4. repeat

I think we need more good test videos.  I searched the web and found
this:

https://en.wikipedia.org/wiki/File:This_is_a_10_second_testvideo_with_bars_and_tone.webm
which is exactly as advertised.  It displays 

I've attached it to this BZ as BarsAndTone-10-sec-test.webm

Created attachment 1696226 [details]
Brief video displaying color bars and moving text

Do you need something in particular from me?

I would just check yelp and see if the bug as originally reported (screencast in the first comment) still exists. If not, close.

I was able to reserve ibm-z-102 from Beaker; I'll run the test Wednesday AM.

If your it works, then I'll close the bug as you suggest.  However, if it
doesn't work, we need a simplified test program.

Sorry, my last sentence was garbled; what I MEANT to say was:

If your suggestion works, then I'll close the bug as you suggest.  However, if it
doesn't work, we need a simplified test program.

I tried yelp, but got some archaic interface and couldn't really proceed.

HOWEVER, with Dave Airlie's fixes for BZ 1847064 in place (Mesa 20.1.2-3),
I've noted that:

• gnome-shell appears to work fine;

• Firefox appears to work fine;

• Videos on Youtube work, although the machine I'm working on
  (ibm-z-102) on only has 2 processors and 2 GB RAM, so it isn't
  really powerful enough to cope with the volume of incoming data.

So I think we can count this one as fixed.

If yelp is broken, then a separate bug needs to be reported for that. If you can successfully play videos in yelp and they look OK, then this bug is fixed.

The simplest-possible test program I can give you is /usr/libexec/webkitgtk-4.0/MiniBrowser. If videos are pink there and you need to see them frame-by-frame, that's beyond something I can help with and you'd need to talk with multimedia experts to figure out how to do that. I'm not actually certain if that's possible using web APIs.

OK, I see what you mean.

I tested

https://en.wikipedia.org/wiki/File:This_is_a_10_second_testvideo_with_bars_and_tone.webm

with MiniBrowser, and the colors are still broken,
at least in the preview:

The video shows seven color bars across the top; comparing
ppc64le (correct) vs. s390x, from left to right:

ppc64le:    s390x:
_________   _________
white-ish   white-ish
yellowish   white
cyan-ish    magenta-ish
green       white
magenta     cyan-ish
reddish     white-ish (with tinge of green)
blue        blue

BUT, these wrong colors appear only in the preview.
Once you start the video, the correct colors appear.

Ben, what info specifically do you need from me? At this point, all I can tell you is that there's probably an endianness bug... somewhere. Figuring out where is going to be challenging....

Michael, I was hoping maybe you could provide some insight into
how the MiniBrowser displays the initial frame, but I'm thinking
maybe I can get to that by collecting Mesa's debug output
(shader source, NIR IR) and examining that.  Presumably it will
be the last thing drawn, so the pertinent output should be the
last vertex & fragment shader.

Unfortunately you are way beyond my expertise at this point, sorry. :( I don't know much about graphics asides from high-level WebKit architecture details.

So one thing I don't quite understand about this bug report is that the original report shows the pink color displays in the entire web view (so the problem is a general graphics issue not related to the video), whereas it sounds like the bug you're seeing now is really limited to only the video itself, and only the video preview before playback begins. So that's odd.

If you're interested in how MiniBrowser displays the page in general, the first thing to check is: is the page using accelerated compositing mode? Try with: WEBKIT_FORCE_COMPOSITING_MODE=1 and WEBKIT_DISABLE_COMPOSITING_MODE=1 and see if that makes a difference, as that could narrow things down a lot. When accelerated compositing is enabled, then the page is rendered with OpenGL (so with mesa), and otherwise it's rendered using cairo and mesa shouldn't be involved at all. By default, AC mode is entered if the page uses CSS transforms (maybe also in other cases?); otherwise, the page uses software rendering. The web view will enter and exit AC mode depending on the needs of the web content. But it sounds like there is no problem in any of this code, because if so the entire web view would be pink.

The video playback itself uses GStreamer, and that's always going to wind up using mesa via GStreamerGL. But if this bug is only occurring for the video preview before the video is played, that kinda suggests the problem is maybe not related to GStreamer.

I don't see any evidence to suspect mesa here. It seems more likely to be an endianness issue inside WebKit. You *should* be able to guarantee that mesa isn't used by trying WEBKIT_DISABLE_COMPOSITING=1 to take it out of the equation. I guess it *might* be possible that it's used to generate the video preview via GStreamerGL, but I doubt it.

Oh, I missed tpopela is comment #2:

(In reply to Tomas Popela from comment #2)
> There is a workaround:
> 
> export WEBKIT_DISABLE_COMPOSITING_MODE=1
> 
> with it it works as expected, so it's not WebKitGTK+'s fault, but video
> drivers issue. I will reassign this to mesa for now to further investigate
> it there. Martin can you please try to find out what graphics adapter is on
> that machine and what drivers are used?

That's proof that accelerated compositing mode is to blame, and certainly suggestive of a possible issue in mesa.

Anyway, it sounds like the original bug here is definitely fixed. We're not sure what's going wrong with the image preview. My first guess was a problem in the JPEG decoder. I asked you to test opening the image directly, and you say it works fine, so something else must be wrong somewhere. Anyway, it's pretty unlikely to be a mesa problem, and definitely not the problem originally reported here, so closing.

Sorry, I was running tests while you were closing this BZ....

I figured out my problem with yelp: I needed to install
gnome-user-docs and gnome-getting-started-docs;
yelp startup screen started to look good.

Then:

-> Getting Started with GNOME
• looks good
 -> Switching Tasks
  • background text that should be dark gray becomes bluish, so somebody's alpha channel is on blue instead
  • the progress bar that should be dark gray is blue
  • see some of the text underneath, and the moving mouse outline in the middle of the video
  • only plays about 18 seconds of the 45-second video
  • Start/Pause button that SHOULD go back to start triangle stays at pause
  • click on background, text stays violet/bluish and the video previews are blank
  • video won't play again, no matter how many times I click on control button

Per advice in comment 2, I did this:

% WEBKIT_DISABLE_COMPOSITING_MODE=1 yelp

There is no Mesa involvement (I also ran
% WEBKIT_DISABLE_COMPOSITING_MODE=1 GALLIVM_DEBUG="ir" yelp
to confirm that there is no Mesa involvement), and the results are
somewhat different but at least as bad:
• video background is still bluish/violet
• only plays about 5 seconds of the video

Created attachment 1700522 [details]
Mesa debug log (GALLIVM_DEBUG="ir" yelp) on ppc64le (correct)

Here is a log from
GALLIVM_DEBUG="ir" gdb /usr/bin/yelp;

note the annotation

# about to start "Getting Started -> Switch Tasks" video

Created attachment 1700523 [details]
Mesa log from GALLIVM_DEBUG="ir" gdb /usr/bin/yelp on S390x (wrong)

Here is another Mesa log from

% GALLIVM_DEBUG="ir" gdb /usr/bin/yelp

on s390x; as noted in the comment above, this
exhibits similar behavior to the original.

Again, note the annotation

# about to click on Switch Tasks

Note also the similarity between S390x and ppc64le logs
up to the annotations; sometimes one, sometimes the other
has a setup_variant that is absent in the other, but they
are otherwise very similar.

Suggest examining differences with 'meld'

> % WEBKIT_DISABLE_COMPOSITING_MODE=1 yelp
>
>There is no Mesa involvement (I also ran
>% WEBKIT_DISABLE_COMPOSITING_MODE=1 GALLIVM_DEBUG="ir" yelp
>to confirm that there is no Mesa involvement), and the results are
>somewhat different but at least as bad:
>• video background is still bluish/violet
>• only plays about 5 seconds of the video

This has to be different from the originally-reported bug, which reported pink throughout the entire web view and only when accelerated compositing mode is used. I'll reassign to WebKit.

I'm going to go ahead and close this since we are now in maintenance support phase for RHEL 7, and this issue is not likely to qualify for an errata even if we were to somehow fix it.