Bug 1206060

Summary: incorrect IMAGE_BYTE_ORDER on big-endian arch
Product: [Fedora] Fedora Reporter: Dan Horák <dan>
Component: xorg-x11-serverAssignee: X/OpenGL Maintenance List <xgl-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: bphinz, dan, jcajka, jcapik, jstodola, twaugh, xgl-maint
Target Milestone: ---   
Target Release: ---   
Hardware: ppc64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-04-21 12:55:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 467765, 1071880, 1206039, 1210407    
Attachments:
Description Flags
crazy colour in VNC during installation
none
XFCE desktop over VNC from F-22 ppc64
none
f22-beta-tc2-vnc1.jpg
none
f22-beta-tc2-vnc2.jpg
none
f22-beta-tc2-vnc3.jpg
none
packet capture for connecting to the XFCe session none

Description Dan Horák 2015-03-26 09:00:27 UTC
Created attachment 1006705 [details]
crazy colour in VNC during installation

Very likely the composes were using a either vnc itself or some library compiled by a buggy gcc (bug 1194704) and should be fixed in next compose.

Comment 1 Tim Waugh 2015-03-26 09:46:38 UTC
Please confirm the version of tigervnc-server in use.

Comment 2 Dan Horák 2015-03-26 10:00:42 UTC
tigervnc-1.4.2-3.fc22.s390x
and
tigervnc-1.4.2-3.fc22.ppc64

Comment 3 Tim Waugh 2015-03-26 11:23:24 UTC
Do you need me to take some action, or does 'should be fixed in the next compose' mean that a fix is already made and there is nothing for me to do?

Comment 4 Dan Horák 2015-03-26 11:32:13 UTC
Yes, nothing to do for you at the moment, we should already have all the necessary bits rebuilt, so this issue should go away in the next compose. But we want to track it.

Comment 5 Dan Horák 2015-03-30 14:50:21 UTC
moving to Beta tracker, so we will recheck it with the Beta composes

Comment 6 Dan Horák 2015-04-08 10:09:54 UTC
Tim, seems there is a real problem with VNC on big endians, even with the latest tigervnc-1.4.3-2.fc22 the colours are still wrong

Comment 7 Dan Horák 2015-04-08 10:25:07 UTC
Created attachment 1012163 [details]
XFCE desktop over VNC from F-22 ppc64

Can be reproduced by connecting from eg. F-20 client connecting to XFCE desktop running on ppc64 (big endian) F-22 host.

Comment 8 Dan Horák 2015-04-08 10:51:26 UTC
and using tigervnc-server-1.4.2-3.fc22.ppc64

Comment 9 Tim Waugh 2015-04-08 16:01:52 UTC
http://ppc.koji.fedoraproject.org/kojifiles/packages/xorg-x11-server/1.17.1/2.fc22/data/logs/ppc64/build.log <- is this the build log for the xorg-x11-server that would have been in the build root for the most recent tigervnc build?

If so, it looks ok: "checking whether byte ordering is bigendian... yes"

That sets the IMAGE_BYTE_ORDER and BITMAP_BIT_ORDER macros that hw/vnc/* uses.

In which case it shifts the investigation to the over-the-wire data: so could you please run tcpdump to capture the RFB traffic when connecting?

tcpdump -U -s0 -w rfb.pcap port 5900

Comment 10 Jaromír Cápík 2015-04-08 16:26:05 UTC
Created attachment 1012297 [details]
f22-beta-tc2-vnc1.jpg

Comment 11 Jaromír Cápík 2015-04-08 16:27:06 UTC
Created attachment 1012299 [details]
f22-beta-tc2-vnc2.jpg

Comment 12 Jaromír Cápík 2015-04-08 16:27:30 UTC
Created attachment 1012300 [details]
f22-beta-tc2-vnc3.jpg

Comment 13 Jaromír Cápík 2015-04-08 16:27:47 UTC
I downgraded to tigervnc-server-1.3.1-11.fc22.ppc64 from a running system and it still shows wrong colors. But, some of the fluxbox themes are partially drawn with correct colors. However, I see also some strange artifacts in some cases. For example the default theme has some fonts mirrored from the right to the left. The green theme has fonts mirrored, but the colors are correct. See the attachments.

Comment 14 Dan Horák 2015-04-08 16:35:25 UTC
(In reply to Tim Waugh from comment #9)
> http://ppc.koji.fedoraproject.org/kojifiles/packages/xorg-x11-server/1.17.1/
> 2.fc22/data/logs/ppc64/build.log <- is this the build log for the
> xorg-x11-server that would have been in the build root for the most recent
> tigervnc build?
> 
> If so, it looks ok: "checking whether byte ordering is bigendian... yes"
> 
> That sets the IMAGE_BYTE_ORDER and BITMAP_BIT_ORDER macros that hw/vnc/*
> uses.

the build is
http://ppc.koji.fedoraproject.org/koji/buildinfo?buildID=298677
and has the endianity detected correctly

> In which case it shifts the investigation to the over-the-wire data: so
> could you please run tcpdump to capture the RFB traffic when connecting?
> 
> tcpdump -U -s0 -w rfb.pcap port 5900

Comment 15 Tim Waugh 2015-04-09 10:27:30 UTC
Need the TCP capture for further analysis.

Comment 16 Dan Horák 2015-04-09 10:44:28 UTC
Created attachment 1012647 [details]
packet capture for connecting to the XFCe session

You can also access the ppc64 machines we use for testing.

Comment 17 Tim Waugh 2015-04-09 11:53:13 UTC
This is an xorg-x11-server bug.

In include/servermd.h:

#if X_BYTE_ORDER == X_LITTLE_ENDIAN
#define IMAGE_BYTE_ORDER        LSBFirst
#define BITMAP_BIT_ORDER        LSBFirst
#elif X_BYTE_ORDER == X_BIG_ENDIAN
#define IMAGE_BYTE_ORDER        MSBFirst
#define BITMAP_BIT_ORDER        MSBFirst
#else
#error "Too weird to live."
#endif

We do know that X_BYTE_ORDER gets defined correctly, from this line in configure.ac:

AC_C_BIGENDIAN(AC_DEFINE(X_BYTE_ORDER, X_BIG_ENDIAN, [byte order]),
               AC_DEFINE(X_BYTE_ORDER, X_LITTLE_ENDIAN, [byte order]))

combined with this line in the build output:

checking whether byte ordering is bigendian... yes

In order words, we definitely have:

#define X_BYTE_ORDER X_BIG_ENDIAN

However, where is X_BIG_ENDIAN defined? Well, that comes from X11/Xarch.h. But is it included whenever servermd.h is? No.

by adjusting servermd.h as follows and rebuilding the problem becomes clear:

#if !defined(X_LITTLE_ENDIAN) || !defined(X_BIG_ENDIAN)
#error Include X11/Xarch.h before servermd.h
#endif

e.g.:

In file included from mibitblt.c:63:0:
../include/servermd.h:56:2: error: #error Include X11/Xarch.h before servermd.h
 #error Include X11/Xarch.h before servermd.h

and, crucially,

In file included from ../../randr/randrstr.h:46:0,
                 from XserverDesktop.h:45,
                 from vncExtInit.h:26,
                 from xvnc.cc:39:
../../include/servermd.h:56:2: error: #error Include X11/Xarch.h before servermd.h
 #error Include X11/Xarch.h before servermd.h

Changing component to xorg-x11-server as the problem seems not to be limited to tigervnc (see the mibitblt.c case above).

Comment 18 Adam Jackson 2015-04-09 14:25:23 UTC
I think it makes the most sense to just include Xarch.h from the config headers.  I'm building that change in xorg-x11-server-1.17.1-8, can you rebuild against that and test?

Comment 19 Tim Waugh 2015-04-09 15:26:58 UTC
Not sure how to rebuild against that candidate build for ppc64, so I've pulled in the relevant patch and have applied it explicitly in this build:
https://koji.fedoraproject.org/koji/buildinfo?buildID=627087

Comment 20 Tim Waugh 2015-04-09 15:47:57 UTC
Thanks Adam, works in my testing.

Comment 21 Dan Horák 2015-04-10 07:03:49 UTC
Ajax, yes, your fix in xorg-x11-server-1.17.1-8 works, please submit an update in bodhi.

Comment 22 Jakub Čajka 2015-04-21 12:55:15 UTC
Version 1.17.1-9.fc22.ppc64 is included in Fedora 22 ppc64 Beta_RC1, and it works correctly. Closing.