Bug 1650166

Summary: Xorg crash after RHEL7.6 update
Product: Red Hat Enterprise Linux 7 Reporter: amit yadav <ayadav>
Component: xorg-x11-serverAssignee: Adam Jackson <ajax>
Status: CLOSED ERRATA QA Contact: Desktop QE <desktop-qa-list>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.6CC: afox, ajax, alanm, amike, ayadav, cpippin, cww, dbasant, dkochuka, fernando, jkoten, jraising, jsolomon, jualvare, jwright, kyoneyam, mkolbas, ofourdan, pbhoot, poflynn, sfroemer, simon.rupf, syamamot, tpelka, vchoudha, vpakolu, yuokada, yzheng
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: xorg-x11-server-1.20.1-6.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1652915 1658972 (view as bug list) Environment:
Last Closed: 2019-08-06 12:42:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1658972    
Attachments:
Description Flags
Attachig Xorg logs captured by ABRT.
none
[PATCH v2] xfree86: LeaveVT from xf86CrtcCloseScreen none

Description amit yadav 2018-11-15 13:25:42 UTC
Description of problem:
After updating the RHEL7.5 systems to RHEL7.6, Xorg is crashing. We are getting the following backtracks on VMware guests:

0: /usr/bin/X (xorg_backtrace+0x55) [0x55eef1250185]
1: /usr/bin/X (0x55eef109f000+0x1b4e09) [0x55eef1253e09]
2: /lib64/libpthread.so.0 (0x7f1a3058f000+0xf5d0) [0x7f1a3059e5d0]
3: /usr/bin/X (RROutputIsLeased+0x9e) [0x55eef11b7cfe]
4: /usr/bin/X (xf86DPMSSet+0x10d) [0x55eef116974d]
5: /usr/lib64/xorg/modules/drivers/vmware_drv.so (0x7f1a2c77e000+0x10ab7) [0x7f1a2c78eab7]
6: /usr/lib64/xorg/modules/drivers/vmware_drv.so (0x7f1a2c77e000+0xe5b3) [0x7f1a2c78c5b3]
7: /usr/bin/X (0x55eef109f000+0xc88dc) [0x55eef11678dc]
8: /usr/bin/X (0x55eef109f000+0xe7bf8) [0x55eef1186bf8]
9: /usr/bin/X (0x55eef109f000+0x132d53) [0x55eef11d1d53]
10: /usr/bin/X (0x55eef109f000+0x604c4) [0x55eef10ff4c4]
11: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x7f1a301e43d5]
12: /usr/bin/X (0x55eef109f000+0x4a4ce) [0x55eef10e94ce]


Version-Release number of selected component (if applicable):
xorg-x11-server-Xorg-1.20.1-5.1.el7.x86_64
xorg-x11-drv-vmware-13.2.1-1.el7.1.x86_64
kernel-3.10.0-957.el7.x86_64

How reproducible:
Always(on customer side)

Steps to Reproduce:
1. Update RHEL7.5 systems and reboot

Actual results:
Xorg crash, getting blank screen

Expected results:
Xorg should not crash

Additional info:
I tried to reproduce the issue in our VMware lab environment, but I am unable to do that. I updated the system without any issue. We are getting multiple cases related to Xorg crash after RHEL7.6 update. Xorg is crashing with almost every xorg driver. This has greatly increased the volume of our total case. Hence, I am keeping this bug on urgent priority.

I didn't face this Xorg crash issue on system update, but when I am switching users in GUi session, Xorg is crashing with same traces.

I have not encountered this xorg crash problem on system update, but when I'm switching users in GUI session, the Xorg is crashing with the same backtrace.

Comment 2 amit yadav 2018-11-15 13:37:20 UTC
Can it be related to this bug?

https://bugs.freedesktop.org/show_bug.cgi?id=106960

Comment 3 Deepu K S 2018-11-15 13:40:35 UTC
This issue seems to be same as https://bugs.freedesktop.org/show_bug.cgi?id=106960

Comment 6 Olivier Fourdan 2018-11-21 12:00:26 UTC
Can you please provide the Xorg logs of an afefcted system?

Comment 7 Olivier Fourdan 2018-11-21 13:15:49 UTC
So, rebuilding the backtrace manually from the Xorg logs with the crash gives:

...
RROutputIsLeased()      0x118cfe at xorg-server-1.20.1/randr/../include/privates.h:136
xf86DPMSSet()            0xca74d at xorg-server-1.20.1/hw/xfree86/modes/xf86Crtc.c:3026
vmwgfx_disable_scanout() 0x10ab7 at xf86-video-vmware-13.2.1/vmwgfx/vmwgfx_crtc.c:116
drv_leave_vt()            0xe5b3 at xf86-video-vmware-13.2.1/vmwgfx/vmwgfx_driver.c:1213
xf86CrtcCloseScreen()    0xc88dc at xorg-server-1.20.1/hw/xfree86/modes/xf86Crtc.c:770
CursorCloseScreen()      0xe7bf8 at xorg-server-1.20.1/xfixes/cursor.c:206
present_close_screen()  0x132d53 at xorg-server-1.20.1/present/present_screen.c:71
dix_main()               0x604c4
_start()                 0x4a4ce

That shows the crash occurs when closing the screen, i.e. when the Xserver terminates, not a crash that kills the Xserver in the middle of a running session.

I agree that this crash looks like upstream bug fdo#106960, but the fix for this bug is already included in xserver-1.20.1 so most likely the fix upstream is incomplete: https://gitlab.freedesktop.org/xorg/xserver/commit/101d15c

RROutputIsLeased() reads:

120 Bool
121 RROutputIsLeased(RROutputPtr output)
122 {
123     ScreenPtr screen = output->pScreen;
124     rrScrPrivPtr scr_priv = rrGetScrPriv(screen);
125     RRLeasePtr lease;
126     int o;
127 
128     xorg_list_for_each_entry(lease, &scr_priv->leases, list) {
129         for (o = 0; o < lease->numOutputs; o++)
130             if (lease->outputs[o] == output)
131                 return TRUE;
132     }
133     return FALSE;
134 }

The crash occurs in dixGetPrivate() so my guess is that the output is already freed/corrupted when we get there and therefore output->pScreen is invalid.

Comment 8 dinesh babu ramayanam 2018-11-22 10:47:36 UTC
Created attachment 1507912 [details]
Attachig Xorg logs captured by ABRT.

Comment 19 Olivier Fourdan 2018-11-23 13:59:06 UTC
Created attachment 1508283 [details]
[PATCH v2] xfree86: LeaveVT from xf86CrtcCloseScreen

This is an _updated_ version of the original patch included in el7.6 to fix bug 1635747 / bug 1489977 (i.e. that patch replaces the one wit hthe same name in the xorg-x11-server downstream package)

It keeps the call to `LeaveVT()` from `xf86CrtcCloseScreen()` but does it *after*  clearing up the output data so that in a DDX such as xf86-video-vmware calls back into DPMS in its `LeaveVT()` handler, we don't end up accessing freed data.

Comment 52 A. Fernando 2019-06-28 05:21:01 UTC
My x-server crashes when I attempt to run xfreerdp command. 
I have already published my backtraces - https://bugzilla.redhat.com/show_bug.cgi?id=1659113

Comment 53 A. Fernando 2019-06-28 05:23:53 UTC
Would you be able to direct me to the change logs for the new xorg-x11-server-1.20.1-6.el7 ?

Thanks

Comment 54 Olivier Fourdan 2019-06-28 06:27:27 UTC
(In reply to A. Fernando from comment #52)
> My x-server crashes when I attempt to run xfreerdp command. 

xfreerdp crashing the Xserver is a different issue from this bug, already tracked in bug 1659113, let's not mix those.

Comment 56 errata-xmlrpc 2019-08-06 12:42:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2079