Bug 2073863

Summary: [ESXi][RHEL-8.6] Failed to login RHEL-8.6.0 GA VM on ESXi 7.0 GA host
Product: Red Hat Enterprise Linux 8 Reporter: Bo Yang <boyang>
Component: gnutlsAssignee: Daiki Ueno <dueno>
Status: CLOSED CURRENTRELEASE QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.6CC: cavery, dueno, fmuellner, jadahl, ldu, tpopela, yacao
Target Milestone: betaKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-21 09:55:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2116610    
Bug Blocks:    
Attachments:
Description Flags
logs under /var.
none
gnome-shell log none

Description Bo Yang 2022-04-11 00:07:16 UTC
Created attachment 1871686 [details]
logs under /var.

Description of problem:
Our partner VMware team is testing a RHEL-8.6.0-20220302.5 Beta on their ESXi 7.0.0 15843807 GA, after installation, rebooting, they can't login the VM via GUI.


Version-Release number of selected component (if applicable):
Host: VMware ESXi, 7.0.0, 15843807
Guest: RHEL-8.6.0-20220302.5


How reproducible:
80%~


Steps to Reproduce:
1. Install a RHEL-8.6.0-20220302.5 Beta as the guest in above ESXi.
2. After the installation and rebooting, try to login this VM with GUI.


Actual results:
1. After step 2, Failed to login the VM.


Expected results:
2. After step 2, login the VM well.


Additional info:
1. SSH works well.
2. Check the logs under /var/logs with attachment.

Comment 1 Florian Müllner 2022-05-11 12:01:48 UTC
gnome-shell crashes:

> Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4
> Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session[7773]: gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4

Any chance to get a backtrace of that crash?

Comment 2 Tomas Popela 2022-06-15 11:53:49 UTC
(In reply to Florian Müllner from comment #1)
> gnome-shell crashes:
> 
> > Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4
> > Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session[7773]: gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4
> 
> Any chance to get a backtrace of that crash?

Setting needinfo for this. Also Bo, would it be possible to test with the up to date 8.6.0?

Comment 3 Bo Yang 2022-06-22 01:15:12 UTC
(In reply to Tomas Popela from comment #2)
> (In reply to Florian Müllner from comment #1)
> > gnome-shell crashes:
> > 
> > > Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4
> > > Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session[7773]: gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4
> > 
> > Any chance to get a backtrace of that crash?
> 
> Setting needinfo for this. Also Bo, would it be possible to test with the up
> to date 8.6.0?


Sure, will contact VMware team to handle this.

Comment 4 ldu 2022-06-28 10:04:19 UTC
Yan,
Could you help confirm whether the RHEL 8.6 GA still have this issue on this hardware?

Thanks,
Lili Du

Comment 5 Yan Jin 2022-06-29 03:34:50 UTC
(In reply to ldu from comment #4)
> Yan,
> Could you help confirm whether the RHEL 8.6 GA still have this issue on this
> hardware?
> 
> Thanks,
> Lili Du

Yes, the issue still exists when running RHEL 8.6 GA (Server with GUI) on ESXi 7.0 GA

Comment 6 Bo Yang 2022-07-21 01:58:10 UTC
(In reply to Florian Müllner from comment #1)
> gnome-shell crashes:
> 
> > Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4
> > Apr  7 23:28:27 pek2-gosv-16-dhcp232 gnome-session[7773]: gnome-session-binary[7773]: WARNING: Application 'org.gnome.Shell.desktop' killed by signal 4
> 
> Any chance to get a backtrace of that crash?
From partner's re-testing, "the issue still exists when running RHEL 8.6 GA (Server with GUI) on ESXi 7.0 GA".
If logs.zip (attachment) doesn't contain the backtrace what you want, could you please point out where or how can partner capture them?

Comment 7 Florian Müllner 2022-08-12 11:23:22 UTC
(In reply to Bo Yang from comment #6)

> If logs.zip (attachment) doesn't contain the backtrace what you want

It doesn't contain information beyond the bit that I quoted above.

> could you please point out where or how can partner capture them?

 1. install debug symbols for glib, mutter and gnome-shell
 2. a) if using systemd, coredumpctl should capture the crash, and
       you can access it via 'coredumpctl gdb gnome-shell'
    b) otherwise, log into the system remotely (with ssh for example),
       then attach gdb to the gnome-shell process before reproducing
       the crash
3. get the backtrace from gdb:
    (gdb) thread apply all backtrace

Note on 2: Use coredumpctl if you can. Debugging gnome-shell is a bit tricky, because it composes the screen (read: stopping gnome-shell will stop updating the screen, so the screen contents will freeze). When using wayland, the "freeze" is more complete and includes the shortcuts to switch to a tty.

Comment 8 Bo Yang 2022-08-22 07:47:16 UTC
Great supporting for this part update.



YAN JIN, sorry late to needinfo you to review above steps to capture the backgrace, if possible and free, please help to verify these steps.

Comment 9 Yan Jin 2022-09-01 04:16:06 UTC
Created attachment 1908862 [details]
gnome-shell log

Comment 10 Florian Müllner 2022-10-21 12:37:26 UTC
(In reply to Yan Jin from comment #9)
> Created attachment 1908862 [details]
> gnome-shell log

Thanks, this is helpful.

The crash is caused by an illegal instruction deep in gnutls/glib-networking code.

Comment 11 Bo Yang 2022-10-26 05:34:32 UTC
(In reply to Florian Müllner from comment #10)
> (In reply to Yan Jin from comment #9)
> > Created attachment 1908862 [details]
> > gnome-shell log
> 
> Thanks, this is helpful.
> 
> The crash is caused by an illegal instruction deep in gnutls/glib-networking
> code.


Thanks for your debugging.

@dueno please help to check this issue. Any requirement, please tell us. Thanks.

Comment 13 Bo Yang 2022-11-15 06:36:19 UTC
Hello YAN JIN,


This issue might be the same issue as bug 2116610, a hotfix build about gnutls has been uploaded to https://people.redhat.com/~ldu/rpm/ .
If you are free, please help to try this fix.


Thanks a loooot.

Comment 14 Bo Yang 2022-11-16 02:12:37 UTC
As YAN JIN BZ account issue, I paste her mail about the verification.


'''
The gnutls package works.
My steps:
1.Reproduce the issue with RHEL 8.7 VM on ESXi 7.0 GA host
2. ssh login the VM
3. rpm --force -i gnutls-3.6.16-4.el8_6.0.0.hotfix.1.bz2131152.x86_64.rpm and reboot
The VM can be logged in successfully
Thanks for the fixing
'''


@dueno
Thanks for providing this suggestion. If possible, please help to update the BZ status.

Comment 16 Bo Yang 2022-11-16 09:26:09 UTC
@dueno

Thanks for your suggestion. 
YAN JIN is our partner (VMware), have synced with team about this BZ status, want to close it after official fix.
Then we can sent the final build to YAN JIN to verify and close it.