Bug 1846133

Summary: [rhel-7.9] skylake (06-4e-03) microcode update hangs
Product: Red Hat Enterprise Linux 7 Reporter: Eugene Syromiatnikov <esyr>
Component: microcode_ctlAssignee: Eugene Syromiatnikov <esyr>
Status: CLOSED ERRATA QA Contact: Jeff Bastian <jbastian>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.9CC: ccheney, jbastian, pvlasin, skozina
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: microcode_ctl-2.1-73.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1846119 Environment:
Last Closed: 2020-09-29 20:12:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eugene Syromiatnikov 2020-06-10 21:03:25 UTC 
+++ This bug was initially created as a clone of Bug #1846119 +++

Description of problem:
Microcode revision 0xdc for CPU 06-4e-03 (Skylake) can cause the system to freeze when applying the early microcode update.  Late updates seem to be ok, at least on this system.

[root@intel-skylake-y-01 ~]# lscpu | egrep -i -e family -e model -e stepping
CPU family:          6
Model:               78
Model name:          Intel(R) Core(TM) m5-6Y54 CPU @ 1.10GHz
Stepping:            3

[root@intel-skylake-y-01 ~]# cat /sys/devices/system/cpu/cpu0/microcode/version 
0x2d

[root@intel-skylake-y-01 ~]# yum -y update microcode_ctl
...

[root@intel-skylake-y-01 ~]# rpm -q microcode_ctl
microcode_ctl-20191115-4.20200602.2.el8_2.x86_64

[root@intel-skylake-y-01 ~]# rpm -q --provides microcode_ctl | grep iucode_rev.*06-4e-03
iucode_rev(fname:intel/06-4e-03;cpuid:000406e3;pf_mask:0xc0) = 0xdc
iucode_rev(fname:intel/06-4e-03;platform_id:0x40) = 0xdc
iucode_rev(fname:intel/06-4e-03;platform_id:0x80) = 0xdc

[root@intel-skylake-y-01 ~]# echo 1 > /sys/devices/system/cpu/microcode/reload

[root@intel-skylake-y-01 ~]# cat /sys/devices/system/cpu/cpu0/microcode/version 
0xdc

[root@intel-skylake-y-01 ~]# dracut --force --early-microcode

[root@intel-skylake-y-01 ~]# reboot
...

<<<HANG AFTER GRUB SCREEN>>>


Version-Release number of selected component (if applicable):
microcode_ctl-20191115-4.20200602.2.el8_2.x86_64

How reproducible:
always

Steps to Reproduce:
1. install RHEL-8.2.0
2. apply z-stream update to microcode_ctl-20191115-4.20200602.2.el8_2.x86_64
3. dracut --force --early-ucode
4. reboot

Actual results:
hang after grub

Expected results:
normal boot

Additional info:
Upstream bug report:
https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/31
Comment 1 Eugene Syromiatnikov 2020-06-16 17:44:12 UTC 
06-5e-03 is to be blacklisted as well per [1].

[1] https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/31#issuecomment-644885826
Comment 11 Jeff Bastian 2020-06-24 22:15:04 UTC 
Verified with microcode_ctl-2.1-73.el7.

By default, the microcode update is blocked for 06-4e-03 (the "fail" result indicates the CPU is not running the latest microcode revision) which is good since it prevents the hangs:
https://beaker.engineering.redhat.com/recipes/8478437#tasks

If I force the update, the system hangs (as seen in the console.log), and I was only able to boot the system by adding dis_ucode_ldr to the kernel args:
https://beaker.engineering.redhat.com/recipes/8478438#tasks
Comment 13 errata-xmlrpc 2020-09-29 20:12:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (microcode_ctl bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3968