Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
(In reply to Kyle Walker from comment #0)
> 2018-08-14 09:56:51 microcode: CPU29 sig=0x406f1, pf=0x1, revision=0xb000017
...
> microcode : 184549407
Is this correct? There is a discrepancy here: 184549407 = 0xb00001f
So, the kernel log shows the version is 0xb000017 when booting, but somehow it got updated to 0xb00001f according to cpuinfo, and yet the new microcode_ctl package has version 0xb00002e.
I'm confused how 0xb00001f entered the picture.
For context, see bug 1574592 comment 20 which was the bz to apply Intel's patches to fix the hanging issue on Broadwell-EP microcode updates.
Kernel 2.6.32-754.1.1.el6 safely and successfully updated the microcode from 0xb000019 to 0xb00002e in our QE testing.
Is there possibly still an issue updating from the slightly older 0xb000017...?
Rachel and I re-tested today and we cannot reproduce this on our 06-4f-01 CPUs which have microcode 0xb000019 in the BIOS. (Unfortunately Beaker doesn't record microcode versions so it's difficult to find one with 0xb000017.)
What type of systems are the customers using? Can we get sosreports from the systems that hang? That will help us to check if we have a similar system in Beaker.
~ ~ ~
Testing notes
From https://beaker.engineering.redhat.com/recipes/5575947/logs/console.log
intel-wildcatpass-07.khw.lab.eng.bos.redhat.com
RHEL-6.10-updates-20180809.0 Server x86_64
kernel-2.6.32-754.3.5.el6.x86_64
microcode_ctl-1.17-33.3.el6_10.x86_64
microcode: CPU0 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
microcode: CPU1 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
microcode: CPU2 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
microcode: CPU3 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
microcode: CPU4 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
...
microcode: CPU85 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
microcode: CPU86 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
microcode: CPU87 sig=0x406f1, pf=0x1, revision=0xb000019
platform microcode: firmware: requesting intel-ucode/06-4f-01
Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba
microcode: CPU0 updated to revision 0xb00002e, date = 2018-04-19
microcode: CPU1 updated to revision 0xb00002e, date = 2018-04-19
microcode: CPU2 updated to revision 0xb00002e, date = 2018-04-19
microcode: CPU3 updated to revision 0xb00002e, date = 2018-04-19
microcode: CPU4 updated to revision 0xb00002e, date = 2018-04-19
....
microcode: CPU85 updated to revision 0xb00002e, date = 2018-04-19
microcode: CPU86 updated to revision 0xb00002e, date = 2018-04-19
microcode: CPU87 updated to revision 0xb00002e, date = 2018-04-19
Also, if the sosreport does not include it, please ask which BIOS version(s) are on the systems that hang. We may need to upgrade/downgrade the BIOS on a Beaker system in order to get a reproducer.
Description of problem: System hang during boot following an update to both the kernel and microcode_ctl package indicated below: Version-Release number of selected component (if applicable): kernel-2.6.32-754.3.5.el6.x86_64 microcode_ctl-1.17-33.3.el6_10.x86_64 How reproducible: Easily, on specific customer hardware Steps to Reproduce: 1. Update the kernel and microcode to the versions indicated 2. Reboot 3. Actual results: System hangs with output similar to: 2018-08-14 09:56:51 microcode: CPU29 sig=0x406f1, pf=0x1, revision=0xb000017 2018-08-14 09:56:51 platform microcode: firmware: requesting intel-ucode/06-4f-01 2018-08-14 09:56:51 microcode: CPU30 sig=0x406f1, pf=0x1, revision=0xb000017 2018-08-14 09:56:51 platform microcode: firmware: requesting intel-ucode/06-4f-01 2018-08-14 09:56:51 microcode: CPU31 sig=0x406f1, pf=0x1, revision=0xb000017 2018-08-14 09:56:51 platform microcode: firmware: requesting intel-ucode/06-4f-01 2018-08-14 09:56:51 Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba 2018-08-14 09:56:51 microcode: CPU0 updated to revision 0xb00002e, date = 2018-04-19 <hangs here> Expected results: Continued successful boot Additional info: Due to the newly introduced check_caveats mechanism, booting into the kernel-2.6.32-754.el6.x86_64 version avoids the problem, but doesn't address it in full. Example cpuinfo: $ awk '/processor.* 0$/,/^$/' proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz stepping : 1 microcode : 184549407 cpu MHz : 3399.975 cache size : 20480 KB physical id : 0 siblings : 12 core id : 0 cpu cores : 6 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch arat epb xsaveopt pln pts dtherm invpcid_single pti retpoline tpr_shadow vnmi flexpriority ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx cqm_llc cqm_occup_llc bogomips : 6799.95 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: