RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1532283 - System hang during boot following update to microcode_ctl-1.17-25.2.el6_9.x86_64
Summary: System hang during boot following update to microcode_ctl-1.17-25.2.el6_9.x86_64
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: microcode_ctl
Version: 6.9
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: rc
: 6.10
Assignee: Petr Oros
QA Contact: Rachel Sibley
URL:
Whiteboard:
Depends On:
Blocks: 1425544
TreeView+ depends on / blocked
 
Reported: 2018-01-08 15:02 UTC by Kyle Walker
Modified: 2021-06-10 14:07 UTC (History)
15 users (show)

Fixed In Version: microcode_ctl-1.17-25.3.el6_9.x86_64
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-21 12:45:11 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1533978 1 None None None 2021-06-10 14:10:41 UTC
Red Hat Knowledge Base (Solution) 3314661 0 None None None 2018-01-09 11:10:59 UTC

Internal Links: 1533978

Description Kyle Walker 2018-01-08 15:02:15 UTC 
Description of problem:
  Following an update of the microcode_ctl package, a system hangs during boot with messages related to the microcode load operation. Observed at this time only on systems with the following CPU model information:

  $ awk '/model/||/stepping/||/family/||/microcode/ {if(!seen[$0]++) print $0}' proc/cpuinfo
  cpu family	: 6
  model		: 79
  model name	: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
  stepping	: 1
  microcode	: 184549407


Version-Release number of selected component (if applicable):
  1:microcode_ctl-1.17-25.2.el6_9.x86_64

How reproducible:
  Easily

Steps to Reproduce:
1. On a system with the above CPU information, install a base RHEL 6.9 deployment 
2. Issue a "yum update"
3. Reboot

Actual results:
  <snip>
  microcode: CPU0 sig=0x406f1, pf=0x1, revision=0xb00001f
  platform microcode: firmware: requesting intel-ucode/06-4f-01
  <snip>
    ^- Hangs here with no further output visible

Expected results:
  A normal boot operation with no hang observed.

Additional info:
  A downgrade of the microcode to the previous revision resolves the hang.
Comment 6 Ion Badulescu 2018-01-11 17:02:07 UTC 
Same here after the update. Downgrading the microcode fixes the issue.

Hardware platform is:

Supermicro X10DDW-i with 2x E5-2667v4 CPUs stepping 1, BIOS 2.0a from 8/17/2016 (the latest available.)

Incidentally, it appears that various kernels may or may not cause the hang to happen, and also the behavior is not consistent between machines with otherwise identical hardware.

Case in point:

- on one machine with the above specs, booting kernel-2.6.32-696.16.1.el6.x86_64 causes a hard hang when loading the microcode.
- on the same machine, booting kernel-2.6.32-696.18.7.el6.x86_64 causes a hard reset when loading the microcode.
- on the same machine, booting a custom (locally built) kernel based on 3.10.107 boots up fine.

However:

- on another machine with the above specs, booting the custom 3.10.107-based kernel causes a hard hang.

Downgrading the microcode fixes the problem in all the problem cases encountered.
Comment 12 Oliver Freyermuth 2018-01-12 14:26:47 UTC 
We also see this on downstream distros, e.g. CentOS 6, SL 6, CentOS 7 etc. 

Checking the version of 06-4f-01, it seems the revision packaged in RHEL6 is
0xb000025
while the last revision released officially by Intel in the last microcode package from 2018-01-08 is
0xb000021

So I understand some pre-production version has been included and shipped to enterprise customers? 
Debian, Gentoo and others still ship 0xb000021 (as released officially by Intel)...
Comment 15 Ion Badulescu 2018-01-12 18:46:23 UTC 
Guidance we've received from Intel directly suggests that they're aware of problems in the 0xb000025 firmware for Broadwell E/EP (as well as in other firmware revisions for other CPUs) and they recommend delaying the deployment of this firmware into production.
Comment 18 Bill Verzal 2018-01-17 15:54:40 UTC 
So we applied this on ~1300 servers last week (a mix of physical and virtual).

What is the impact on ESX based OS instances?
Note You need to log in before you can comment on or make changes to this bug.