Can you please provide your kernel version, CPU model name, and output of "dmesg | grep -i microcode"?

kernel: 3.10.0-862.11.6.el7.x86_64

microcode: microcode updated early to revision 0x84, date = 2018-01-21
microcode: sig=0x806e9, pf=0x80, revision=0x84
microcode: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba

cpu: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz

(In reply to Petr Šplíchal from comment #4)
> kernel: 3.10.0-862.11.6.el7.x86_64
> 
> microcode: microcode updated early to revision 0x84, date = 2018-01-21
> microcode: sig=0x806e9, pf=0x80, revision=0x84
> microcode: Microcode Update Driver: v2.01 <tigran.co.uk>,
> Peter Oruba

I presume that the output is for the older version of microcode_ctl installed (as the latest version provides revision 0x8e of the 06-8e-09 microcode). Anyway, it looks like the issue is within the fact that due to [1] and [2], microcode_ctl package avoids including microcode to initramfs if kernel doesn't contain a fix for [3] (in case of 7.5.z it's versions of the kernel package older than 3.10.0-862.14.1), and microcode that has been loaded late (via a sysfs trigger[4]) can't be loaded again early on resume[5], as it is not saved in-kernel (contrary to a microcode that is loaded from initramfs[6]), and t470s probably can't resume properly without an updated microcode (if mere difference in microcode revisions and not just t470s system firmware specifics leads to that result, that would be pretty unfortunate, though). If that's correct, then either one of the following actions may help:
 * Updating to a kernel that has version 3.10.0-862.14.1 or newer;
 * Creating an override file, /etc/microcode_ctl/ucode_with_caveats/force-early-intel , for example (see [7] or /usr/share/doc/microcode_ctl-2.1 for details), and regenerate initramfs with "dracut -f" (with additional "--kver TARGET_KERNEL_VERSION" if needed).

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1596627
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1596627#c77
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1607899
[4] http://git.app.eng.bos.redhat.com/git/rhel-7.5.z.git/tree/arch/x86/kernel/cpu/microcode/core.c?id=054d3e911370a64a3bcbb9c62fa8f79bbce14ebf#n539
[5] http://git.app.eng.bos.redhat.com/git/rhel-7.5.z.git/tree/arch/x86/kernel/cpu/microcode/core.c?id=054d3e911370a64a3bcbb9c62fa8f79bbce14ebf#n671
[6] http://git.app.eng.bos.redhat.com/git/rhel-7.5.z.git/tree/arch/x86/mm/init.c?id=cae57bba8a7dd8d1fd178e8ba46bc01269bc075f
[7] http://pkgs.devel.redhat.com/cgit/rpms/microcode_ctl/tree/README.caveats?h=rhel-7.6

Created attachment 1484004 [details]
KBL-U microcode from microcode-20180703.tgz

I forgot one more thing to check: in order to verify that the regression hasn't been caused by a microcode update, may I ask you to try to replace /usr/share/microcode_ctl/ucode_with_caveats/early/intel-ucode/06-8e-09 microcode file with the one provided in the attachment (without enforcing addition of microcode files to initramfs and/or kernel update)? I expect that it won't change anything.

Do you mean to update to the latest microcode_ctl package, then
overwrite the attached file and try suspend/wake without reboot?

(In reply to Petr Šplíchal from comment #8)
> Do you mean to update to the latest microcode_ctl package, then
> overwrite the attached file and try suspend/wake without reboot?

Yes.

There is no /usr/share/microcode_ctl/ucode_with_caveats/early/intel-ucode/06-8e-09 file.
But I found /usr/share/microcode_ctl/ucode_with_caveats/intel/intel-ucode/06-8e-09 and
replaced it instead. After suspending the system I could not wake it up. After reboot it
was still broken.

(In reply to Petr Šplíchal from comment #10)
> There is no
> /usr/share/microcode_ctl/ucode_with_caveats/early/intel-ucode/06-8e-09 file.
> But I found
> /usr/share/microcode_ctl/ucode_with_caveats/intel/intel-ucode/06-8e-09 and
> replaced it instead. After suspending the system I could not wake it up.
> After reboot it
> was still broken.

Thanks, then it's definitely not the microcode itself. Have you an opportunity to verify that workarounds in [1] work, by chance?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1628958#c6

I've got suspend/wake working back by the following steps:

    * Install kernel-3.10.0-862.14.1.el7.x86_64.rpm
    * Downgrade, upgrade microcode_ctl
    * Reboot the machine

Without downgrade/reboot the issue was still present.

(In reply to Petr Šplíchal from comment #12)
> I've got suspend/wake working back by the following steps:
> 
>     * Install kernel-3.10.0-862.14.1.el7.x86_64.rpm
>     * Downgrade, upgrade microcode_ctl
>     * Reboot the machine
> 
> Without downgrade/reboot the issue was still present.

Reboot (or at least kexec into the new kernel) is needed indeed, in order to get the new kernel loaded, but the necessity of downgrade/upgrade sounds strange, placing of the microcode during initramfs generation for the new kernel is handled by the dracut module.

OK, I think that it is confirmed that the issue is within absence of microcode in initramfs.

I have not had the time to investigate thoroughly but a Lenovo T460s also refused to wake up from suspend with microcode_ctl-2.1-29.16.el7_5.x86_64.

Downgrading back to microcode_ctl-2.1-29.10.el7_5.x86_64 fixed it.

(In reply to Eugene Syromiatnikov from comment #13)
> Reboot (or at least kexec into the new kernel) is needed indeed,
> in order to get the new kernel loaded, but the necessity of
> downgrade/upgrade sounds strange, placing of the microcode
> during initramfs generation for the new kernel is handled by the
> dracut module.

I tried first to only update the kernel and reboot but the issue
was still present. Then did downgrade/upgrade, but it was still
present. Only after one more reboot I got it working.

From what I can see it seems that it's necessary to run the
microcode_ctl rpm scripts with the updated kernel running.

I had to "yum downgrade microcode_ctl" ... that changed my microcode_ctl-2.1-29.16.el7_5.x86_64 to microcode_ctl-2.1-29.10.el7_5.x86_64, then reboot and all got back to normal. Now I can wake up from suspend without problems.

Hardware: Lenovo Thinkpad W510
CPU: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
Kernel: 3.10.0-862.11.6.el7.x86_64

Same for me on two different systems:

Desktop: Mobo: ASUSTeK model: H97-PLUS
Linux  3.10.0-862.11.6.el7.x86_64
CentOS Linux release 7.5.1804

Notbook Lenovo T430: 
Linux  3.10.0-862.11.6.el7.x86_64
CentOS Linux release 7.5.1804

After sleep, both got back to power, but did not show any image on the screens and were not reachable via network.

It helped:
yum downgrade microcode_ctl
(which means: microcode_ctl-2.1-29.16.el7_5.x86_64 
           -> microcode_ctl-2.1-29.10.el7_5.x86_64)


Thanks!

Same here on Dell Inspiron 3558 (core i3-5015U): unable to wake up after updating to microcode_ctl-2.1-29.16

I have tested today in a Lenovo T430s and RHEL 7.6 Workstation with two kernels: 

- kernel 3.10.0-862.11-el7
- kernel 3.10.0-953.el7

and microcode_ctl-2.1-47.el7.

Step 1. Run 'dracut --force --early-microcode' and reboot, then try to suspend and resume.
Step 2. Run 'dracut --force --no-early-microcode' and reboot and try to suspend and resume again.

To suspend, the power button is pressed. To resume, the power button is pressed again.

- With kernel 862, both step 1 and 2 lead to a reboot just after resume.
- With kernel 953, step 1 is able to resume correctly. But step 2 (no-early-microcode) leads to a reboot.

In summary, only with newest kernel and --early-microcode resume is done correctly.

(In reply to Hayrol Reyes from comment #18)
> I had to "yum downgrade microcode_ctl" ... that changed my
> microcode_ctl-2.1-29.16.el7_5.x86_64 to
> microcode_ctl-2.1-29.10.el7_5.x86_64, then reboot and all got back to
> normal. Now I can wake up from suspend without problems.
> 
> Hardware: Lenovo Thinkpad W510
> CPU: Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
> Kernel: 3.10.0-862.11.6.el7.x86_64

This worked for me as well:
"yum downgrade microcode_ctl" went from & to the same versions as in comment 18.
Hardware: Lenovo Thinkpad X220
CPU: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
Kernel: 3.10.0-862.11.6.el7.x86_64

this is a series microcode bug. 
not many people know how to downgrade a package ( if they find this post at all).

Lenovo T440s, RHEL-7.5, kernel 3.10.0-862.14.4.el7.x86_64 .

Same problem, laptop dead once it goes to sleep mode, only option is to hold power button to shutdown and turn on again. Very frustrating.

Fixed using Step 1 from https://bugzilla.redhat.com/show_bug.cgi?id=1628958#c25 :

"Step 1. Run 'dracut --force --early-microcode' and reboot, then try to suspend and resume."

Laptop can wake up from sleep mode fine now.

Previous microcode signature was from 2018-01-18, now it is:
root $ dmesg | grep -i microcode
[    0.000000] microcode: microcode updated early to revision 0x24, date = 2018-04-02
[    1.856682] microcode: sig=0x40651, pf=0x40, revision=0x24
[    1.856868] microcode: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba

Note: with older kernel 3.10.0-862.9.1.el7 I hadn't such issue, it seems to me like this issue started when upgraded to 3.10.0-862.11.6.el7 (or because of related packages pulled during update).

Uninstall/reinstall of microcode_ctl package had no effect on this issue.

Lenovo X1 Carbon 3rd gen
Intel(R) Core(TM) i5-5200U CPU @ 2.20GH
latest BIOS 1.27

kernel 3.10.0-957.1.3.el7.x86_64

Sometimes after waking up from suspend back to power, no image on sceen and no response. I must hold power button to power off and power on it again.

It's random. Sometimes this happens after first suspend, sometimes after 10th suspend.

I see the issue on 2 Systems as well:

Model: T460s and X1 Carbon 3rd Gen
Kernel: 3.10.0-957.1.3.el7.x86_64

T460s:
dmesg | grep -i microcode
[    0.723182] microcode: sig=0x406e3, pf=0x80, revision=0xc6
[    0.723305] microcode: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba


X1 Carbon 3rd Gen:
dmesg | grep -i microcode
[    0.000000] microcode: microcode updated early to revision 0x25, date = 2017-01-27
[    0.646675] microcode: sig=0x306d4, pf=0x40, revision=0x25
[    0.646897] microcode: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba
[   17.181885] microcode: updated to revision 0x2b, date = 2018-03-22


# suspend and resume fail
microcode_ctl-2.1-47.el7.x86_64

# suspend and resume 
microcode_ctl-2.1-29.16.el7_5.x86_64

Verified on the T460s at least, a BIOS update is working as well (to the latest version), however, we can not do that manually on all possible affected notebooks

What would be the recommended fix/workaround for a larger deployment?

(In reply to Oliver Haessler from comment #46)
> I see the issue on 2 Systems as well:
> 
> Model: T460s and X1 Carbon 3rd Gen
> Kernel: 3.10.0-957.1.3.el7.x86_64
> 
> T460s:
> dmesg | grep -i microcode
> [    0.723182] microcode: sig=0x406e3, pf=0x80, revision=0xc6
> [    0.723305] microcode: Microcode Update Driver: v2.01
> <tigran.co.uk>, Peter Oruba

It is strange that the issue occurs on T460s, as the microcode version shipped with microcode_ctl-2.1-47.el7.x86_64 is exactly 0xc6, so with both 2.1-47 and 2.1-29 there shouldn't be any microcode update at all (and any difference in behaviour, as a result).

> X1 Carbon 3rd Gen:
> dmesg | grep -i microcode
> [    0.000000] microcode: microcode updated early to revision 0x25, date =
> 2017-01-27
> [    0.646675] microcode: sig=0x306d4, pf=0x40, revision=0x25
> [    0.646897] microcode: Microcode Update Driver: v2.01
> <tigran.co.uk>, Peter Oruba
> [   17.181885] microcode: updated to revision 0x2b, date = 2018-03-22

This is strange. It looks like initramfs contains some stale microcode (revision 0x25 of microcode file 06-3d-04 was shipped up until microcode_ctl-2.1-27.el7.x86_64, where it was updated to revision 0x28), can you please describe the steps this configuration has been achieved?

And I can't reproduce a situation where initramfs for kernel  3.10.0-957.1.3 is not updated with a default microcode_ctl installation.

> What would be the recommended fix/workaround for a larger deployment?

There are some configuration/override mechanisms described in /usr/share/doc/microcode_ctl/README.caveats, I think the easiest one would be to create a file named "/etc/microcode_ctl/ucode_with_caveats/force-early" and trigger an initramfs regeneration (by running "dracut -f", for example).

Oliver, any thoughts on Comment#47 please?

Not sure if this is still the same bug but I've been affected by it since last year when the microcode_ctl update was pointed out as the culprit; adding more information about my specific setup in case it is helpful for troubleshooting this.


The situation:

System doesn't resume from suspend to RAM; this behaviour doesn't always happen, sometimes it will work, but more often than not it will fail, showing a black screen and being unresponsive to anything (hard reset required).


System was upgraded recently to 7.6 (partially to see if it would fix this issue):

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Workstation release 7.6 (Maipo)

$ uname -a
Linux aldebaran.ibm.com 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 x86_64 GNU/Linux


It's a T460 laptop:

$ dmidecode --type 1
System Information
        Manufacturer: LENOVO
        Product Name: 20FMS50T1J
        Version: ThinkPad T460
        Serial Number: PC0HRVXZ
        UUID: 9680d54c-2bd8-11b2-a85c-bb2274ca0559
        Wake-up Type: Power Switch
        SKU Number: LENOVO_MT_20FM_BU_Think_FM_ThinkPad T460
        Family: ThinkPad T460


Microcode information:

$ rpm -qi microcode_ctl 
Name        : microcode_ctl
Epoch       : 2
Version     : 2.1
Release     : 47.el7
Architecture: x86_64
Install Date: Wed 13 Feb 2019 01:15:37 PM WET

$ dmesg | grep -i microcode
[    1.224312] microcode: sig=0x406e3, pf=0x80, revision=0xc6
[    1.224430] microcode: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba


BIOS: I have recently updated the BIOS using the Lenovo supplied CD (using an USB stick and geteltorito etc).

$ dmesg | grep " BIOS "
[    0.000000] DMI: LENOVO 20FMS50T1J/20FMS50T1J, BIOS R06ET66W (1.40 ) 01/25/2019
[    0.043943] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.043944] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.114891] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[   16.696654] thinkpad_acpi: ThinkPad BIOS R06ET66W (1.40 ), EC unknown


I haven't changed anything in the BIOS setting (maybe I should, but apart from toggling UEFI/Legacy for the update process I left it as is).


Xorg: I have acceleration enabled via the Intel driver, mentioning it because I have also saw this mentioned sometimes by those affected.

$ glxinfo -B
name of display: :0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Intel Open Source Technology Center (0x8086)
    Device: Mesa DRI Intel(R) HD Graphics 520 (Skylake GT2)  (0x1916)
    Version: 18.0.5
    Accelerated: yes
    Video memory: 3072MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 3.0
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) HD Graphics 520 (Skylake GT2) 
OpenGL core profile version string: 4.5 (Core Profile) Mesa 18.0.5
OpenGL core profile shading language version string: 4.50


I can provide more information if needed, or try anything that could help in identifying the cause (and hopefully a fix).

(In reply to Frederico Muñoz from comment #54)
[...]
> $ dmesg | grep -i microcode
> [    1.224312] microcode: sig=0x406e3, pf=0x80, revision=0xc6
> [    1.224430] microcode: Microcode Update Driver: v2.01
> <tigran.co.uk>, Peter Oruba

0xc6 is the microcode revision for the Intel CPU with family 0x6, model 0x4e, stepping 0x3 that is shipped in Intel microcode package version 20180807a. Judging by the provided output, no microcode update is performed during boot. Since you've mentioned that the system's firmware has been updated recently, I presume that the microcode update is performed by the system's firmware, before the OS boot stage. Considering the above, I see it unlikely that the observed behaviour is related to the microcode_ctl package issue in question.

Hello,
Original issue reported in this bug has been resolved with recent kernel update.
There's no further feedback on Comment#47 from Oliver. Hopefully that means Olvier's issues are also resolved.
As for issues reported in Comment#54, Eugene explains in Comment#55 that there's no microcode update delivered by the OS package since the firmware provided one is already new enough. That means this issue is caused by microcode provided by fw, not OS, and likely the same problem can be hit without microcode_ctl package installed.
Frederico, can you try to uninstall the microcode_ctl package and reproduce on your systems?

Stanislav, Eugene,

I removed microcode_ctl:

[root@aldebaran ~]# dmesg |grep microc
[    1.245206] microcode: sig=0x406e3, pf=0x80, revision=0xc6
[    1.245325] microcode: Microcode Update Driver: v2.01 <tigran.co.uk>, Peter Oruba
[root@aldebaran ~]# rpm -qi microcode_ctl.x86_64 
package microcode_ctl.x86_64 is not installed


Still doesn't work, had a failure to resume this morning (the problem is even weirder since it doesn't happen _always_, just _eventually_, but this was always the case for me), so at least for my setup it isn't due to microcode_ctl.

I will try to find other bug reports about this behaviour (which is affecting many others around me as well) to see if there is something I can contribute to; thank you for your help.

It happens _occasionally_ to my daughter's Thinkpad too: becomes unresponsive after suspend (have to power-cycle it).
Our Acer laptops don't seem to be affected.

Thank you all for replies. As Frederico confirmed in Comment#57, his issues are bound to the system's fw microcode, not the one provided with RHEL. There isn't any clear unresolved microcode_ctl related issues opened any more, thus I'm closing this bug.
Please open new bugs specifically for each unresolved problem which would allow us to focus on each of these directly. Thank you!