Bug 1501362

Summary: Intel microcode not updating --> [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x25 (or later)
Product: [Fedora] Fedora Reporter: Chatty <mail>
Component: microcode_ctlAssignee: Anton Arapov <aarapov>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 28CC: aarapov, airlied, ajax, awilliam, brajkovic, bskeggs, cedric.bellegarde, eparis, esandeen, extras-qa, gmarr, hdegoede, henryju, hlenor, ichavero, itamar, jarodwilson, jforbes, jglisse, jonathan, josef, juloml, jwboyer, jylo06g, kernel-maint, knutjbj, labbott, linville, mail, mchehab, mjg59, nhorman, pikachu.2014, poros, quintela, robatino, skozina, steved, t.hisch
Target Milestone: ---Keywords: CommonBugs, Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
URL: https://forums.fedoraforum.org/showthread.php?t=315649
Whiteboard: RejectedBlocker RejectedFreezeException https://fedoraproject.org/wiki/Common_F27_bugs#surface-book-microcode
Fixed In Version: microcode_ctl-2.1-19.fc26 microcode_ctl-2.1-22.fc26 microcode_ctl-2.1-22.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-20 17:34:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Info of cpuinfo, dmesg, ...
none
Requested infos
none
Test if microcode is vulernable to meltdown (as in if the microcode has been successfully applied / is complete as supplied from Intel).
none
spectre-meltdown-checker for my i5-4670K none

Description Chatty 2017-10-12 12:49:12 UTC
Description of problem: When booting live raw hide it display


Version-Release number of selected component (if applicable): e.g. 20171009


How reproducible: every time


Steps to Reproduce:
1. Boot live rawhide from USB key on Medion S6417 (hw details: http://www.notebookinfo.de/produkte/medion-akoya-s6417-md-99651/md99651/00029835/)
2. see error message
3.

Actual results: TSC_DEADLINE disabled


Expected results: TSC_DEADLINE enabled like in prior Fedora release like F26


Additional info:
I am not an expert but I downloaded latest Intel microcode and copied all files from intel-ucode to /lib/firmware/intel-ucode/ and did dracut -fv at the end but changes where not persisted obviously.

Comment 1 Jia Yuan Lo 2017-10-18 02:56:42 UTC
Not sure this bug should be filed against dracut like the bug report below:
https://bugzilla.redhat.com/show_bug.cgi?id=1476039

And if this didn't get fix, it should be a blocker for f27
Its a huge deal before fixing anything else if the cpu doesn't work properly

Comment 2 Fedora Blocker Bugs Application 2017-10-18 04:07:49 UTC
Proposed as a Blocker and Freeze Exception for 27-final by Fedora user truboxl using the blocker tracking app because:

 Affects some Intel hardware not update microcode early (to the latest version), may cause lockups.
Should be fixed before anything else

Comment 3 Chatty 2017-10-18 07:54:01 UTC
I just tried ubuntu-17.10-beta2-desktop-amd64.iso on the same machine and got the same behavior (TSC_Deadline disabled, update to microcode 0x25).

Comment 4 Adam Williamson 2017-10-18 16:49:41 UTC
I don't think this really a case of "CPU doesn't work properly", per http://wiki.osdev.org/APIC_timer tsc-deadline mode is a relatively new and certainly 'optional' feature (everything will work without it). Older CPUs don't have it at all.

So I don't think this is a blocker, but shipping the updated microcode would be good to avoid 'worrying' messages like this, so I'm probably +1 FE. Re-assigning to appropriate package; Anton, it looks like the Intel microcode in the package hasn't been updated since 20170707?

Comment 5 Chatty 2017-10-18 18:38:06 UTC
This bug also happens with Ubtuntu 17.10-beta2: https://bugs.launchpad.net/ubuntu/+source/evince/+bug/1724650

Comment 6 Adam Williamson 2017-10-18 18:48:30 UTC
I'd expect it'll happen on anything that has a sufficiently recent kernel but not a sufficiently recent set of Intel microcode updates.

Comment 7 Anton Arapov 2017-10-20 09:03:48 UTC
Hello Chatty,

could you attach a text file here with the output of following commands, after you booted cleanly (without forcing microcode load manually)
# dmidecode -t 4
# cat /proc/cpuinfo
# dmesg
# rpm -q microcode_ctl
# ls -l /lib/firmware/intel-ucode/

thanks,

Comment 8 Chatty 2017-10-20 21:25:35 UTC
Created attachment 1341448 [details]
Info of cpuinfo, dmesg, ...

Comment 9 JMlich 2017-10-21 12:08:09 UTC
Two x86-apic patches on TSC_DEADLINE  waiting 
in https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/queue-4.13

However doesn't match fully issue conditions.

Comment 10 Geoffrey Marr 2017-10-23 21:28:03 UTC
Discussed during the 2017-10-23 blocker review meeting: [1]

The decision to classify this bug as a RejectedBlocker and a RejectedFreezeException was made as this seems to only be a warning about a possible feature that could be implemented and rushing to fix/add it could be potentially dangerous.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2017-10-23/f27-blocker-review.2017-10-23-16.00.txt

Comment 11 Chatty 2017-10-29 06:12:27 UTC
The solution to this bug is very easy: I inserted intel_ucode.img (taken from archlinux-2017.10.01-x86_64.iso) as first initrd and rawhide booted with TSC enabled as it should.

Comment 12 Thomas Hisch 2017-11-08 18:56:34 UTC
In my case (Fedora 27-beta live image, Asus motherboard, Intel i7) the error message is

[Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x22 (or later).

After this message two dracut-pre-udev messages and 2 messages from my sd drive are displayed and then the system hangs -> I can't boot the kernel on the live image!

Comment 13 Chatty 2017-11-08 19:20:52 UTC
Does it work with the updated microcode (see #11).

Comment 14 Chatty 2017-11-08 19:22:01 UTC
Does it work with the updated microcode (see #11)?

Comment 15 Adam Williamson 2017-11-09 00:26:39 UTC
Thomas: it's very unlikely the hang has anything to do with the message. You just happen to notice the message because the system hangs; if it didn't, the message would be displayed but you probably wouldn't notice it.

Comment 16 Fedora Update System 2017-11-22 08:39:09 UTC
microcode_ctl-2.1-19.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-c8f03e1046

Comment 17 Thomas Hisch 2017-11-22 11:23:22 UTC
Note that my issue reported in comment 12 seems to be solved with the final F27 live image. I still get microcode warning, though.

Comment 18 Chatty 2017-11-24 14:21:56 UTC
Please also update Rawhide. Fedora-Workstation-Live-x86_64-Rawhide-20171123 still doesn't boot without manually inserting the microcode update.

Comment 19 Fedora Update System 2017-11-25 01:33:16 UTC
microcode_ctl-2.1-19.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-c8f03e1046

Comment 20 Fedora Update System 2017-11-28 16:24:46 UTC
microcode_ctl-2.1-19.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Chatty 2017-11-28 17:06:08 UTC
This package needs to be integrated into the installation ISO otherwise affected systems can't boot! This means, this bug is not yet resolved!

Please tell me in which ISO it is integrated so I can test?

Comment 22 Adam Williamson 2017-11-28 18:04:19 UTC
It is not, and we have no plans to respin the installer images at this time. So far you're the only person reporting that this somehow prevents your system from booting; I haven't seen any other report of this. Anton, did the information Chatty provided in #c8 tell you anything interesting?

Comment 23 Chatty 2017-11-28 19:17:32 UTC
It happens on the Surface Book which I'm sure quite a lot of people own. Mine is updated to the most recent version. I hope Microsoft will fix the Intel ME security flaw along with the microcode any time soon but so far they didn't.

If you don't fix it for F27 it would be great if you fix it at least for F28 onwards.

Comment 24 Anton Arapov 2017-11-29 08:37:14 UTC
(In reply to Adam Williamson from comment #22)
> It is not, and we have no plans to respin the installer images at this time.
> So far you're the only person reporting that this somehow prevents your
> system from booting; I haven't seen any other report of this. Anton, did the
> information Chatty provided in #c8 tell you anything interesting?

Adam, not really. I still have no idea about the nature of this issue and how exactly it was fixed. Intel doesn't tell anything about the firmware blobs and the issues it fixes. That said, if this update helped - we may hope that it was fixed and it wasn't just a coincidence.

I have no idea how popular Surface Book is, so can't tell whether we want to respin the installer images. I tend to think - in case it would be a big issue we would have more reports.

Comment 25 Chatty 2017-11-29 09:09:34 UTC
Due to the nature of the bug (happening so early in the boot process) I guess most people don't know what to report as they can't come up with logs etc. Having said that I am also surprised that there are so little reports as this affects all recent distros. Anyway, I've got my workaround (µcode update as first initrd).

Comment 26 Stanislav Kozina 2017-11-29 10:47:59 UTC
If we don't want to respin ISOs there are some workarounds which I believe should work, although not easy:
 - 1) Install F26 and update to F27 latest
 - 2) Configure Anaconda via kickstart to enable the update repo for installation rather that the GA repo. Anaconda will then install the updated package directly.

Comment 27 Adam Williamson 2017-11-29 16:46:12 UTC
Network installs have the updates repository enabled by default.

Comment 28 Bojan Rajkovic 2017-12-18 05:24:54 UTC
I'm seeing the same issue with the F27 Live ISO on a Surface Book w/ Performance Base as well. I'm currently making an F26 image, but can confirm Chatty's exact experience.

Comment 29 Adam Williamson 2017-12-18 19:04:00 UTC
Thanks for the confirmation. I'll write up a note for Common Bugs documenting the case for the Surface Book...

Comment 30 Fedora End Of Life 2018-02-20 15:24:52 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

Comment 31 Knut J BJuland 2018-03-14 20:13:54 UTC
Intel Rolls Out Updated, Post-Spectre CPU Microcode (20180312), will it be included in update rpm?

Comment 32 Fedora Update System 2018-03-15 12:15:53 UTC
microcode_ctl-2.1-22.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5767adea74

Comment 33 Fedora Update System 2018-03-15 12:22:09 UTC
microcode_ctl-2.1-22.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-172ca25120

Comment 34 Fedora Update System 2018-03-15 15:58:45 UTC
microcode_ctl-2.1-22.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-172ca25120

Comment 35 Fedora Update System 2018-03-15 16:28:49 UTC
microcode_ctl-2.1-22.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5767adea74

Comment 36 Cédric Bellegarde 2018-03-19 09:42:22 UTC
Fedora 28 here, microcode is not auto updated.

Comment 37 Fedora Update System 2018-03-20 17:34:35 UTC
microcode_ctl-2.1-22.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 38 Fedora Update System 2018-03-20 18:20:50 UTC
microcode_ctl-2.1-22.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 39 Adam Williamson 2018-04-04 21:31:00 UTC
Chatty, Bojan - could either of you check whether F28 images boot OK on the Surface Book? Thanks!

Comment 40 Chatty 2018-04-05 17:39:56 UTC
All images (even the one's from bug report date) are booting now since Microsoft updated the UEFI bios (and the microcode) due to Spectre and Meltdown as I reported already here: https://bugzilla.redhat.com/show_bug.cgi?id=1500057#c6

I can however test a current image on my Medion S6417 laptop whether TSC_DEADLINE error is fixed.

Comment 41 Adam Williamson 2018-04-05 18:06:32 UTC
Well, all images are booting *so long as you've updated the system firmware*, I guess. Thanks for the info.

Comment 42 Julien HENRY 2018-06-12 06:40:55 UTC
Created attachment 1450329 [details]
Requested infos

I am still seeing this annoying message with an updated Fedora 28. Is it expected?

Comment 43 user 2018-07-06 11:42:58 UTC
Created attachment 1456975 [details]
Test if microcode is vulernable to meltdown (as in if the microcode has been successfully applied / is complete as supplied from Intel).

Comment 44 user 2018-07-06 11:49:53 UTC
(In reply to Julien HENRY from comment #42)
> Created attachment 1450329 [details]
> Requested infos
> 
> I am still seeing this annoying message with an updated Fedora 28. Is it
> expected?

I second this, either the microcode is not properly updated or it's incomplete.

I'm on Fedora Atomic Workstation r28 with initramfs regeneration enabled, package installed, dracut -fv doesn't change anything either. All updates applied. 

The microcode just won't get updated. See the spectre-meltdown checker script above. Intel Core I5 6200

$ cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 78
model name	: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
stepping	: 3
microcode	: 0x33
cpu MHz		: 800.022
cache size	: 3072 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 4800.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

Comment 45 Julien HENRY 2018-07-06 11:52:21 UTC
Created attachment 1456976 [details]
spectre-meltdown-checker for my i5-4670K

I have attached the output of spectre-meltdown-checker

Comment 46 Justin M. Forbes 2018-07-06 15:34:59 UTC
Intel still has not released microcode updates for everything, though they did post a new bundle on 7/03. Microcode_ctl should be updated with this at some point very soon.

Comment 47 Red Hat Bugzilla 2023-09-14 04:09:51 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days