Bug 1729268 - AMD Ryzen 3000 CPUs (Zen2) do not boot due to broken rdrand instruction
Summary: AMD Ryzen 3000 CPUs (Zen2) do not boot due to broken rdrand instruction
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 30
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1729462 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-11 17:51 UTC by Florian Stock
Modified: 2019-08-05 21:05 UTC (History)
17 users (show)

Fixed In Version: systemd-241-9.gitb67ecf2.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-05 21:05:18 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Florian Stock 2019-07-11 17:51:22 UTC
Description of problem:

AMDs of the Ryzen 3000 (Zen2) Familiy do not boot.

Problem is the use of a broken instruction (rdrand) by systemd.

Version-Release number of selected component (if applicable):
afaik old systemd (like in F28) work, because they dont use the rdrand instruction. Newest systemd seems (not yet in Fedora) to have a patch for this problem 
https://github.com/systemd/systemd/pull/12536/commits/1c53d4a070edbec8ad2d384ba0014d0eb6bae077)

Systemd stops hanging in an endless loop trying to get random numbers, but the instruction does not produce them.

How reproducible:
Always

Steps to Reproduce:
1. Boot with an installed System 
2.
3.

Actual results:

Systemd hangs at boot

Expected results:

Systemd boots the system

Additional info:
Maybe the problem could also be fixed by amd patching/correcting the  firmware/microcode for the instruction.

Comment 1 Zbigniew Jędrzejewski-Szmek 2019-07-11 20:07:00 UTC
I'm not convinced that this commit solves the issue (at least with the newest AMD cpus), but it shouldn't hurt.
I'll backport the commit into F29+.

Comment 2 Florian Stock 2019-07-12 04:46:39 UTC
Afaik, this is also the patch the debians used to avoid  the bug at all (they had it before the Zen2-release) and it seems to be also the solution for the ubuntus:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1835809

(Of course the right solution would be to make AMD fix their processors).

Comment 3 Zbigniew Jędrzejewski-Szmek 2019-07-12 11:09:51 UTC
*** Bug 1729462 has been marked as a duplicate of this bug. ***

Comment 4 sunsi.lucas 2019-07-12 11:11:51 UTC
Sorry for the duplicate bug! I didn't see this one, but it sure is nice to see that someone is looking at this already!
Can I do anything to help on this effort?

Comment 5 Zbigniew Jędrzejewski-Szmek 2019-07-12 12:15:37 UTC
Yes! Testing will be needed when the new build comes out. I'd appreciate feedback if it solves the issue.

Comment 6 sunsi.lucas 2019-07-12 15:53:49 UTC
Don't know if this changes anything (probably not?) but hey, who knew AMD would be onto this https://www.phoronix.com/scan.php?page=news_item&px=AMD-Releases-Linux-Zen2-Fix

Comment 7 ryan.fedora 2019-07-19 16:34:56 UTC
I can confirm this issue with Fedora 30 installer.

Attempted install with Fedora 30 Workstation on a Zen 2 (AMD Ryzen 3700X, 570X MSI Creation).  
Available to test any solution fix.

Comment 8 sunsi.lucas 2019-07-19 19:11:38 UTC
Yeah, I just got my 3600X and am ready to test solutions as well, if needed!

Comment 9 Pavel L 2019-07-20 15:36:39 UTC
Are there any updates? 

Most of other distros had it resolved in just few days...

Comment 10 Fedora Update System 2019-07-21 09:21:25 UTC
FEDORA-2019-5feaad3f65 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-5feaad3f65

Comment 11 sunsi.lucas 2019-07-21 15:18:14 UTC
I have just installed my new R5 3600X. Since Fedora 30 won't boot, I chrooted into it and am trying to install the above mentioned packaged, with no success, not because it doesn't work but because I do not work haha :D I enabled the repos properly and dnf list only shows releases up until 241-8. How can I see the 241-9 build so I can try it?

Comment 12 sunsi.lucas 2019-07-21 15:28:30 UTC
(In reply to sunsi.lucas from comment #11)
> I have just installed my new R5 3600X. Since Fedora 30 won't boot, I
> chrooted into it and am trying to install the above mentioned packaged, with
> no success, not because it doesn't work but because I do not work haha :D I
> enabled the repos properly and dnf list only shows releases up until 241-8.
> How can I see the 241-9 build so I can try it?

From what I can gather, it isn't in testing yet, right?
How does it have 1 karma then? How soon can we expect it on testing?
I'm not asking these questions to pressure anyone, I'm just taking this oportunity
to learn how the whole process work and be a part of the testing phase, which would be awesome haha !

Comment 13 Pavel L 2019-07-21 15:58:37 UTC
(In reply to sunsi.lucas from comment #11)
> I have just installed my new R5 3600X. Since Fedora 30 won't boot, I
> chrooted into it and am trying to install the above mentioned packaged, with
> no success, not because it doesn't work but because I do not work haha :D I
> enabled the repos properly and dnf list only shows releases up until 241-8.
> How can I see the 241-9 build so I can try it?

The packages were requested to be tested. They haven't been tested and they are not in repositories yet.
I've tried to install rpm's provided in the build manually but there are some conflicts with dnf so I abandoned it for now.
critpath was set to 1 so hopefully this bugfix will be in testing soon...

Comment 14 sunsi.lucas 2019-07-21 16:03:08 UTC
(In reply to Pavel L from comment #13)
> (In reply to sunsi.lucas from comment #11)
> > I have just installed my new R5 3600X. Since Fedora 30 won't boot, I
> > chrooted into it and am trying to install the above mentioned packaged, with
> > no success, not because it doesn't work but because I do not work haha :D I
> > enabled the repos properly and dnf list only shows releases up until 241-8.
> > How can I see the 241-9 build so I can try it?
> 
> The packages were requested to be tested. They haven't been tested and they
> are not in repositories yet.
> I've tried to install rpm's provided in the build manually but there are
> some conflicts with dnf so I abandoned it for now.
> critpath was set to 1 so hopefully this bugfix will be in testing soon...

Got it! I'll watch some Bates Motel now and check back later then!
Gotta say, this stuff is more exciting to me that I thought it would be.
Think I'm gonna help in testing more often

Comment 15 Fedora Update System 2019-07-22 01:10:26 UTC
systemd-241-9.gitb67ecf2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-5feaad3f65

Comment 16 sunsi.lucas 2019-07-22 01:44:08 UTC
Sadly I had to nuke my fedora installation and replace it with a working manjaro one because I need this PC for work in the morning :( I waited up until an hour ago to test but the timing was bad and now I can't test the build  properly anymore. In any case I'm keeping a close eye on this matter because I love fedora and really want the Ryzen 3000 supporting ISO, starting from this issue. I feel sad that I wasn't useful but glad this is moving along (: mixed feelings Sunday

Comment 17 Pavel L 2019-07-22 02:15:20 UTC
Am I missing something?

# dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2019-5feaad3f65
Last metadata expiration check: 0:01:58 ago on Sun 21 Jul 2019 10:06:05 PM EDT.
No security updates needed, but 124 updates available
Dependencies resolved.
Nothing to do.
Complete!


Thank you

Comment 18 Colby Jenn 2019-07-22 05:26:54 UTC
I tested the update earlier today with a ryzen 3600, and it failed to fix the issue.

Unfortunately, I "had to" return my motherboard in order to buy an upgrade, so I won't be able to do any testing with that setup. I'm getting the new motherboard sometime next week; I'll re-test with that as soon as I'm able.

Comment 19 Colby Jenn 2019-07-22 06:11:43 UTC
(In reply to Pavel L from comment #17)
> Am I missing something?
> 
> # dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2019-5feaad3f65
> Last metadata expiration check: 0:01:58 ago on Sun 21 Jul 2019 10:06:05 PM
> EDT.
> No security updates needed, but 124 updates available
> Dependencies resolved.
> Nothing to do.
> Complete!
> 
> 
> Thank you

I get the same result.

I was able to install the update earlier by downloading the rpms with koji and installing them manually, but I can't seem to manage that at the moment either.

Comment 20 Santiago Martinez 2019-07-22 16:33:25 UTC
This commit doesn't solves the issue. I tested the update today with a ryzen 3700x, and it failed to fix the issue with rdrand.

I installed the systemd update, replace the old AMD CPU with new Ryzen CPU and the system doesn't boot. If i replace again the Ryzen CPU with old AMD CPU, Fedora boot without problem.

Comment 21 Zbigniew Jędrzejewski-Szmek 2019-07-22 16:36:09 UTC
Did you update the initramfs?

Comment 22 Santiago Martinez 2019-07-22 18:33:14 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #21)
> Did you update the initramfs?

Hello, please ignore my previous comment.

The update is correct and now I'm writing using the new Ryzen CPU. After recreate the initramfs with dracut before changing CPUs the system boot without error.

This systemd update is correct and fixes the issue with rdrand.

Comment 23 Pavel L 2019-07-22 18:40:19 UTC
Great to hear.

I did notice someone uploaded systemd packages to testing-repo so I was able to update them via dnf.
However I do not have a working fedora bootable image at the moment to test it. My grub2 need to get reinstalled.

Comment 24 Fedora Update System 2019-07-23 01:07:53 UTC
systemd-241-9.gitb67ecf2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 25 Brian Daniels 2019-07-23 01:23:58 UTC
Is a new install iso available or planned with this update?

Comment 26 sunsi.lucas 2019-07-23 09:39:43 UTC
(In reply to Brian Daniels from comment #25)
> Is a new install iso available or planned with this update?

I'm also interested in this, since I'm now locked out! :')

Comment 27 Santiago Martinez 2019-07-23 13:38:49 UTC
(In reply to sunsi.lucas from comment #26)
> (In reply to Brian Daniels from comment #25)
> > Is a new install iso available or planned with this update?
> 
> I'm also interested in this, since I'm now locked out! :')

As systemd was pushed to stable,  you can check Respin SIG (https://fedoraproject.org/wiki/Respins-SIG). They provide the community with an updated ISO Image.
The last update was from 2019-07-19 so in the next future they must provide an updated version.

The images are available at https://dl.fedoraproject.org/pub/alt/live-respins/

Comment 28 sunsi.lucas 2019-07-23 13:41:44 UTC
(In reply to Santiago Martinez from comment #27)
> (In reply to sunsi.lucas from comment #26)
> > (In reply to Brian Daniels from comment #25)
> > > Is a new install iso available or planned with this update?
> > 
> > I'm also interested in this, since I'm now locked out! :')
> 
> As systemd was pushed to stable,  you can check Respin SIG
> (https://fedoraproject.org/wiki/Respins-SIG). They provide the community
> with an updated ISO Image.
> The last update was from 2019-07-19 so in the next future they must provide
> an updated version.
> 
> The images are available at
> https://dl.fedoraproject.org/pub/alt/live-respins/

Does this mean the official Workstation image won't be updated? Or does it just mean that the respin will be faster?
Sorry for the questions, kind of new to all this :D

Comment 29 Timur Kristóf 2019-07-23 15:03:28 UTC
(In reply to sunsi.lucas from comment #28)
> Does this mean the official Workstation image won't be updated? Or does it
> just mean that the respin will be faster?
> Sorry for the questions, kind of new to all this :D

The official images usually don't get updated, at least I have never seen it happen ― please correct me if I'm wrong.
AFAIK this is why the Respin SIG exists, so you can get an up-to-date image in those rare cases when the official one doesn't boot.

Comment 30 Timur Kristóf 2019-07-23 15:15:27 UTC
(In reply to Brian Daniels from comment #25)
> Is a new install iso available or planned with this update?

I asked the Respin SIG, and they said you are welcome to join them in the #fedora-respins channel on Freenode, where they can provide you with an image that you guys can use for testing purposes. If all goes well, they say the new systemd will be included in their next release.

Comment 31 sunsi.lucas 2019-07-23 16:55:15 UTC
(In reply to Timur Kristóf from comment #30)
> (In reply to Brian Daniels from comment #25)
> > Is a new install iso available or planned with this update?
> 
> I asked the Respin SIG, and they said you are welcome to join them in the
> #fedora-respins channel on Freenode, where they can provide you with an
> image that you guys can use for testing purposes. If all goes well, they say
> the new systemd will be included in their next release.

I'm there! Thanks for pointing me the direction

Comment 32 Ben Williams 2019-07-23 18:32:31 UTC
The Respin Sig was Created mainly to deal with the huge amount of updates after the install of the release isos (currently in F30 is about 1.2G of updates from the GOLD iso)
But yes we have isos for testing come to #fedora-respins on irc.freenode.net 

Ben Williams Respin Sig Lead

Comment 33 Fomalhaut 2019-07-24 20:21:41 UTC
I installed a new AMD Ryzen 3600X and tried it out with the F30 - it did not work with all the latest updates (including https://bodhi.fedoraproject.org/updates/FEDORA-2019-5feaad3f65).
--------------
[FAILED] Failed to start Show Plymouth Boot Screen.
See 'systemctl status plymouth-start.service' for details.
[FAILED] Failed to start dracut initqueue hook.
See 'systemctl status dracut-initqueue.service' for details.
...
[FAILED] Failed to start System Initailization.
See 'systemctl status sysinit.target' for details.
[DEPEND] Dependency failed for Basic System.
[DEPEND] Dependency failed for Initrd Default Target.
...
[***   ] A start job is running for /dev/mapper/rfrmx-root (47s / no limit)
_
--------------
Hardware:
- CPU: AMD Ryzen 3600X
- MB: Asus Prome X370-Pro (with latest BIOS 5008, with the support Zen2)
-

Comment 34 Ben Williams 2019-07-25 12:51:44 UTC
Fomalhaut  did you build your own iso?
I have had a couple of people with AMD R5 3600x processors test the isos from the Fedora Respins Sig and it is working for them come see us in #fedora-respins on irc.freenode.net

Comment 35 Brian Daniels 2019-07-26 02:19:13 UTC
The iso from #fedora-respins is booting fine for me (F30-KDE-x86_64-20190723.iso) on a Asus Pro WS X570-Ace, Bios Ver. 0702, Ryzen 7 3700x.  Many thanks for the fix!

Comment 36 Fomalhaut 2019-07-26 03:33:09 UTC
How can I restore the work of an already installed F30? There are no updates to fix the problem even in koji.
The required systemd version is installed, but it did not help.
----------------
# rpm -qa | grep systemd- | grep 241
systemd-rpm-macros-241-9.gitb67ecf2.fc30.noarch
systemd-libs-241-9.gitb67ecf2.fc30.i686
systemd-241-9.gitb67ecf2.fc30.x86_64
systemd-udev-241-9.gitb67ecf2.fc30.x86_64
systemd-pam-241-9.gitb67ecf2.fc30.x86_64
systemd-container-241-9.gitb67ecf2.fc30.x86_64
systemd-libs-241-9.gitb67ecf2.fc30.x86_64
----------------
Returned to 1600X, until there is a solution.

Comment 37 Fomalhaut 2019-07-26 03:35:13 UTC
With the F30-KDE-x86_64-20190723.iso, the system booted with 3600X fine.

Comment 38 Timur Kristóf 2019-07-26 17:10:27 UTC
(In reply to Fomalhaut from comment #36)
> How can I restore the work of an already installed F30? There are no updates
> to fix the problem even in koji.
> The required systemd version is installed, but it did not help.
> Returned to 1600X, until there is a solution.

Would be interesting to see what that ISO has that your installed system doesn't. If the package versions are okay, then maybe (just a wild guess) you need to regenerate your initrd or something, in your already installed system.

Comment 39 Fomalhaut 2019-07-26 20:11:03 UTC
The kernel update came and the initrd was also rebuilt with it. It all worked! :-)
Thanks!

Comment 40 Dietrich 2019-07-28 11:17:51 UTC
For all those wondering how to fix their broken install:

get a working live iso either Fedora 28:

https://dl.fedoraproject.org/pub/fedora/linux/releases/28/Workstation/x86_64/iso/

or one of the new ones.

Then you need to chroot into the old system:

in detail here: https://unix.stackexchange.com/questions/72592/chroot-in-to-reinstall-grub2-reinstall-mnt-is-empty

mkdir /mnt/fedora

mount all the drives you need for your installation... for me it was:

mount UUID=aaaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaa /mnt/fedora -o subvol=FedoraRoot
mount UUID=aaaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaa /mnt/fedora/home -o subvol=FedoraHome (not really needed)
mount UUID=aaaaaaaaa-aaaa-aaaa-aaaa-bbbbbbbbb /mnt/fedora/boot (needed! do not forget!)

then mount all the temporary stuff:

cd /mnt/fedora
mount -o bind /dev dev
mount -o bind /proc proc
mount -o bind /sys sys
mount -t tmpfs tmpfs tmp

then chroot and update:

chroot /mnt/fedora
dnf distro-sync

Comment 41 Dietrich 2019-07-28 11:19:07 UTC
You can get the UUID numbers by using either

cat /etc/fstab

or

sudo blkid

Comment 42 Ben Williams 2019-08-02 14:35:58 UTC
The F30-20190730 Updates isos have the fix for this.
I would like to thank the many testers of our test isos.
any questions about our respins come se us on irc.freenode.net #fedora-respins

Comment 43 Zbigniew Jędrzejewski-Szmek 2019-08-05 21:05:18 UTC
Let's close this. If images need to be changed, it should be tracked elsewhere.


Note You need to log in before you can comment on or make changes to this bug.