Bug 1612608

Summary: kvm_amd fails to load due to ccp module hang-on-load on 4.17.11-200 kernel
Product: [Fedora] Fedora Reporter: John Apple II <jappleii>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 28CC: airlied, alex.williamson, amit, berrange, bskeggs, cfergeau, crobinso, dwmw2, ewk, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kernel-maint, labbott, linville, mchehab, mikko.tanner, mjg59, pbonzini, raveit65.sun, rjones, steved, steve, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-06 14:07:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description John Apple II 2018-08-05 23:16:27 UTC
Description of problem:

Module kvm_amd fails to load on module ccp

Attempting to rmmod -f ccp provides a Resource busy message.

kvm_amd loaded correctly under 4.16.7 kernel

# modprobe -v kvm_amd


Description of Hardware:
AMD Ryzen 1950X
Gigabyte x399 motherboard (April 2018 BIOS)
SVM and SMT enabled (confirmed that these show enabled in lscpu and 32 CPU's are visible)


Version-Release number of selected component (if applicable):

Linux localhost.localdomain 4.17.11-200.fc28.x86_64 #1 SMP Mon Jul 30 15:21:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux


How reproducible:


Steps to Reproduce:
1. Use 1950X AMD system and X399 motherboard (April/2018
2. Upgrade to kernel 4.17.11-200
3. Reboot the system
4. Attempt to use any KVM function (errors)
5. Check that kvm_amd loaded (it does not)
6. Attempt to load kvm_amd module (fails by hanging on loading ccp module)

Actual results:

virsh start vm_1
fails because /dev/kvm does not exist

kvm_amd fails to load on cpp module
[root] modprobe -v kvm_amd
insmod /lib/modules/4.17.11-200.fc28.x86_64/kernel/virt/lib/irqbypass.ko.xz
insmod /lib/modules/4.17.11-200.fc28.x86_64/kernel/arch/x86/kvm/kvm.ko.xz
insmod /lib/modules/4.17.11-200.fc28.x86_64/kernel/kernel/drivers/crypto/ccp/ccp.ko.xz
<hangs here until timeout occurs>


Expected results:

Kernel boots and system loads correctly, with kvm loading (as it was prior to the upgrade)


Additional info:

Module was loading correctly under previous kernel 4.16.7.

Comment 1 John Apple II 2018-08-05 23:19:34 UTC
Seems very similar to this thread:
https://www.reddit.com/r/VFIO/comments/8yux5r/error_loading_kvmamd/

Comment 2 John Apple II 2018-08-05 23:44:31 UTC
Also tested with previous 4.17.3 kernel - same results with ccp hanging on loading with kvm_amd module.

Comment 3 John Apple II 2018-08-09 00:25:10 UTC
This bug still occurs on 4.17.12-200.fc28.x86_64 packages - tested this morning.

Comment 4 John Apple II 2018-08-12 06:16:48 UTC
Realized I put in the wrong component.  Changing kernel to qemu.

Comment 5 John Apple II 2018-08-12 23:25:47 UTC
I've hunted down the issue:
I upgraded to a BIOS version with AGESA 1.0.0.5

When I dropped back to the previous BIOS level using AGESA 1.0.0.3 Patch 4, everything started working again and ccp and kvm_amd loading correctly.

Comment 6 Mikko Tanner 2018-08-13 13:31:29 UTC
Disabling CONFIG_CRYPTO_DEV_SP_PSP in the kernel 4.17 configuration (as suggested in that Reddit thread above) and recompiling fixes the hangup.

- Gigabyte X399 Designare EX with F10 bios (AGESA 1.0.0.6 / 1.1.0.0)
- Threadripper 1900X
- kernel 4.17.12

Kernel 4.15.18 works fine.

Comment 7 Cole Robinson 2018-08-13 19:15:25 UTC
kvm module is kernel territory, changing component back

Alex, I see you commented in the reddit thread, positing this is a kernel misconfiguration... is this something that can/should be fixed in fedora kernel config or is this something else?

Comment 8 Wolfgang Ulbrich 2018-08-27 19:00:54 UTC
Same problem with gigabyte x470 board / ryzen 5 1400 after bios update.
AGESA 1.0.0.2c --> AGESA 1.0.0.4
[root@mother rave]# uname -r
4.17.18-200.fc28.x86_64

Beside from that the kvm_amd module doesnt load, shutting down the box take more than 5 min. now. Systemd-shutdown hangs but i don't see any useful infos in logs.

Resetting the bios back to AGESA 1.0.0.2c solved all problems.

Comment 9 Wolfgang Ulbrich 2018-09-01 22:47:38 UTC
I just rebuild current f28 kernel with CONFIG_CRYPTO_DEV_SP_PSP disabled, and it fixes the ugly shutdown issues but not the broken kvm-amd modules load.
I added this content to kernel-local file in fedora git repo.

[rave@mother ~]$ cat kernel/kernel-local 
# This file is intentionally left empty in the stock kernel. Its a nicety
# added for those wanting to do custom rebuilds with altered config opts.

# CONFIG_CRYPTO_DEV_SP_PSP is not set

The resulting scratch build is here https://koji.fedoraproject.org/koji/taskinfo?taskID=29425990

Did i something wrong?
I am not a specialist with rebuilding a kernel.

Would be nice if someone from kernel team can take a look at it.
Amd ryzen is i a very nice and common proc, and AMD did an amazing job with the AGESA 1.0.0.4 bios update,  eg. speed up booting and memory stability.

I am happy to provide more infos if needed.

Thank you.

Comment 10 Wolfgang Ulbrich 2018-09-01 23:36:28 UTC
Ok, i did a mistake during testing.
Latest fedora kernel rebuild with 

# CONFIG_CRYPTO_DEV_SP_PSP is not set

fixes the kvm-modul load and system shutdown issues.
https://koji.fedoraproject.org/koji/taskinfo?taskID=29425990

But i don't wanna rebuild the kernel every week in fedora ;)

Comment 11 Wolfgang Ulbrich 2018-09-02 13:25:10 UTC
I tested a bit more and the issue was introduced during early kernel-4.16.0-rc1 developing.
Last good kernel is:
kernel-4.16.0-0.rc0.git9.1.fc28
First bad kernel is:
kernel-4.16.0-0.rc1.git0.1.fc28

In other words,
first bad commit is https://src.fedoraproject.org/cgit/rpms/kernel.git/commit/?h=f28&id=94e04cd684a20d47e9529936b8beea179faaee81

commit limited to x86_64 https://src.fedoraproject.org/cgit/rpms/kernel.git/diff/kernel-x86_64.config?h=f28&id=94e04cd684a20d47e9529936b8beea179faaee81

diff --git a/kernel-x86_64.config b/kernel-x86_64.config
index ffb2bbe..bf1c130 100644
--- a/kernel-x86_64.config
+++ b/kernel-x86_64.config
@@ -519,6 +519,7 @@ CONFIG_BOOT_PRINTK_DELAY=y
 CONFIG_BOUNCE=y
 CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT=y
+# CONFIG_BPF_KPROBE_OVERRIDE is not set
 CONFIG_BPF_STREAM_PARSER=y
 CONFIG_BPF_SYSCALL=y
 CONFIG_BPQETHER=m
@@ -909,6 +910,7 @@ CONFIG_CRYPTO_DEV_QAT_C62XVF=m
 CONFIG_CRYPTO_DEV_QAT_DH895xCC=m
 CONFIG_CRYPTO_DEV_QAT_DH895xCCVF=m
 # CONFIG_CRYPTO_DEV_SP_CCP is not set
+CONFIG_CRYPTO_DEV_SP_PSP=y
 CONFIG_CRYPTO_DEV_VIRTIO=m
 CONFIG_CRYPTO_DH=y
 CONFIG_CRYPTO_DRBG_CTR=y
@@ -2642,6 +2644,7 @@ CONFIG_KS0108_PORT=0x378
 CONFIG_KSM=y
 CONFIG_KSZ884X_PCI=m
 CONFIG_KVM_AMD=m
+CONFIG_KVM_AMD_SEV=y
 # CONFIG_KVM_DEBUG_FS is not set
 CONFIG_KVM_DEVICE_ASSIGNMENT=y
 CONFIG_KVM_GUEST=y


I don't know if there is a related change in patch-4.16-rc1.xz

I hope this will help kernel developers to fix this annoying issue.

Comment 12 Laura Abbott 2018-09-06 14:07:39 UTC

*** This bug has been marked as a duplicate of bug 1608242 ***

Comment 13 Red Hat Bugzilla 2023-09-14 04:32:45 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days