Bug 1667249

Summary: Fail to launch AMD SEV VM with assigned PCI device
Product: Red Hat Enterprise Linux 8 Reporter: Alex Williamson <alex.williamson>
Component: qemu-kvmAssignee: Gary R Hook (AMD) <ghook>
Status: CLOSED ERRATA QA Contact: Pei Zhang <pezhang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.0CC: alex.williamson, blc, brijesh.singh, chayang, ctatman, hhuang, jinzhao, jkachuck, juzhang, knoel, mtessun, pezhang, rbalakri, toneata, virt-maint, zhguo
Target Milestone: rcKeywords: ZStream
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-2.12.0-74.module+el8.1.0+3227+57d66ad3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1714745 (view as bug list) Environment:
Last Closed: 2019-11-05 20:47:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1615620, 1654309, 1660905, 1663270, 1710589, 1714745    

Description Alex Williamson 2019-01-17 20:38:20 UTC 
Description of problem:

On an AMD SEV enabled host with an SEV enabled guest, attaching an assigned device to the VM results in a failure to start the VM:

qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x0: sev_ram_block_added: failed to register region (0x7fd96e6bb000+0x20000) error 'Cannot allocate memory'

In this case the assigned device is a simple Intel 82574L NIC:

01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 89, NUMA node 0
	Memory at fb9c0000 (32-bit, non-prefetchable) [size=128K]
	Memory at fb900000 (32-bit, non-prefetchable) [size=512K]

Note the error indicates the region as (base+size) where a size of 0x20000 is 128K, which matches that of BAR0 for the device.  dmesg on the host also reports:

SVM: SEV: Failure locking 32 pages.

Further analysis shows that SEV guests make use of the RAMBlock notifier in QEMU to add page pinnings for SEV where the kernel side of the call only knows how to pin pages with get_user_pages() which faults on non-page backed mappings such as the mmap of an MMIO BAR.

Consulting with AMD, Brijesh is aware of this issue an intends to post patches upstream to make the ram_device flag of the region visible to the SEV code such that these regions can be skipped in QEMU.

Version-Release number of selected component (if applicable):
kernel-4.18.0-61.el8.x86_64
qemu-kvm-2.12.0-57.module+el8+2683+02b3b955.x86_64

How reproducible:
100%

Steps to Reproduce:
1. add a vfio-pci assigned device to a host and VM previously configured with SEV support enabled
2.
3.

Actual results:
VM fails to launch

Expected results:
VM works normally

Additional info:
Comment 2 Brijesh Singh 2019-01-18 16:28:19 UTC 
patch v1 is posted: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04370.html
Comment 3 Brijesh Singh 2019-03-12 14:30:19 UTC 
V2 posted here: https://marc.info/?t=154776220600003&r=1&w=2
Comment 4 Brijesh Singh 2019-03-12 14:35:12 UTC 
v2 patch link:

https://patchwork.kernel.org/patch/10796689/
https://patchwork.kernel.org/patch/10796691/
Comment 5 Gary R Hook (AMD) 2019-03-13 22:01:53 UTC 
Waiting for them to be added to the repo...
Comment 6 Gary R Hook (AMD) 2019-04-05 22:19:54 UTC 
Patches are ready; trying to figure out how to submit.

Looks like I need:

The ITR field in this BZ set.
Move this to RHEL-8.1 qemu-kvm (I'll submit to RHEL-8.1.0)
Clone this to RHEL-AV-8.0.1 (To submit to RHEL-AV-8.0.1 as a bug fix)

...I think?

Karen I got some guidance from Danilo, but there appear to be several things amiss here. Hoping you can provide guidance?
Comment 7 Karen Noel 2019-04-05 23:17:47 UTC 
(In reply to Gary R Hook from comment #6)
> Patches are ready; trying to figure out how to submit.
> 
> Looks like I need:
> 
> The ITR field in this BZ set.

Done.

> Move this to RHEL-8.1 qemu-kvm (I'll submit to RHEL-8.1.0)
> Clone this to RHEL-AV-8.0.1 (To submit to RHEL-AV-8.0.1 as a bug fix)
> 
> ...I think?
> 
> Karen I got some guidance from Danilo, but there appear to be several things
> amiss here. Hoping you can provide guidance?


What is the upstream QEMU version? And the upstream commit IDs? 

Comment #4 references 2 kernel patches. Is there different BZ for those? Thanks!
Comment 9 Gary R Hook (AMD) 2019-04-08 13:53:32 UTC 
The two patches go together as a set. Both should be applied to 2.x and 3.x. (i.e. these need to go onto RHEL 8 (for qemu 2.12) in z-stream, and into whatever version is part of 8.1.

Upstream commits are
2ddb89b00f947f785c9ca6742f28f954e3b75e62
cedc0ad539afbbb669dba9e73dfad2915bc1c25b

Since this BZ is intended for 8.0 I'll need another BZ for 8.1. I've not opened that yet.

I'm trying to understand how to set up my patch post but the qemu version / repo branch / product name combinations are still confusing me. Have been asking Danilo for guidance.
Comment 11 Eduardo Habkost 2019-04-12 18:20:38 UTC 
Bug was cloned to RHEL-AV as bug 1699448.
Comment 15 Danilo de Paula 2019-05-20 20:39:30 UTC 
Fix included in qemu-kvm-2.12.0-74.module+el8.1.0+3227+57d66ad3
Comment 18 Joseph Kachuck 2019-05-29 22:18:45 UTC 
Hello,
This bug has been copied as 8.0.0 z-stream bug # 1714745

Thank You
Joe Kachuck
Comment 20 Pei Zhang 2019-07-09 08:29:56 UTC 
== Steps:
1. Boot VM with device assignment and SEV
/usr/libexec/qemu-kvm \
-enable-kvm \
-cpu EPYC \
-smp 4 \
-m 4G \
-object sev-guest,id=sev0,cbitpos=47,reduced-phys-bits=1 \
-machine q35,memory-encryption=sev0 \
-drive if=pflash,format=raw,unit=0,file=/usr/share/edk2/ovmf/sev/OVMF_CODE.secboot.fd,readonly \
-drive if=pflash,format=raw,unit=1,file=/usr/share/edk2/ovmf/sev/OVMF_VARS.fd \
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device pcie-root-port,id=root.3,chassis=3 \
-device pcie-root-port,id=root.4,chassis=4 \
-device pcie-root-port,id=root.5,chassis=5 \
-device virtio-scsi-pci,iommu_platform=on,id=scsi0,bus=root.1,addr=0x0 \
-drive file=/home/sev_guest.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scssi0-0-0-0,bootindex=1 \
-netdev tap,id=hostnet0,vhost=off \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=18:66:da:57:dd:03,bus=root.2,iommu_platform=true \
-vnc :0 \
-monitor stdio \
-serial unix:/tmp/console,server,nowait \
-device vfio-pci,host=0000:e3:00.0,bus=root.3 \
-device vfio-pci,host=0000:e3:00.1,bus=root.4 \


== Reproduced with qemu-kvm-2.12.0-73.module+el8.1.0+3196+302d7a44.x86_64:

After step1, qemu quit with error:
(qemu) qemu-kvm: -device vfio-pci,host=0000:e3:00.0,bus=root.3: sev_ram_block_added: failed to register region (0x7f99f5c00000+0x1000000) error 'Cannot allocate memory'

So this bug has been reproduced.

== Verified with qemu-kvm-2.12.0-80.module+el8.1.0+3572+48154135.x86_64:

After step1, guest can boot successfully. 

So this bug has been fixed very well, move to 'VERIFIED'.
Comment 22 errata-xmlrpc 2019-11-05 20:47:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345