RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1667249 - Fail to launch AMD SEV VM with assigned PCI device
Summary: Fail to launch AMD SEV VM with assigned PCI device
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 8.1
Assignee: Gary R Hook (AMD)
QA Contact: Pei Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1615620 1654309 1660905 1663270 1710589 1714745
TreeView+ depends on / blocked
 
Reported: 2019-01-17 20:38 UTC by Alex Williamson
Modified: 2019-11-05 20:48 UTC (History)
16 users (show)

Fixed In Version: qemu-kvm-2.12.0-74.module+el8.1.0+3227+57d66ad3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1714745 (view as bug list)
Environment:
Last Closed: 2019-11-05 20:47:34 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:3345 0 None None None 2019-11-05 20:48:12 UTC

Description Alex Williamson 2019-01-17 20:38:20 UTC 
Description of problem:

On an AMD SEV enabled host with an SEV enabled guest, attaching an assigned device to the VM results in a failure to start the VM:

qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x0: sev_ram_block_added: failed to register region (0x7fd96e6bb000+0x20000) error 'Cannot allocate memory'

In this case the assigned device is a simple Intel 82574L NIC:

01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 89, NUMA node 0
	Memory at fb9c0000 (32-bit, non-prefetchable) [size=128K]
	Memory at fb900000 (32-bit, non-prefetchable) [size=512K]

Note the error indicates the region as (base+size) where a size of 0x20000 is 128K, which matches that of BAR0 for the device.  dmesg on the host also reports:

SVM: SEV: Failure locking 32 pages.

Further analysis shows that SEV guests make use of the RAMBlock notifier in QEMU to add page pinnings for SEV where the kernel side of the call only knows how to pin pages with get_user_pages() which faults on non-page backed mappings such as the mmap of an MMIO BAR.

Consulting with AMD, Brijesh is aware of this issue an intends to post patches upstream to make the ram_device flag of the region visible to the SEV code such that these regions can be skipped in QEMU.

Version-Release number of selected component (if applicable):
kernel-4.18.0-61.el8.x86_64
qemu-kvm-2.12.0-57.module+el8+2683+02b3b955.x86_64

How reproducible:
100%

Steps to Reproduce:
1. add a vfio-pci assigned device to a host and VM previously configured with SEV support enabled
2.
3.

Actual results:
VM fails to launch

Expected results:
VM works normally

Additional info:
Comment 2 Brijesh Singh 2019-01-18 16:28:19 UTC 
patch v1 is posted: https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg04370.html
Comment 3 Brijesh Singh 2019-03-12 14:30:19 UTC 
V2 posted here: https://marc.info/?t=154776220600003&r=1&w=2
Comment 4 Brijesh Singh 2019-03-12 14:35:12 UTC 
v2 patch link:

https://patchwork.kernel.org/patch/10796689/
https://patchwork.kernel.org/patch/10796691/
Comment 5 Gary R Hook (AMD) 2019-03-13 22:01:53 UTC 
Waiting for them to be added to the repo...
Comment 6 Gary R Hook (AMD) 2019-04-05 22:19:54 UTC 
Patches are ready; trying to figure out how to submit.

Looks like I need:

The ITR field in this BZ set.
Move this to RHEL-8.1 qemu-kvm (I'll submit to RHEL-8.1.0)
Clone this to RHEL-AV-8.0.1 (To submit to RHEL-AV-8.0.1 as a bug fix)

...I think?

Karen I got some guidance from Danilo, but there appear to be several things amiss here. Hoping you can provide guidance?
Comment 7 Karen Noel 2019-04-05 23:17:47 UTC 
(In reply to Gary R Hook from comment #6)
> Patches are ready; trying to figure out how to submit.
> 
> Looks like I need:
> 
> The ITR field in this BZ set.

Done.

> Move this to RHEL-8.1 qemu-kvm (I'll submit to RHEL-8.1.0)
> Clone this to RHEL-AV-8.0.1 (To submit to RHEL-AV-8.0.1 as a bug fix)
> 
> ...I think?
> 
> Karen I got some guidance from Danilo, but there appear to be several things
> amiss here. Hoping you can provide guidance?


What is the upstream QEMU version? And the upstream commit IDs? 

Comment #4 references 2 kernel patches. Is there different BZ for those? Thanks!
Comment 9 Gary R Hook (AMD) 2019-04-08 13:53:32 UTC 
The two patches go together as a set. Both should be applied to 2.x and 3.x. (i.e. these need to go onto RHEL 8 (for qemu 2.12) in z-stream, and into whatever version is part of 8.1.

Upstream commits are
2ddb89b00f947f785c9ca6742f28f954e3b75e62
cedc0ad539afbbb669dba9e73dfad2915bc1c25b

Since this BZ is intended for 8.0 I'll need another BZ for 8.1. I've not opened that yet.

I'm trying to understand how to set up my patch post but the qemu version / repo branch / product name combinations are still confusing me. Have been asking Danilo for guidance.
Comment 11 Eduardo Habkost 2019-04-12 18:20:38 UTC 
Bug was cloned to RHEL-AV as bug 1699448.
Comment 15 Danilo de Paula 2019-05-20 20:39:30 UTC 
Fix included in qemu-kvm-2.12.0-74.module+el8.1.0+3227+57d66ad3
Comment 18 Joseph Kachuck 2019-05-29 22:18:45 UTC 
Hello,
This bug has been copied as 8.0.0 z-stream bug # 1714745

Thank You
Joe Kachuck
Comment 20 Pei Zhang 2019-07-09 08:29:56 UTC 
== Steps:
1. Boot VM with device assignment and SEV
/usr/libexec/qemu-kvm \
-enable-kvm \
-cpu EPYC \
-smp 4 \
-m 4G \
-object sev-guest,id=sev0,cbitpos=47,reduced-phys-bits=1 \
-machine q35,memory-encryption=sev0 \
-drive if=pflash,format=raw,unit=0,file=/usr/share/edk2/ovmf/sev/OVMF_CODE.secboot.fd,readonly \
-drive if=pflash,format=raw,unit=1,file=/usr/share/edk2/ovmf/sev/OVMF_VARS.fd \
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device pcie-root-port,id=root.3,chassis=3 \
-device pcie-root-port,id=root.4,chassis=4 \
-device pcie-root-port,id=root.5,chassis=5 \
-device virtio-scsi-pci,iommu_platform=on,id=scsi0,bus=root.1,addr=0x0 \
-drive file=/home/sev_guest.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scssi0-0-0-0,bootindex=1 \
-netdev tap,id=hostnet0,vhost=off \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=18:66:da:57:dd:03,bus=root.2,iommu_platform=true \
-vnc :0 \
-monitor stdio \
-serial unix:/tmp/console,server,nowait \
-device vfio-pci,host=0000:e3:00.0,bus=root.3 \
-device vfio-pci,host=0000:e3:00.1,bus=root.4 \


== Reproduced with qemu-kvm-2.12.0-73.module+el8.1.0+3196+302d7a44.x86_64:

After step1, qemu quit with error:
(qemu) qemu-kvm: -device vfio-pci,host=0000:e3:00.0,bus=root.3: sev_ram_block_added: failed to register region (0x7f99f5c00000+0x1000000) error 'Cannot allocate memory'

So this bug has been reproduced.

== Verified with qemu-kvm-2.12.0-80.module+el8.1.0+3572+48154135.x86_64:

After step1, guest can boot successfully. 

So this bug has been fixed very well, move to 'VERIFIED'.
Comment 22 errata-xmlrpc 2019-11-05 20:47:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345
Note You need to log in before you can comment on or make changes to this bug.