Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1416180

Summary: QEMU VFIO based block driver for NVMe devices
Product: Red Hat Enterprise Linux 7 Reporter: Ademar Reis <areis>
Component: qemu-kvm-rhevAssignee: Fam Zheng <famz>
Status: CLOSED ERRATA QA Contact: CongLi <coli>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: aliang, areis, chayang, dgibson, famz, juzhang, michen, mtessun, qzhang, virt-maint, xuwei, yama, yhong
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.12.0-4.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1416182 1519004 (view as bug list) Environment:
Last Closed: 2018-11-01 11:01:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1416182, 1519004, 1519005, 1590471, 1590472    

Description Ademar Reis 2017-01-24 19:11:07 UTC
This BZ tracks the upstream work currently being done by Fam to introduce a VFIO based NVMe driver to QEMU:

https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg02812.html

Date: Wed, 21 Dec 2016 00:31:35 +0800
From: Fam Zheng <famz>
Subject: [Qemu-devel] [PATCH 0/4] RFC: A VFIO based block driver for NVMe device

This series adds a new protocol driver that is intended to achieve about 20%
better performance for latency bound workloads (i.e. synchronous I/O) than
linux-aio when guest is exclusively accessing a NVMe device, by talking to the
device directly instead of through kernel file system layers and its NVMe
driver.

This applies on top of Stefan's block-next tree which has the busy polling
patches - the new driver also supports it.

A git branch is also available as:

    https://github.com/famz/qemu nvme

See patch 4 for benchmark numbers.

Tests are done on QEMU's NVMe emulation and a real Intel P3700 SSD NVMe card.
Most of dd/fio/mkfs/kernel build and OS installation testings work well, but an
weird write fault looking similar to [1] is consistently seen when installing
RHEL 7.3 guest, which is still under investigation.

[1]: http://lists.infradead.org/pipermail/linux-nvme/2015-May/001840.html

Also, the ram notifier is not enough for hot plugged block device because in
that case the notifier is installed _after_ ram blocks are added so it won't
get the events.

Comment 1 Fam Zheng 2017-07-10 08:35:40 UTC
Upstream progress update:

https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg00995.html

Comment 2 Fam Zheng 2018-01-17 06:19:48 UTC
Upstream patches:

https://lists.gnu.org/archive/html/qemu-devel/2018-01/msg03375.html

Comment 4 Yongxue Hong 2018-06-09 01:20:29 UTC
Reproduce on ppc64le:

[root@ibm-p8-garrison-04 ~]# uname -r
3.10.0-862.3.2.el7.ppc64le
[root@ibm-p8-garrison-04 ~]# /usr/libexec/qemu-kvm  -version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-3.el7)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[root@ibm-p8-garrison-04 ~]# lspci | grep NVMe
0001:01:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)

[root@ibm-p8-garrison-04 ~]# /usr/libexec/qemu-kvm -device vfio-pci,id=pf,host=0001:01:00.0  -drive file=nvme://0001:01:00.0/1qemu-kvm: -drive file=nvme://0001:01:00.0/1: Driver 'nvme' is not whitelisted

Comment 9 Yongxue Hong 2018-06-13 04:15:34 UTC
Reproduced with rpm(comment 8)

QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-3.el7.dwg201806131316)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[root@ibm-p8-garrison-03 cmds]# lspci | grep NVMe
0001:01:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)

[root@ibm-p8-garrison-03 cmds]# /usr/libexec/qemu-kvm -device vfio-pci,id=pf,host=0001:01:00.0  -drive file=nvme://0001:01:00.0/1
qemu-kvm: -drive file=nvme://0001:01:00.0/1: VFIO IOMMU check failed

Boot qemu command failed with SN100 Series NVMe SSD device on ppc64le.

Comment 10 Fam Zheng 2018-06-13 06:20:50 UTC
The right syntax:

/usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0

(remove -device vfio-pci,... part).

Comment 11 Yongxue Hong 2018-06-13 06:44:51 UTC
(In reply to Fam Zheng from comment #10)
> The right syntax:
> 
> /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0
> -device virtio-blk,drive=drive0
> 
> (remove -device vfio-pci,... part).

Hi Fam,

[root@ibm-p8-garrison-03 ~]# /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0
qemu-kvm: -drive file=nvme://0001:01:00.0/1,if=none,id=drive0: VFIO IOMMU check failed

Still failed to execute command.

Thanks.

Comment 12 Fam Zheng 2018-06-13 06:57:24 UTC
Please check if your system has enabled IOMMU.

Comment 13 Miroslav Rezanina 2018-06-19 18:32:50 UTC
Fix included in qemu-kvm-rhev-2.12.0-4.el7

Comment 14 Yongxue Hong 2018-06-20 02:58:41 UTC
(In reply to Miroslav Rezanina from comment #13)
> Fix included in qemu-kvm-rhev-2.12.0-4.el7

(workspace) [root@ibm-p8-garrison-03 vmt]# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-4.el7)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
(workspace) [root@ibm-p8-garrison-03 vmt]# /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0
qemu-kvm: -drive file=nvme://0001:01:00.0/1,if=none,id=drive0: VFIO IOMMU check failed

Still failed with latest qemu.

Comment 16 CongLi 2018-08-30 08:48:47 UTC
Tested nvme:// with basic block testing and block migration(drive mirror+nbd) testing.

NVMe SSD: Intel P3700

Know issue:
Bug 1587992 - nvme:// image creation support

Set this bug to 'VERIFIED'.

Comment 18 errata-xmlrpc 2018-11-01 11:01:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443