RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1416180 - QEMU VFIO based block driver for NVMe devices
Summary: QEMU VFIO based block driver for NVMe devices
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Fam Zheng
QA Contact: CongLi
URL:
Whiteboard:
Depends On:
Blocks: 1416182 1519004 1519005 1590471 1590472
TreeView+ depends on / blocked
 
Reported: 2017-01-24 19:11 UTC by Ademar Reis
Modified: 2018-11-01 11:01 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1416182 1519004 (view as bug list)
Environment:
Last Closed: 2018-11-01 11:01:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ademar Reis 2017-01-24 19:11:07 UTC
This BZ tracks the upstream work currently being done by Fam to introduce a VFIO based NVMe driver to QEMU:

https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg02812.html

Date: Wed, 21 Dec 2016 00:31:35 +0800
From: Fam Zheng <famz>
Subject: [Qemu-devel] [PATCH 0/4] RFC: A VFIO based block driver for NVMe device

This series adds a new protocol driver that is intended to achieve about 20%
better performance for latency bound workloads (i.e. synchronous I/O) than
linux-aio when guest is exclusively accessing a NVMe device, by talking to the
device directly instead of through kernel file system layers and its NVMe
driver.

This applies on top of Stefan's block-next tree which has the busy polling
patches - the new driver also supports it.

A git branch is also available as:

    https://github.com/famz/qemu nvme

See patch 4 for benchmark numbers.

Tests are done on QEMU's NVMe emulation and a real Intel P3700 SSD NVMe card.
Most of dd/fio/mkfs/kernel build and OS installation testings work well, but an
weird write fault looking similar to [1] is consistently seen when installing
RHEL 7.3 guest, which is still under investigation.

[1]: http://lists.infradead.org/pipermail/linux-nvme/2015-May/001840.html

Also, the ram notifier is not enough for hot plugged block device because in
that case the notifier is installed _after_ ram blocks are added so it won't
get the events.

Comment 1 Fam Zheng 2017-07-10 08:35:40 UTC
Upstream progress update:

https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg00995.html

Comment 2 Fam Zheng 2018-01-17 06:19:48 UTC
Upstream patches:

https://lists.gnu.org/archive/html/qemu-devel/2018-01/msg03375.html

Comment 4 Yongxue Hong 2018-06-09 01:20:29 UTC
Reproduce on ppc64le:

[root@ibm-p8-garrison-04 ~]# uname -r
3.10.0-862.3.2.el7.ppc64le
[root@ibm-p8-garrison-04 ~]# /usr/libexec/qemu-kvm  -version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-3.el7)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[root@ibm-p8-garrison-04 ~]# lspci | grep NVMe
0001:01:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)

[root@ibm-p8-garrison-04 ~]# /usr/libexec/qemu-kvm -device vfio-pci,id=pf,host=0001:01:00.0  -drive file=nvme://0001:01:00.0/1qemu-kvm: -drive file=nvme://0001:01:00.0/1: Driver 'nvme' is not whitelisted

Comment 9 Yongxue Hong 2018-06-13 04:15:34 UTC
Reproduced with rpm(comment 8)

QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-3.el7.dwg201806131316)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[root@ibm-p8-garrison-03 cmds]# lspci | grep NVMe
0001:01:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)

[root@ibm-p8-garrison-03 cmds]# /usr/libexec/qemu-kvm -device vfio-pci,id=pf,host=0001:01:00.0  -drive file=nvme://0001:01:00.0/1
qemu-kvm: -drive file=nvme://0001:01:00.0/1: VFIO IOMMU check failed

Boot qemu command failed with SN100 Series NVMe SSD device on ppc64le.

Comment 10 Fam Zheng 2018-06-13 06:20:50 UTC
The right syntax:

/usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0

(remove -device vfio-pci,... part).

Comment 11 Yongxue Hong 2018-06-13 06:44:51 UTC
(In reply to Fam Zheng from comment #10)
> The right syntax:
> 
> /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0
> -device virtio-blk,drive=drive0
> 
> (remove -device vfio-pci,... part).

Hi Fam,

[root@ibm-p8-garrison-03 ~]# /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0
qemu-kvm: -drive file=nvme://0001:01:00.0/1,if=none,id=drive0: VFIO IOMMU check failed

Still failed to execute command.

Thanks.

Comment 12 Fam Zheng 2018-06-13 06:57:24 UTC
Please check if your system has enabled IOMMU.

Comment 13 Miroslav Rezanina 2018-06-19 18:32:50 UTC
Fix included in qemu-kvm-rhev-2.12.0-4.el7

Comment 14 Yongxue Hong 2018-06-20 02:58:41 UTC
(In reply to Miroslav Rezanina from comment #13)
> Fix included in qemu-kvm-rhev-2.12.0-4.el7

(workspace) [root@ibm-p8-garrison-03 vmt]# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-4.el7)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
(workspace) [root@ibm-p8-garrison-03 vmt]# /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0
qemu-kvm: -drive file=nvme://0001:01:00.0/1,if=none,id=drive0: VFIO IOMMU check failed

Still failed with latest qemu.

Comment 16 CongLi 2018-08-30 08:48:47 UTC
Tested nvme:// with basic block testing and block migration(drive mirror+nbd) testing.

NVMe SSD: Intel P3700

Know issue:
Bug 1587992 - nvme:// image creation support

Set this bug to 'VERIFIED'.

Comment 18 errata-xmlrpc 2018-11-01 11:01:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443


Note You need to log in before you can comment on or make changes to this bug.