Bug 1416180 - QEMU VFIO based block driver for NVMe devices
Summary: QEMU VFIO based block driver for NVMe devices
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Fam Zheng
QA Contact: CongLi
URL:
Whiteboard:
Depends On:
Blocks: 1519005 1590471 1590472 1416182 1519004
TreeView+ depends on / blocked
 
Reported: 2017-01-24 19:11 UTC by Ademar Reis
Modified: 2018-11-01 11:01 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1416182 1519004 (view as bug list)
Environment:
Last Closed: 2018-11-01 11:01:10 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Ademar Reis 2017-01-24 19:11:07 UTC
This BZ tracks the upstream work currently being done by Fam to introduce a VFIO based NVMe driver to QEMU:

https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg02812.html

Date: Wed, 21 Dec 2016 00:31:35 +0800
From: Fam Zheng <famz@redhat.com>
Subject: [Qemu-devel] [PATCH 0/4] RFC: A VFIO based block driver for NVMe device

This series adds a new protocol driver that is intended to achieve about 20%
better performance for latency bound workloads (i.e. synchronous I/O) than
linux-aio when guest is exclusively accessing a NVMe device, by talking to the
device directly instead of through kernel file system layers and its NVMe
driver.

This applies on top of Stefan's block-next tree which has the busy polling
patches - the new driver also supports it.

A git branch is also available as:

    https://github.com/famz/qemu nvme

See patch 4 for benchmark numbers.

Tests are done on QEMU's NVMe emulation and a real Intel P3700 SSD NVMe card.
Most of dd/fio/mkfs/kernel build and OS installation testings work well, but an
weird write fault looking similar to [1] is consistently seen when installing
RHEL 7.3 guest, which is still under investigation.

[1]: http://lists.infradead.org/pipermail/linux-nvme/2015-May/001840.html

Also, the ram notifier is not enough for hot plugged block device because in
that case the notifier is installed _after_ ram blocks are added so it won't
get the events.

Comment 1 Fam Zheng 2017-07-10 08:35:40 UTC
Upstream progress update:

https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg00995.html

Comment 2 Fam Zheng 2018-01-17 06:19:48 UTC
Upstream patches:

https://lists.gnu.org/archive/html/qemu-devel/2018-01/msg03375.html

Comment 4 Yongxue Hong 2018-06-09 01:20:29 UTC
Reproduce on ppc64le:

[root@ibm-p8-garrison-04 ~]# uname -r
3.10.0-862.3.2.el7.ppc64le
[root@ibm-p8-garrison-04 ~]# /usr/libexec/qemu-kvm  -version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-3.el7)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[root@ibm-p8-garrison-04 ~]# lspci | grep NVMe
0001:01:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)

[root@ibm-p8-garrison-04 ~]# /usr/libexec/qemu-kvm -device vfio-pci,id=pf,host=0001:01:00.0  -drive file=nvme://0001:01:00.0/1qemu-kvm: -drive file=nvme://0001:01:00.0/1: Driver 'nvme' is not whitelisted

Comment 9 Yongxue Hong 2018-06-13 04:15:34 UTC
Reproduced with rpm(comment 8)

QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-3.el7.dwg201806131316)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[root@ibm-p8-garrison-03 cmds]# lspci | grep NVMe
0001:01:00.0 Non-Volatile memory controller: HGST, Inc. Ultrastar SN100 Series NVMe SSD (rev 05)

[root@ibm-p8-garrison-03 cmds]# /usr/libexec/qemu-kvm -device vfio-pci,id=pf,host=0001:01:00.0  -drive file=nvme://0001:01:00.0/1
qemu-kvm: -drive file=nvme://0001:01:00.0/1: VFIO IOMMU check failed

Boot qemu command failed with SN100 Series NVMe SSD device on ppc64le.

Comment 10 Fam Zheng 2018-06-13 06:20:50 UTC
The right syntax:

/usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0

(remove -device vfio-pci,... part).

Comment 11 Yongxue Hong 2018-06-13 06:44:51 UTC
(In reply to Fam Zheng from comment #10)
> The right syntax:
> 
> /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0
> -device virtio-blk,drive=drive0
> 
> (remove -device vfio-pci,... part).

Hi Fam,

[root@ibm-p8-garrison-03 ~]# /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0
qemu-kvm: -drive file=nvme://0001:01:00.0/1,if=none,id=drive0: VFIO IOMMU check failed

Still failed to execute command.

Thanks.

Comment 12 Fam Zheng 2018-06-13 06:57:24 UTC
Please check if your system has enabled IOMMU.

Comment 13 Miroslav Rezanina 2018-06-19 18:32:50 UTC
Fix included in qemu-kvm-rhev-2.12.0-4.el7

Comment 14 Yongxue Hong 2018-06-20 02:58:41 UTC
(In reply to Miroslav Rezanina from comment #13)
> Fix included in qemu-kvm-rhev-2.12.0-4.el7

(workspace) [root@ibm-p8-garrison-03 vmt]# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-4.el7)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
(workspace) [root@ibm-p8-garrison-03 vmt]# /usr/libexec/qemu-kvm -drive file=nvme://0001:01:00.0/1,if=none,id=drive0 -device virtio-blk,drive=drive0
qemu-kvm: -drive file=nvme://0001:01:00.0/1,if=none,id=drive0: VFIO IOMMU check failed

Still failed with latest qemu.

Comment 16 CongLi 2018-08-30 08:48:47 UTC
Tested nvme:// with basic block testing and block migration(drive mirror+nbd) testing.

NVMe SSD: Intel P3700

Know issue:
Bug 1587992 - nvme:// image creation support

Set this bug to 'VERIFIED'.

Comment 18 errata-xmlrpc 2018-11-01 11:01:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443


Note You need to log in before you can comment on or make changes to this bug.