Bug 1743322

Summary: [v2v] Migration performance regression [rhel-7.7.z]
Product: Red Hat Enterprise Linux 7 Reporter: RAD team bot copy to z-stream <autobot-eus-copy>
Component: qemu-kvm-rhevAssignee: Maxim Levitsky <mlevitsk>
Status: CLOSED ERRATA QA Contact: Tingting Mao <timao>
Severity: high Docs Contact:
Priority: high    
Version: 7.6CC: areis, bthurber, chayang, coli, dagur, dmetzger, eblake, fdupont, istein, izuckerm, jferlan, jinzhao, jomurphy, jprause, juzhang, kwolf, mlevitsk, mtessun, mxie, nsoffer, rjones, timao, tzheng, virt-maint, yuhuang, zili
Target Milestone: rcKeywords: Performance, Regression, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.12.0-33.el7_7.3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1648622 Environment:
Last Closed: 2019-10-22 15:24:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1648622    
Bug Blocks:    

Description RAD team bot copy to z-stream 2019-08-19 16:12:16 UTC
This bug has been copied from bug #1648622 and has been proposed to be backported to 7.7 z-stream (EUS).

Comment 6 Ilanit Stein 2019-09-03 10:57:04 UTC
Hello Maxim,

This bug (MODIFIED) is marked as fixed on: qemu-kvm-rhev-2.12.0-33.el7_7.3.

Currently on the RHV-4.3.5/6, RHEL-7.7 hosts we have this version:
qemu-kvm-rhev-2.12.0-33.el7.x86_64
Is this version contain the fix?
Or do we need to wait, for a later RHEL-7.7.X version?

Thanks,
Ilanit.

Comment 7 Maxim Levitsky 2019-09-03 12:25:39 UTC
I don't really know - I don't yet have much experience with z streams.

Comment 9 Tingting Mao 2019-09-18 07:33:11 UTC
Verified this bug as below, the write performance improved both by qemu-nbd and nbdkit. So set this bug as verified. Thanks.


Tested with:
1. Info of nbdkit tools
# rpm -qa | grep nbdkit
nbdkit-1.8.0-1.el7.x86_64
nbdkit-plugin-vddk-1.8.0-1.el7.x86_64
nbdkit-plugin-python-common-1.8.0-1.el7.x86_64
nbdkit-devel-1.8.0-1.el7.x86_64
nbdkit-example-plugins-1.8.0-1.el7.x86_64
nbdkit-basic-plugins-1.8.0-1.el7.x86_64
nbdkit-plugin-python2-1.8.0-1.el7.x86_64
nbdkit-debuginfo-1.8.0-1.el7.x86_64

2. The target block file info
# lsblk /dev/nvme0n1
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0 745.2G  0 disk
├─nvme0n1p1 259:1    0   105G  0 part
└─nvme0n1p2 259:2    0     6G  0 part

3. The source image file:
# dd if=/dev/urandom of=test.img bs=5M count=1024
# qemu-img info test.img
image: test.img
file format: raw
virtual size: 5.0G (5368709120 bytes)
disk size: 5.0G


In ‘qemu-kvm-rhev-2.12.0-33.el7’:
With qemu-nbd:
1. Export the target nvme block over nbd
# qemu-nbd -f raw /dev/nvme0n1p2 -p 9000 -t

2. Convert the image
# time qemu-img convert test.img nbd:localhost:9000 -p -n
    (100.00/100%)

real    0m31.978s
user    0m2.126s
sys    0m7.115s


With nbdkit:
# time nbdkit file file=/dev/nvme0n1p2 -p 9000 --run 'qemu-img convert test.img -p -n nbd:localhost:9000'
    (100.00/100%)

real    0m17.974s
user    0m0.517s
sys    0m3.926s



In ‘qemu-kvm-rhev-2.12.0-33.el7_7.4’:
With qemu-nbd:
1. Export the target nvme block over nbd
# qemu-nbd -f raw /dev/nvme0n1p2 -p 9000 -t

2. Convert the image
# time qemu-img convert test.img nbd:localhost:9000 -p -n
    (100.00/100%)

real    0m21.728s
user    0m1.151s
sys    0m4.982s


With nbdkit:
#  time nbdkit file file=/dev/nvme0n1p2 -p 9000 --run 'qemu-img convert test.img -p -n nbd:localhost:9000'
    (100.00/100%)

real    0m10.232s
user    0m0.388s
sys    0m3.600s

Comment 12 Ilanit Stein 2019-09-19 08:21:07 UTC
Tingting,

I have on my RHEL-7.7 host these versions:
qemu-kvm-rhev-2.12.0-33.el7.x86_64
# uname -r
3.10.0-957.21.3.el7.x86_64

As far as I remember, this qemu-kvm-rhev version was there, before this bug was fixed.
What is your qemu-kvm-rhev please? (where the bug was verified)

Here in the bug, Fixed in version field is: qemu-kvm-rhev-2.12.0-33.el7_7.3, is this the version you have?

Comment 15 errata-xmlrpc 2019-10-22 15:24:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3179