Bug 1513362

Summary: QEMU's AIO subsystem gets stuck inhibiting all I/O operations on virtio-blk-pci devices [rhel-7.4.z]
Product: Red Hat Enterprise Linux 7 Reporter: Oneata Mircea Teodor <toneata>
Component: qemu-kvm-rhevAssignee: Stefan Hajnoczi <stefanha>
Status: CLOSED CURRENTRELEASE QA Contact: aihua liang <aliang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.4CC: aliang, chayang, coli, dhill, gwatson, juzhang, kiyyappa, knoel, ljozsa, lolyu, michen, mircea.vutcovici, mrezanin, mschuppe, mtessun, ngu, pbonzini, pingl, qzhang, rhodain, sarita.maini, shivapriya.o.hiremath, slopezpa, stefanha, toneata, virt-maint, xuwei, yhong
Target Milestone: rcKeywords: ZStream
Target Release: ---Flags: toneata: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-16.el7_4.12 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1508886
: 1525497 1525500 1525502 1525504 1525505 1525507 1525510 (view as bug list) Environment:
Last Closed: 2018-02-12 18:27:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1508886    
Bug Blocks: 1525497, 1525500, 1525502, 1525504, 1525505, 1525507, 1525510    

Description Oneata Mircea Teodor 2017-11-15 09:52:54 UTC
This bug has been copied from bug #1508886 and has been proposed to be backported to 7.4 z-stream (EUS).

Comment 3 Miroslav Rezanina 2017-11-16 08:25:49 UTC
Fix included in qemu-kvm-rhev-2.9.0-16.el7_4.12

Comment 5 aihua liang 2017-11-23 03:05:29 UTC
Code checking done in qemu-kvm-rhev-2.9.0-16.el7_4.12,PASS.

Sanity test on qemu-kvm-rhev-2.9.0-16.el7_4.12+ 3.10.0-693.2.1.el7.x86_64,PASS.

Comment 6 CongLi 2017-11-23 03:11:51 UTC
(In reply to aihua liang from comment #5)
> Code checking done in qemu-kvm-rhev-2.9.0-16.el7_4.12,PASS.
> 
> Sanity test on qemu-kvm-rhev-2.9.0-16.el7_4.12+
> 3.10.0-693.2.1.el7.x86_64,PASS.

Hi Stefan,

QE could not reproduce this bug, sanity testing and code inspection based on BZ1508886#c27 pass, do you think is it ok or still need ask customer to have a try ?

Thanks.

Comment 7 Stefan Hajnoczi 2017-11-23 11:27:45 UTC
(In reply to CongLi from comment #6)
> (In reply to aihua liang from comment #5)
> > Code checking done in qemu-kvm-rhev-2.9.0-16.el7_4.12,PASS.
> > 
> > Sanity test on qemu-kvm-rhev-2.9.0-16.el7_4.12+
> > 3.10.0-693.2.1.el7.x86_64,PASS.
> 
> Hi Stefan,
> 
> QE could not reproduce this bug, sanity testing and code inspection based on
> BZ1508886#c27 pass, do you think is it ok or still need ask customer to have
> a try ?

The customer confirmed that a custom build with the patch solves the problem.

Thanks for doing the code inspection.  I would consider this BZ good to go.

Comment 8 CongLi 2017-11-24 01:16:48 UTC
(In reply to Stefan Hajnoczi from comment #7)
> (In reply to CongLi from comment #6)
> > (In reply to aihua liang from comment #5)
> > > Code checking done in qemu-kvm-rhev-2.9.0-16.el7_4.12,PASS.
> > > 
> > > Sanity test on qemu-kvm-rhev-2.9.0-16.el7_4.12+
> > > 3.10.0-693.2.1.el7.x86_64,PASS.
> > 
> > Hi Stefan,
> > 
> > QE could not reproduce this bug, sanity testing and code inspection based on
> > BZ1508886#c27 pass, do you think is it ok or still need ask customer to have
> > a try ?
> 
> The customer confirmed that a custom build with the patch solves the problem.
> 
> Thanks for doing the code inspection.  I would consider this BZ good to go.

Thanks Stefan for the confirmation.

Based on comment 5 and comment 7, set this bug to 'VERIFIED'.

Thanks.

Comment 9 Kevin Wolf 2017-12-21 18:28:07 UTC
*** Bug 1524770 has been marked as a duplicate of this bug. ***

Comment 12 shivapriya.o.hiremath 2018-02-02 21:54:54 UTC
We are facing the same issue in OSP 10 deployment where the spawning of a huge VM gets stuck. We would want to know how to get the custom build with the patch mentioned in this bugzilla. 

We have downloaded a source RPM (.src.rpm) from http://ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RHOS/SRPMS/, specifically qemu-kvm-rhev-2.9.0-16.el7_4.13.src.rpm.

Since this is a source RPM, we are yet to build the RPM from this file. We followed through the steps https://wiki.centos.org/HowTos/RebuildSRPM on how to build source RPMs, including installing dependencies, such as gcc and kernel-headers, but there are a ton of dependencies. 

We have used the 'yum-builddep <src rpm>' command to install some of the dependencies, but there are yet other packages that aren't available. These are the ff.:
•	bluez-libs-devel
•	brlapi-devel
•	gperftools-devel
•	libfdt-devel >= 1.4.3
•	lbiscsi-devel
•	libseccomp-devel >= 2.3.0
•	libssh2-devel
•	lzo-devel
•	pciutils-devel
•	snapp-devel

Can you guide us on how to add these dependencies on RHEL OSP and let us know if we are missing any repositories?

Comment 13 Miya Chen 2018-02-05 02:21:32 UTC
Shivapriya, I believe you need to get the official binary rpm through subscription channel.
@Oneata, do you have any guidance about that? Thanks.

Comment 14 shivapriya.o.hiremath 2018-02-05 19:48:33 UTC
Thank you, Miya. 

Since this needs to be done on OSP compute nodes where we don't enable any RHEL subscription, we will not be able to download from official channel.
Can you please guide us to obtain the binary rpm elsewhere, so that we can install it on compute node directly?

Comment 15 Martin Schuppert 2018-02-06 07:03:19 UTC
(In reply to shivapriya.o.hiremath from comment #14)
> Thank you, Miya. 
> 
> Since this needs to be done on OSP compute nodes where we don't enable any
> RHEL subscription, we will not be able to download from official channel.
> Can you please guide us to obtain the binary rpm elsewhere, so that we can
> install it on compute node directly?

qemu-kvm-rhev package for OSP10 was tracked in BZ1525502 and got released via
https://access.redhat.com/errata/RHSA-2018:0058 . You could load it from the 
errata.

Comment 16 Martin Schuppert 2018-02-06 12:58:57 UTC
(In reply to shivapriya.o.hiremath from comment #14)
> Thank you, Miya. 
> 
> Since this needs to be done on OSP compute nodes where we don't enable any
> RHEL subscription, we will not be able to download from official channel.

we strongly recommend to enable the needed channels [1] either direct to the RHN 
or using e.g. a satellite server and use the official procedure as described in [1]
to keep the environment up to date.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/director_installation_and_usage/index#sect-Repository_Requirements
[2] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/upgrading_red_hat_openstack_platform/index#sect-Updating_the_Environment

Comment 17 shivapriya.o.hiremath 2018-02-12 18:27:13 UTC
Thank you Miya and Martin. Updating the qemu-kvm-rhev package to the latest through official subscription channel fixed the issue.

Comment 18 Hanna Czenczek 2019-05-13 19:56:53 UTC
*** Bug 1667530 has been marked as a duplicate of this bug. ***