Bug 1650652

Summary:	lvm io_setup failing due to exhausted aio contexts
Product:	Red Hat Enterprise Linux 7	Reporter:	John Pittman <jpittman>
Component:	lvm2	Assignee:	LVM and device-mapper development team <lvm-team>
lvm2 sub component:	Other	QA Contact:	cluster-qe <cluster-qe>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	agk, alex.wang, bubrown, cmarthal, davem, dhcpme, fgarciad, hbiswas, heinzm, jbowen, jbrassow, jmagrini, jualvare, lvm-team, mcsontos, msnitzer, nweddle, prajnoha, prockai, rbednar, rhandlin, sbradley, tcleveng, teigland, zkabelac
Version:	7.6	Keywords:	ZStream
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	lvm2-2.02.184-1.el7	Doc Type:	If docs needed, set a value
Doc Text:	Previously, if there were no aio contexts available on the system, the lvm utility failed to set up aio and terminated unexpectedly with the "io_setup failed" error message. With this update, lvm now falls back to using sync I/O when io_setup fails. As a result, the described problem no longer occurs.	Story Points:	---
Clone Of:
Clones:	1656498 (view as bug list)		Environment:
Last Closed:	2019-08-06 13:10:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1656498

Description John Pittman 2018-11-16 18:16:47 UTC

Description of problem:

After update, io_setup failing due to exhausted aio contexts.

Nov 14 08:01:34 host lvm: io_setup failed
Nov 14 08:01:34 host lvm: Failed to create bcache io engine.
Nov 14 08:01:34 host lvm: No PV info found on /dev/xvdf for PVID .

[sosreport]$ cat proc/sys/fs/aio-max-nr 
65536
[sosreport]$ cat proc/sys/fs/aio-nr 
65536

Version-Release number of selected component (if applicable):

lvm2.x86_64 7:2.02.180-10.el7_6.2
device-mapper.x86_64 7:1.02.149-10.el7_6.2
kernel-3.10.0-957.el7

Steps to Reproduce:

In customer environment, all they have to do us update to levels
mentioned and run lvm command.

Actual results:

io_setup failure

Expected results:

io_setup success

Additional info:

Errors do not persist after downgrade lvm2.x86_64 7:2.02.177-4.el7 and
device-mapper.x86_64 7:1.02.146-4.el7.

Comment 4 David Teigland 2018-11-20 15:09:53 UTC

Fixed by these two commits which fall back to using sync io when io_setup fails
(or if use_aio=0 in lvm.conf):

https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=f8ce9bf3bc38df8f89021517d7967d94ace129df

https://sourceware.org/git/?p=lvm2.git;a=commitdiff;h=cb5405ded86209979eef54433d0b90942b2f976a

A possible workaround is to increase the system's aio request limit by setting a larger fs.aio-max-nr using sysctl.

Comment 12 jbowen 2018-12-10 02:55:10 UTC

A bug fix release, 2.02.183, has been tagged upstream: https://sourceware.org/git/?p=lvm2.git;a=tag;h=f21d9eb8527258479805f5a8132dd9e6e3e572d8

Comment 14 Roman Bednář 2019-07-04 07:28:33 UTC

Marking verified, lvm2 can now fallback to sync io if aio fails.


1) reduce max aio limit for easier reproducibility:

# sudo sysctl -w fs.aio-max-nr=5
fs.aio-max-nr = 5


2) create aio load in separate session:

# fio --name=randomwrite --ioengine=libaio --iodepth=1 --rw=randwrite --size=4k --numjobs=100 --runtime=1h --time_based


3) check aio number hits the limit:

# cat /proc/sys/fs/aio-nr
5


4) attempt 'vgs' command

# vgs
  Failed to set up async io, using sync io.
  VG            #PV #LV #SN Attr   VSize  VFree
  rhel_virt-371   1   2   0 wz--n- <7.00g    0



lvm2-2.02.185-2.el7.x86_64

Comment 16 errata-xmlrpc 2019-08-06 13:10:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2253