1428637 – LVM commands scan devices rejected by filter

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1428637 - LVM commands scan devices rejected by filter

Summary: LVM commands scan devices rejected by filter

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	6.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	rc
Target Release:	---
Assignee:	LVM and device-mapper development team
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1424853 1429203
TreeView+	depends on / blocked

Reported:	2017-03-03 01:21 UTC by Nir Soffer
Modified:	2020-05-14 15:42 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-03-16 16:54:32 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Vdsm log with example lvm commands showing this issue (119.91 KB, application/x-xz) 2017-03-03 01:21 UTC, Nir Soffer	no flags	Details
Output of lvs command with -vvvv (512.70 KB, text/plain) 2017-03-03 22:23 UTC, Nir Soffer	no flags	Details
Output of vgs -vvvv with a filter (510.85 KB, text/plain) 2017-03-03 22:28 UTC, Nir Soffer	no flags	Details
Output of lvs command with -vvvv and a filter from user machine (67.30 KB, application/x-xz) 2017-03-04 00:53 UTC, Nir Soffer	no flags	Details
Output of lvs command with -vvvv and a filter from user machine (95.10 KB, application/x-xz) 2017-03-04 00:55 UTC, Nir Soffer	no flags	Details
Output of vgs command with -vvvv and a filter from user machine (67.30 KB, application/x-xz) 2017-03-04 00:56 UTC, Nir Soffer	no flags	Details
Show Obsolete (1) View All

Description Nir Soffer 2017-03-03 01:21:23 UTC

Created attachment 1259402 [details]
Vdsm log with example lvm commands showing this issue

Description of problem:

Using a filter to accept only certain devices and rejecting any other device, lvm
commands such as vgs or lvchange seems to access rejected devices.

This command should access only /dev/mapper/3514f0c54f3a001a2

Thread-36157::DEBUG::2017-02-12 05:01:05,398::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm lvchange --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspend
ed_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/3514f0c54f3a001a2|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  
prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --autobackup n --available n 475f3971-83e8-48d4-af55-957b84c5b42e/8238f824-bd5e-47
31-9436-4b096d7d8b4f 475f3971-83e8-48d4-af55-957b84c5b42e/f4ebee3c-db83-46f6-851f-ab81cf505af0 (cwd None)

But lvm complain about many other devices it should never try to access:

Thread-36157::DEBUG::2017-02-12 05:01:05,358::lvm::291::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  /dev/mapper/3514f0c54f3a001ab: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ab: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ab: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ab was disabled\n  /dev/mapper/3514f0c54f3a001ac: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ac: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ac: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ac was disabled\n  /dev/mapper/3514f0c54f3a001aa: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001aa: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001aa: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001aa was disabled\n  /dev/mapper/3514f0c54f3a001ae: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ae: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ae: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ae was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata was disabled\n  /dev/mapper/3514f0c54f3a001ad: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ad: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ad: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ad was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox was disabled\n  /dev/mapper/3514f0c54f3a001af: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001af: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001af: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001af was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases was disabled\n  /dev/mapper/3514f0c54f3a001b0: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b0: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b0: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b0 was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox was disabled\n  /dev/mapper/3514f0c54f3a001b1: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b1: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b1: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b1 was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master was disabled\n  /dev/mapper/3514f0c54f3a001b2: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b2: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b2: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b2 was disabled\n  /dev/mapper/3514f0c54f3a001b3: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b3: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b3: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b3 was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox was disabled\n  /dev/mapper/3514f0c54f3a001b4: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b4: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b4: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b4 was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b: read failed after 0 of 4096 at 107374116864: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b: read failed after 0 of 4096 at 107374174208: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d: read failed after 0 of 4096 at 107374116864: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d: read failed after 0 of 4096 at 107374174208: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3: read failed after 0 of 4096 at 107374116864: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3: read failed after 0 of 4096 at 107374174208: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3 was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf: read failed after 0 of 4096 at 80530571264: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf: read failed after 0 of 4096 at 80530628608: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67: read failed after 0 of 4096 at 80530571264: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67: read failed after 0 of 4096 at 80530628608: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67 was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8 was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master was disabled\n'; <rc> = 0

Extracting the devices from lvm stderr, we see:

>>> device = re.compile(r"/dev/mapper/[a-zA-Z0-9-]+")
>>> pprint.pprint(set(device.findall(s)))
set(['/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox',
     '/dev/mapper/3514f0c54f3a001aa',
     '/dev/mapper/3514f0c54f3a001ab',
     '/dev/mapper/3514f0c54f3a001ac',
     '/dev/mapper/3514f0c54f3a001ad',
     '/dev/mapper/3514f0c54f3a001ae',
     '/dev/mapper/3514f0c54f3a001af',
     '/dev/mapper/3514f0c54f3a001b0',
     '/dev/mapper/3514f0c54f3a001b1',
     '/dev/mapper/3514f0c54f3a001b2',
     '/dev/mapper/3514f0c54f3a001b3',
     '/dev/mapper/3514f0c54f3a001b4',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox'])

Version-Release number of selected component (if applicable):
lvm2-2.02.118-3.el6_7.4.x86_64
lvm2-libs-2.02.118-3.el6_7.4.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Check lvm commands in vdsm log

Actual results:
lvm access rejected devices

Expected results:
lvm should not access rejected devices

Additional info:
When there are lot of active lvs (this case there are 5600 lvs), commands such as
lvchange on a single lv are stuck for hours during startup. The host cannot be 
activated without restarting vdsm manually, which kill the stuck lvchange 
commands.

I suspect that scanning 5600 devices with broken filter is related, if not the root
cause of this.

This looks like bug 1344206 that was fixed in rhel 7.2.

Comment 5 Greg Scott 2017-03-03 13:56:11 UTC

I just pasted the request into case 01797770.  I'll keep the needs info turned on until we get output from them.

- Greg

Comment 6 Nir Soffer 2017-03-03 22:23:09 UTC

Created attachment 1259746 [details]
Output of lvs command with -vvvv

This is the output of this command on rhel 6.7 system:

lvs -vvvv --config 'devices { filter = ["a|/dev/vda2|", "r|.*|"] }' vg0

The command ignores the filter, and opens many rejected devices:

# egrep "Open.+/dev/mapper" lvs--vvvv.out | head -10
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT

Comment 7 Nir Soffer 2017-03-03 22:24:11 UTC

Forgot to mention that this was tested with

# rpm -qa | grep lvm2
lvm2-2.02.118-2.el6.x86_64
lvm2-libs-2.02.118-2.el6.x86_64

Comment 8 Nir Soffer 2017-03-03 22:28:53 UTC

Created attachment 1259748 [details]
Output of vgs -vvvv with a filter

This is the output of:

vgs -vvvv --config 'devices { filter = ["a|/dev/vda2|", "r|.*|"] }' vg0 > vgs--vvvv.out 2>&1

Same issue, opening rejected devices:

# egrep "Open.+/dev/mapper" vgs--vvvv.out | head -10
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT

Comment 9 Nir Soffer 2017-03-03 22:30:18 UTC

David, isn't this bug 1344206?

Comment 10 Nir Soffer 2017-03-04 00:53:45 UTC

Created attachment 1259814 [details]
Output of lvs command with -vvvv and a filter from user machine

This is the output of 

lvs -vvvv \
    --config 'devices { filter = ["a|/dev/mapper/3514f0c54f3a001a2|", "r|.*|"] }' \
    475f3971-83e8-48d4-af55-957b84c5b42e > lvs--vvvv.out 2>&1

on user machine, showing the same results.

Comment 11 Nir Soffer 2017-03-04 00:55:02 UTC

Created attachment 1259815 [details]
Output of lvs command with -vvvv and a filter from user machine

Comment 12 Nir Soffer 2017-03-04 00:56:14 UTC

Created attachment 1259816 [details]
Output of vgs command with -vvvv and a filter from user machine

This is the output of

vgs -vvvv \
    --config 'devices { filter = ["a|/dev/mapper/3514f0c54f3a001a2|", "r|.*|"] }' \
    475f3971-83e8-48d4-af55-957b84c5b42e > vgs--vvvv.out 2>&1

On a user machine, showing the same results.

Comment 13 Nir Soffer 2017-03-04 14:45:32 UTC

This seems to be fixed in rhel 6.8:

# lvs -vvvv --config 'devices { filter = ["a|/dev/vda2|", "r|.*|"] }' vg0 2>&1 | grep Open
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /etc/lvm/lvm.conf RO

# rpm -qa | grep lvm2
lvm2-2.02.143-7.el6.x86_64
lvm2-libs-2.02.143-7.el6.x86_64

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.8 (Santiago)

Customers affected by this should upgrade to 6.8 or 7.3.

Comment 14 David Teigland 2017-03-06 16:58:45 UTC

(In reply to Nir Soffer from comment #9)
> David, isn't this bug 1344206?

Yes

Comment 15 Nir Soffer 2017-03-08 19:17:30 UTC

This looks like a duplicate of 1344206 - it does not include this patch:

commit b77497cbd86a854add3ffa4ce09a35f165c0d2ba
Author: Peter Rajnoha <prajnoha>
Date:   Tue Sep 8 15:03:15 2015 +0200

    filters: make sure regex filter is evaluated before any filter that needs disk access

Note You need to log in before you can comment on or make changes to this bug.