Bug 1428637

Summary: LVM commands scan devices rejected by filter
Product: Red Hat Enterprise Linux 6 Reporter: Nir Soffer <nsoffer>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
lvm2 sub component: Default / Unclassified (RHEL6) QA Contact: cluster-qe <cluster-qe>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: urgent    
Priority: unspecified CC: agk, gscott, heinzm, jbrassow, jentrena, mkalinin, msnitzer, prajnoha, prockai, teigland, zkabelac
Version: 6.7   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-16 16:54:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1424853, 1429203    
Attachments:
Description Flags
Vdsm log with example lvm commands showing this issue
none
Output of lvs command with -vvvv
none
Output of vgs -vvvv with a filter
none
Output of lvs command with -vvvv and a filter from user machine
none
Output of lvs command with -vvvv and a filter from user machine
none
Output of vgs command with -vvvv and a filter from user machine none

Description Nir Soffer 2017-03-03 01:21:23 UTC
Created attachment 1259402 [details]
Vdsm log with example lvm commands showing this issue

Description of problem:

Using a filter to accept only certain devices and rejecting any other device, lvm
commands such as vgs or lvchange seems to access rejected devices.

This command should access only /dev/mapper/3514f0c54f3a001a2

Thread-36157::DEBUG::2017-02-12 05:01:05,398::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm lvchange --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspend
ed_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/3514f0c54f3a001a2|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  
prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --autobackup n --available n 475f3971-83e8-48d4-af55-957b84c5b42e/8238f824-bd5e-47
31-9436-4b096d7d8b4f 475f3971-83e8-48d4-af55-957b84c5b42e/f4ebee3c-db83-46f6-851f-ab81cf505af0 (cwd None)

But lvm complain about many other devices it should never try to access:

Thread-36157::DEBUG::2017-02-12 05:01:05,358::lvm::291::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '  /dev/mapper/3514f0c54f3a001ab: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ab: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ab: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ab was disabled\n  /dev/mapper/3514f0c54f3a001ac: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ac: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ac: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ac was disabled\n  /dev/mapper/3514f0c54f3a001aa: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001aa: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001aa: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001aa was disabled\n  /dev/mapper/3514f0c54f3a001ae: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ae: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ae: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ae was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata was disabled\n  /dev/mapper/3514f0c54f3a001ad: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001ad: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001ad: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001ad was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox was disabled\n  /dev/mapper/3514f0c54f3a001af: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001af: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001af: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001af was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases was disabled\n  /dev/mapper/3514f0c54f3a001b0: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b0: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b0: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b0 was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox was disabled\n  /dev/mapper/3514f0c54f3a001b1: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b1: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b1: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b1 was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master was disabled\n  /dev/mapper/3514f0c54f3a001b2: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b2: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b2: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b2 was disabled\n  /dev/mapper/3514f0c54f3a001b3: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b3: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b3: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b3 was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox was disabled\n  /dev/mapper/3514f0c54f3a001b4: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/3514f0c54f3a001b4: read failed after 0 of 4096 at 7696581328896: Input/output error\n  /dev/mapper/3514f0c54f3a001b4: read failed after 0 of 4096 at 7696581386240: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/3514f0c54f3a001b4 was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox was disabled\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox was disabled\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b: read failed after 0 of 4096 at 107374116864: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b: read failed after 0 of 4096 at 107374174208: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox was disabled\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases was disabled\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox was disabled\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d: read failed after 0 of 4096 at 107374116864: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d: read failed after 0 of 4096 at 107374174208: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d was disabled\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3: read failed after 0 of 4096 at 107374116864: Input/output error\n  /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3: read failed after 0 of 4096 at 107374174208: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3 was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf: read failed after 0 of 4096 at 80530571264: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf: read failed after 0 of 4096 at 80530628608: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master was disabled\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67: read failed after 0 of 4096 at 80530571264: Input/output error\n  /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67: read failed after 0 of 4096 at 80530628608: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67 was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox was disabled\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master was disabled\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8 was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox was disabled\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata: read failed after 0 of 4096 at 536805376: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata: read failed after 0 of 4096 at 536862720: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases: read failed after 0 of 4096 at 2147418112: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases: read failed after 0 of 4096 at 2147475456: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox: read failed after 0 of 4096 at 134152192: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox: read failed after 0 of 4096 at 134209536: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox was disabled\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master: read failed after 0 of 4096 at 0: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master: read failed after 0 of 4096 at 1073676288: Input/output error\n  /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master: read failed after 0 of 4096 at 1073733632: Input/output error\n  WARNING: Error counts reached a limit of 3. Device /dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master was disabled\n'; <rc> = 0

Extracting the devices from lvm stderr, we see:

>>> device = re.compile(r"/dev/mapper/[a-zA-Z0-9-]+")
>>> pprint.pprint(set(device.findall(s)))
set(['/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-15b31af7--bfe2--4945--8492--a6733e98fc67',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-842da72d--9da4--4913--aac3--948de3e81ebf',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-8728341d--6c28--4d9e--b4b3--da20a05d427d',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-ids',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-inbox',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-leases',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-master',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-metadata',
     '/dev/mapper/018fd3a0--1513--4f80--81f0--ede47686cb8e-outbox',
     '/dev/mapper/3514f0c54f3a001aa',
     '/dev/mapper/3514f0c54f3a001ab',
     '/dev/mapper/3514f0c54f3a001ac',
     '/dev/mapper/3514f0c54f3a001ad',
     '/dev/mapper/3514f0c54f3a001ae',
     '/dev/mapper/3514f0c54f3a001af',
     '/dev/mapper/3514f0c54f3a001b0',
     '/dev/mapper/3514f0c54f3a001b1',
     '/dev/mapper/3514f0c54f3a001b2',
     '/dev/mapper/3514f0c54f3a001b3',
     '/dev/mapper/3514f0c54f3a001b4',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-ids',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-inbox',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-leases',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-master',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-metadata',
     '/dev/mapper/375bfc4e--0747--44b4--9f16--662436445797-outbox',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-ids',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-inbox',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-leases',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-master',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-metadata',
     '/dev/mapper/482fb94d--b6a4--4fae--8728--d6204f1fd273-outbox',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-1b9432e6--a02a--4fa1--99b0--f6a96840b2a3',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-ids',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-inbox',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-leases',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-master',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-metadata',
     '/dev/mapper/9e436258--191d--411e--9c8f--3d7165682e90-outbox',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-ids',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-inbox',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-leases',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-master',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-metadata',
     '/dev/mapper/a214b043--c558--433a--8a00--9c12b41d590f-outbox',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-ids',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-inbox',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-leases',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-master',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-metadata',
     '/dev/mapper/a962de39--4d11--4f45--b3e0--a3e6da75796e-outbox',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-ids',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-inbox',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-leases',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-master',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-metadata',
     '/dev/mapper/b38d008f--b93f--4650--ba91--2f41b3ad4b41-outbox',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-ids',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-inbox',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-leases',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-master',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-metadata',
     '/dev/mapper/c6e07169--a202--4a44--b2ae--440110ce0323-outbox',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ab46be75--4ffc--431f--abc6--65ca363db7d8',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-ids',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-inbox',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-leases',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-master',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-metadata',
     '/dev/mapper/f0a154ad--c212--4702--808a--7f371b4f8c8c-outbox',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ebc94b95--1eba--444a--88f0--5b7af11ece1b',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-ids',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-inbox',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-leases',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-master',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-metadata',
     '/dev/mapper/f8e8526c--02fa--4906--b593--3821f99cb392-outbox',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-ids',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-inbox',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-leases',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-master',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-metadata',
     '/dev/mapper/fc988f63--4051--44a6--988c--09a1ca9b1157-outbox'])

Version-Release number of selected component (if applicable):
lvm2-2.02.118-3.el6_7.4.x86_64
lvm2-libs-2.02.118-3.el6_7.4.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Check lvm commands in vdsm log

Actual results:
lvm access rejected devices

Expected results:
lvm should not access rejected devices

Additional info:
When there are lot of active lvs (this case there are 5600 lvs), commands such as
lvchange on a single lv are stuck for hours during startup. The host cannot be 
activated without restarting vdsm manually, which kill the stuck lvchange 
commands.

I suspect that scanning 5600 devices with broken filter is related, if not the root
cause of this.

This looks like bug 1344206 that was fixed in rhel 7.2.

Comment 5 Greg Scott 2017-03-03 13:56:11 UTC
I just pasted the request into case 01797770.  I'll keep the needs info turned on until we get output from them.

- Greg

Comment 6 Nir Soffer 2017-03-03 22:23:09 UTC
Created attachment 1259746 [details]
Output of lvs command with -vvvv

This is the output of this command on rhel 6.7 system:

lvs -vvvv --config 'devices { filter = ["a|/dev/vda2|", "r|.*|"] }' vg0

The command ignores the filter, and opens many rejected devices:

# egrep "Open.+/dev/mapper" lvs--vvvv.out | head -10
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT

Comment 7 Nir Soffer 2017-03-03 22:24:11 UTC
Forgot to mention that this was tested with

# rpm -qa | grep lvm2
lvm2-2.02.118-2.el6.x86_64
lvm2-libs-2.02.118-2.el6.x86_64

Comment 8 Nir Soffer 2017-03-03 22:28:53 UTC
Created attachment 1259748 [details]
Output of vgs -vvvv with a filter

This is the output of:

vgs -vvvv --config 'devices { filter = ["a|/dev/vda2|", "r|.*|"] }' vg0 > vgs--vvvv.out 2>&1

Same issue, opening rejected devices:

# egrep "Open.+/dev/mapper" vgs--vvvv.out | head -10
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052c81462a280847e8a6e3af8cd RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052a3e8afb36f74e72af6b7b03c RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT
#device/dev-io.c:533         Opened /dev/mapper/360014052761af2654a94a70a60a7ee3f RO O_DIRECT

Comment 9 Nir Soffer 2017-03-03 22:30:18 UTC
David, isn't this bug 1344206?

Comment 10 Nir Soffer 2017-03-04 00:53:45 UTC
Created attachment 1259814 [details]
Output of lvs command with -vvvv and a filter from user machine

This is the output of 

lvs -vvvv \
    --config 'devices { filter = ["a|/dev/mapper/3514f0c54f3a001a2|", "r|.*|"] }' \
    475f3971-83e8-48d4-af55-957b84c5b42e > lvs--vvvv.out 2>&1

on user machine, showing the same results.

Comment 11 Nir Soffer 2017-03-04 00:55:02 UTC
Created attachment 1259815 [details]
Output of lvs command with -vvvv and a filter from user machine

Comment 12 Nir Soffer 2017-03-04 00:56:14 UTC
Created attachment 1259816 [details]
Output of vgs command with -vvvv and a filter from user machine

This is the output of

vgs -vvvv \
    --config 'devices { filter = ["a|/dev/mapper/3514f0c54f3a001a2|", "r|.*|"] }' \
    475f3971-83e8-48d4-af55-957b84c5b42e > vgs--vvvv.out 2>&1

On a user machine, showing the same results.

Comment 13 Nir Soffer 2017-03-04 14:45:32 UTC
This seems to be fixed in rhel 6.8:

# lvs -vvvv --config 'devices { filter = ["a|/dev/vda2|", "r|.*|"] }' vg0 2>&1 | grep Open
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /dev/vda2 RO O_DIRECT
#device/dev-io.c:559         Opened /etc/lvm/lvm.conf RO

# rpm -qa | grep lvm2
lvm2-2.02.143-7.el6.x86_64
lvm2-libs-2.02.143-7.el6.x86_64

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.8 (Santiago)

Customers affected by this should upgrade to 6.8 or 7.3.

Comment 14 David Teigland 2017-03-06 16:58:45 UTC
(In reply to Nir Soffer from comment #9)
> David, isn't this bug 1344206?

Yes

Comment 15 Nir Soffer 2017-03-08 19:17:30 UTC
This looks like a duplicate of 1344206 - it does not include this patch:

commit b77497cbd86a854add3ffa4ce09a35f165c0d2ba
Author: Peter Rajnoha <prajnoha>
Date:   Tue Sep 8 15:03:15 2015 +0200

    filters: make sure regex filter is evaluated before any filter that needs disk access