Created attachment 1727728 [details] vdsm and engine logs Description of problem: Every time (each HE setup with blocked HostedStorage (iscsi or FC), I sent host deactivation to any host in HE env I see in vdsm.log error [storage.LVM] Removing VG mapping failed: Could not remove mapping: Version-Release number of selected component (if applicable): ovirt-engine-4.4.3.8-0.1.el8ev.noarch vdsm-4.40.35-1.el8ev.x86_64 How reproducible:100% on HE env with blocked HostedStorage (iscsi or FC) Steps to Reproduce: 1. POST https://{{host}}/ovirt-engine/api/hosts/6c3b8522-8ea8-4e6e-a331-3bcd334db6df/deactivate <action> <async>true</async> <grace_period> <expiry>10</expiry> </grace_period> </action> Actual results: The host is deactivated after this and no errors in engine.log : In the attached log: 2020-11-03 14:31:26,624+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [ef370003-50ee-419e-a4e1-6cbb53e7ce76] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Host host_mixed_2 was switched to Maintenance mode by admin@internal-authz. the error in vdsm.log (attached) 2020-11-03 14:33:46,695+0200 WARN (monitor/ebdf925) [storage.LVM] All 2 tries have failed: cmd=['/sbin/lvm', 'vgchange', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter=["a|^/dev/mapper/3600a09803830447a4f244c4657595063$|^/dev/mapper/3600a09803830447a4f244c4657595064$|^/dev/mapper/3600a09803830447a4f244c4657595065$|^/dev/mapper/3600a09803830447a4f244c4657595066$|^/dev/mapper/3600a09803830447a4f244c4657595067$|^/dev/mapper/3600a09803830447a4f244c4657595068$|^/dev/mapper/3600a09803830447a4f244c4657595069$|^/dev/mapper/3600a09803830447a4f244c46575a6d6d$|", "r|.*|"] hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', '--available', 'n', 'ebdf925a-19f7-4254-9458-c283872eded3'] rc=5 err=[' Logical volume ebdf925a-19f7-4254-9458-c283872eded3/9587b2a0-a11a-44bd-a1c3-4edd59cf357d in use.', ' Can\'t deactivate volume group "ebdf925a-19f7-4254-9458-c283872eded3" with 1 open logical volume(s)'] (lvm:538) 2020-11-03 14:33:46,696+0200 INFO (monitor/ebdf925) [storage.LVM] Error deactivating VG ebdf925a-19f7-4254-9458-c283872eded3: rc=5 out=[] err=[' Logical volume ebdf925a-19f7-4254-9458-c283872eded3/9587b2a0-a11a-44bd-a1c3-4edd59cf357d in use.', ' Can\'t deactivate volume group "ebdf925a-19f7-4254-9458-c283872eded3" with 1 open logical volume(s)'] (lvm:1491) 2020-11-03 14:33:46,773+0200 ERROR (monitor/ebdf925) [storage.LVM] Removing VG mapping failed: Could not remove mapping: Command ['/usr/sbin/dmsetup', 'remove', 'ebdf925a--19f7--4254--9458--c283872eded3-9587b2a0--a11a--44bd--a1c3--4edd59cf357d'] failed with rc=1 out=b'' err=b'device-mapper: remove ioctl on ebdf925a--19f7--4254--9458--c283872eded3-9587b2a0--a11a--44bd--a1c3--4edd59cf357d failed: Device or resource busy\nCommand failed.\n' (lvm:1167) Expected results: Additional info:
Not sure it's an agent-related problem but that's what we suspect if this problem is hosted-engine specific
*** Bug 1895883 has been marked as a duplicate of this bug. ***
Polina, can you please share the ovirt-hosted-engine-ha log files?
Created attachment 1786779 [details] ovirt-hosted-engine-ha logs provided ovirt-hosted-engine-ha logs
Polina, there were some changes in this area recently, can you please re-test this with the latest 4.5 builds?
Hi Asaf, I checked now on 4.5 setup (vdsm-4.50.0.13-1.el8ev.x86_64, ovirt-engine-4.5.0.5-0.7.el8ev.noarch), HE on ISCSI SD. host deactivation causes the following in vdsm.log [root@oncilla04 vdsm]# tail -f vdsm.log |grep ERRO 2022-05-01 10:53:51,517+0300 ERROR (monitor/734e920) [storage.lvm] Removing VG mapping failed: Command ['/usr/sbin/dmsetup', 'remove', '734e920d--938f--42f5--b069--f65858e11e3b-53bfd425--5276--4da9--a5b1--23eb4d690f72'] failed with rc=1 out=b'' err=b'device-mapper: remove ioctl on 734e920d--938f--42f5--b069--f65858e11e3b-53bfd425--5276--4da9--a5b1--23eb4d690f72 failed: Device or resource busy\nCommand failed.\n' (lvm:1112) 2022-05-01 10:53:51,538+0300 ERROR (monitor/734e920) [storage.lvm] Removing VG mapping failed: Command ['/usr/sbin/dmsetup', 'remove', '734e920d--938f--42f5--b069--f65858e11e3b-05ff219b--81e5--4c86--b5c5--b137b6968f12'] failed with rc=1 out=b'' err=b'device-mapper: remove ioctl on 734e920d--938f--42f5--b069--f65858e11e3b-05ff219b--81e5--4c86--b5c5--b137b6968f12 failed: Device or resource busy\nCommand failed.\n' (lvm:1112) 2022-05-01 10:57:36,384+0300 ERROR (jsonrpc/3) [storage.volumemanifest] Internal block device read failure: 'name=/dev/734e920d-938f-42f5-b069-f65858e11e3b/metadata, offset=1114112, size=512' (blockVolume:103) 2022-05-01 10:57:36,533+0300 ERROR (jsonrpc/6) [storage.volumemanifest] Internal block device read failure: 'name=/dev/734e920d-938f-42f5-b069-f65858e11e3b/metadata, offset=1089536, size=512' (blockVolume:103) 2022-05-01 10:57:36,533+0300 ERROR (jsonrpc/6) [storage.taskmanager.task] (Task='ef55bdff-51e3-4aa1-a9d6-37205fc473d6') Unexpected error (task:877) 2022-05-01 10:57:36,534+0300 ERROR (jsonrpc/6) [storage.dispatcher] FINISH prepareImage error=Error while processing volume meta data: ("('734e920d-938f-42f5-b069-f65858e11e3b', 5): Internal block device read failure: 'name=/dev/734e920d-938f-42f5-b069-f65858e11e3b/metadata, offset=1089536, size=512'",) (dispatcher:83)
Moved to GitHub https://github.com/oVirt/ovirt-hosted-engine-ha/issues/19