Bug 1895863 - Host deactivation causes vdsm.error Removing VG mapping failed (HE setup with iscsi/FC HostedStorage)
Summary: Host deactivation causes vdsm.error Removing VG mapping failed (HE setup with...
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: ovirt-hosted-engine-ha
Classification: oVirt
Component: Agent
Version: ---
Hardware: x86_64
OS: Linux
low
medium
Target Milestone: ---
: ---
Assignee: Asaf Rachmani
QA Contact: Polina
URL:
Whiteboard:
: 1895883 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-09 10:07 UTC by Polina
Modified: 2022-07-22 15:01 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2022-07-22 15:01:00 UTC
oVirt Team: Integration
Embargoed:
sbonazzo: ovirt-4.5-


Attachments (Terms of Use)
vdsm and engine logs (1.07 MB, application/x-tar)
2020-11-09 10:07 UTC, Polina
no flags Details
ovirt-hosted-engine-ha logs (660.25 KB, application/gzip)
2021-05-25 09:28 UTC, Polina
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github oVirt ovirt-hosted-engine-ha issues 19 0 None open Host deactivation causes vdsm.error Removing VG mapping failed (HE setup with iscsi/FC HostedStorage) 2022-07-22 15:00:59 UTC

Internal Links: 1986732

Description Polina 2020-11-09 10:07:42 UTC
Created attachment 1727728 [details]
vdsm and engine logs

Description of problem:
Every time (each HE setup with blocked HostedStorage (iscsi or FC), I sent host deactivation to any host in HE env I see in vdsm.log error [storage.LVM] Removing VG mapping failed: Could not remove mapping:

Version-Release number of selected component (if applicable):
ovirt-engine-4.4.3.8-0.1.el8ev.noarch
vdsm-4.40.35-1.el8ev.x86_64

How reproducible:100% on HE env with blocked HostedStorage (iscsi or FC)


Steps to Reproduce:

1. 
POST https://{{host}}/ovirt-engine/api/hosts/6c3b8522-8ea8-4e6e-a331-3bcd334db6df/deactivate
<action>
    <async>true</async>
    <grace_period>
        <expiry>10</expiry>
    </grace_period>
</action>

Actual results:
The host is deactivated after this and no errors in engine.log :
In the attached log:
2020-11-03 14:31:26,624+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [ef370003-50ee-419e-a4e1-6cbb53e7ce76] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Host host_mixed_2 was switched to Maintenance mode by admin@internal-authz.

the error in vdsm.log (attached)
2020-11-03 14:33:46,695+0200 WARN  (monitor/ebdf925) [storage.LVM] All 2 tries have failed: cmd=['/sbin/lvm', 'vgchange', '--config', 'devices {  preferred_names=["^/dev/mapper/"]  ignore_suspended_devices=1  write_cache_state=0  disable_after_error_count=3  filter=["a|^/dev/mapper/3600a09803830447a4f244c4657595063$|^/dev/mapper/3600a09803830447a4f244c4657595064$|^/dev/mapper/3600a09803830447a4f244c4657595065$|^/dev/mapper/3600a09803830447a4f244c4657595066$|^/dev/mapper/3600a09803830447a4f244c4657595067$|^/dev/mapper/3600a09803830447a4f244c4657595068$|^/dev/mapper/3600a09803830447a4f244c4657595069$|^/dev/mapper/3600a09803830447a4f244c46575a6d6d$|", "r|.*|"]  hints="none"  obtain_device_list_from_udev=0 } global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 } backup {  retain_min=50  retain_days=0 }', '--available', 'n', 'ebdf925a-19f7-4254-9458-c283872eded3'] rc=5 err=['  Logical volume ebdf925a-19f7-4254-9458-c283872eded3/9587b2a0-a11a-44bd-a1c3-4edd59cf357d in use.', '  Can\'t deactivate volume group "ebdf925a-19f7-4254-9458-c283872eded3" with 1 open logical volume(s)'] (lvm:538)
2020-11-03 14:33:46,696+0200 INFO  (monitor/ebdf925) [storage.LVM] Error deactivating VG ebdf925a-19f7-4254-9458-c283872eded3: rc=5 out=[] err=['  Logical volume ebdf925a-19f7-4254-9458-c283872eded3/9587b2a0-a11a-44bd-a1c3-4edd59cf357d in use.', '  Can\'t deactivate volume group "ebdf925a-19f7-4254-9458-c283872eded3" with 1 open logical volume(s)'] (lvm:1491)
2020-11-03 14:33:46,773+0200 ERROR (monitor/ebdf925) [storage.LVM] Removing VG mapping failed: Could not remove mapping: Command ['/usr/sbin/dmsetup', 'remove', 'ebdf925a--19f7--4254--9458--c283872eded3-9587b2a0--a11a--44bd--a1c3--4edd59cf357d'] failed with rc=1 out=b'' err=b'device-mapper: remove ioctl on ebdf925a--19f7--4254--9458--c283872eded3-9587b2a0--a11a--44bd--a1c3--4edd59cf357d  failed: Device or resource busy\nCommand failed.\n' (lvm:1167)


Expected results:


Additional info:

Comment 1 Arik 2020-11-09 12:07:45 UTC
Not sure it's an agent-related problem but that's what we suspect if this problem is hosted-engine specific

Comment 2 Polina 2020-11-09 12:08:31 UTC
*** Bug 1895883 has been marked as a duplicate of this bug. ***

Comment 3 Asaf Rachmani 2021-05-03 08:29:40 UTC
Polina, can you please share the ovirt-hosted-engine-ha log files?

Comment 4 Polina 2021-05-25 09:28:18 UTC
Created attachment 1786779 [details]
ovirt-hosted-engine-ha logs

provided ovirt-hosted-engine-ha logs

Comment 5 Asaf Rachmani 2022-04-28 12:21:36 UTC
Polina, there were some changes in this area recently, can you please re-test this with the latest 4.5 builds?

Comment 6 Polina 2022-05-01 08:04:53 UTC
Hi Asaf,

I checked now on 4.5 setup (vdsm-4.50.0.13-1.el8ev.x86_64, ovirt-engine-4.5.0.5-0.7.el8ev.noarch), HE on ISCSI SD.
host deactivation causes the following in vdsm.log

[root@oncilla04 vdsm]# tail -f vdsm.log |grep ERRO
2022-05-01 10:53:51,517+0300 ERROR (monitor/734e920) [storage.lvm] Removing VG mapping failed: Command ['/usr/sbin/dmsetup', 'remove', '734e920d--938f--42f5--b069--f65858e11e3b-53bfd425--5276--4da9--a5b1--23eb4d690f72'] failed with rc=1 out=b'' err=b'device-mapper: remove ioctl on 734e920d--938f--42f5--b069--f65858e11e3b-53bfd425--5276--4da9--a5b1--23eb4d690f72  failed: Device or resource busy\nCommand failed.\n' (lvm:1112)
2022-05-01 10:53:51,538+0300 ERROR (monitor/734e920) [storage.lvm] Removing VG mapping failed: Command ['/usr/sbin/dmsetup', 'remove', '734e920d--938f--42f5--b069--f65858e11e3b-05ff219b--81e5--4c86--b5c5--b137b6968f12'] failed with rc=1 out=b'' err=b'device-mapper: remove ioctl on 734e920d--938f--42f5--b069--f65858e11e3b-05ff219b--81e5--4c86--b5c5--b137b6968f12  failed: Device or resource busy\nCommand failed.\n' (lvm:1112)
2022-05-01 10:57:36,384+0300 ERROR (jsonrpc/3) [storage.volumemanifest] Internal block device read failure: 'name=/dev/734e920d-938f-42f5-b069-f65858e11e3b/metadata, offset=1114112, size=512' (blockVolume:103)
2022-05-01 10:57:36,533+0300 ERROR (jsonrpc/6) [storage.volumemanifest] Internal block device read failure: 'name=/dev/734e920d-938f-42f5-b069-f65858e11e3b/metadata, offset=1089536, size=512' (blockVolume:103)
2022-05-01 10:57:36,533+0300 ERROR (jsonrpc/6) [storage.taskmanager.task] (Task='ef55bdff-51e3-4aa1-a9d6-37205fc473d6') Unexpected error (task:877)
2022-05-01 10:57:36,534+0300 ERROR (jsonrpc/6) [storage.dispatcher] FINISH prepareImage error=Error while processing volume meta data: ("('734e920d-938f-42f5-b069-f65858e11e3b', 5): Internal block device read failure: 'name=/dev/734e920d-938f-42f5-b069-f65858e11e3b/metadata, offset=1089536, size=512'",) (dispatcher:83)

Comment 7 Sandro Bonazzola 2022-07-22 15:01:00 UTC
Moved to GitHub https://github.com/oVirt/ovirt-hosted-engine-ha/issues/19


Note You need to log in before you can comment on or make changes to this bug.