Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
lvmdetad service is not compatible with the clustered lvm storage used by RHV.
In the past, RHV used to disable lvmetad for lvm commands ran by vdsm, but it
was not good enough, and may lead to data corruption in certain cases.
(See bug 1403836, bug 1374545).
On a hypervisor managed by RHV, lvmetad service is harmful, and we *never* want
to run it, or allow any command that enable it.
To avoid the issues, RHV is:
1. Setting global/use_lvmeta to 0 in /etc/lvm/lvmlocal.conf.
2. Disable and mask lvm2-lvmetad.service
3. Disable and mask lvm2-lvmetad.socket. Since we never want to run lvmetad
service, we don't need socket activation for it.
This works, except this warning in /var/log/messages:
"systemd: Cannot add dependency job for unit lvm2-lvmetad.socket,
ignoring: Unit is masked."
This warning is alarming users, see bug 1462792.
We tried to not disable and mask the socket, but you still get warnings if the
service is masked, so the only way to avoid the warnings now is to unmask both
the socket and service, and disable the service.
With this configuration, the service will start once any command is overriding
global/use_levmetad using --config.
Here is example session, showing what happens after unmasking and enabling
lvm2-lvmetad.service and lvm2-lvmetad.socket.
# systemctl unmask lvm2-lvmetad.service lvm2-lvmetad.socket
Removed symlink /etc/systemd/system/lvm2-lvmetad.service.
Removed symlink /etc/systemd/system/lvm2-lvmetad.socket.
# systemctl enable lvm2-lvmetad.service lvm2-lvmetad.socket
# systemctl start lvm2-lvmetad.socket
# systemctl status lvm2-lvmetad.socket
# systemctl status lvm2-lvmetad.socket
● lvm2-lvmetad.socket - LVM2 metadata daemon socket
Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.socket; enabled; vendor preset: enabled)
Active: active (listening) since Fri 2017-12-08 16:01:25 IST; 13s ago
Docs: man:lvmetad(8)
Listen: /run/lvm/lvmetad.socket (Stream)
Dec 08 16:01:25 voodoo6.tlv.redhat.com systemd[1]: Listening on LVM2 metadata daemon socket.
Dec 08 16:01:25 voodoo6.tlv.redhat.com systemd[1]: Starting LVM2 metadata daemon socket.
# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:lvmetad(8)
Dec 07 04:42:13 voodoo6.tlv.redhat.com systemd[1]: Cannot add dependency job for unit lvm2-lvmetad.service, ignoring: Unit is masked.
...
Now lets run lvs command trying to use lvmetad:
# lvs --config 'global { use_lvmetad = 1 }'
WARNING: Device for PV bSRiP0-GjWy-IbWV-BuAU-yoRi-XDPQ-HEr1CX not found or rejected by a filter.
WARNING: Device for PV VblUHM-IuSm-KurY-4wm9-KIJd-FeLY-Ge5Jd0 not found or rejected by a filter.
WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/metadata while checking used and assumed devices.
WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/outbox while checking used and assumed devices.
WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/xleases while checking used and assumed devices.
WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/leases while checking used and assumed devices.
WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/ids while checking used and assumed devices.
WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/inbox while checking used and assumed devices.
WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/master while checking used and assumed devices.
WARNING: Device for PV nEGdUC-xyqA-jG2V-eCK1-ap5H-ACAN-H85WTh not found or rejected by a filter.
WARNING: Couldn't find all devices for LV ovirt-local/pool0_tmeta while checking used and assumed devices.
WARNING: Couldn't find all devices for LV ovirt-local/pool0_tdata while checking used and assumed devices.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
30d786c0-312f-4d0e-856c-67734fb8136b 507281e9-0205-444f-aed7-750d062a70e1 -wi-----p- 128.00m
c0a6e665-0f10-42e7-a552-9108c9b26153 507281e9-0205-444f-aed7-750d062a70e1 -wi-----p- 128.00m
ids 507281e9-0205-444f-aed7-750d062a70e1 -wi-ao--p- 128.00m
inbox 507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 128.00m
leases 507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 2.00g
master 507281e9-0205-444f-aed7-750d062a70e1 -wi-ao--p- 1.00g
metadata 507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 512.00m
outbox 507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 128.00m
xleases 507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 1.00g
pool0 ovirt-local twi---tzp- 40.00g 0.00 0.49
test ovirt-local Vwi-a-tzp- 10.00g pool0 0.00
lv_home vg0 -wi-ao---- 924.00m
lv_root vg0 -wi-ao---- 14.60g
lv_swap vg0 -wi-ao---- 4.00g
WARNING: Not using lvmetad because config setting use_lvmetad=0.
WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2017-12-08 16:02:22 IST; 6min ago
Docs: man:lvmetad(8)
Main PID: 25128 (lvmetad)
CGroup: /system.slice/lvm2-lvmetad.service
└─25128 /usr/sbin/lvmetad -f
Dec 08 16:02:22 voodoo6.tlv.redhat.com systemd[1]: Started LVM2 metadata daemon.
Dec 08 16:02:22 voodoo6.tlv.redhat.com systemd[1]: Starting LVM2 metadata daemon...
The service was started - why would we run a service which is harmful on this
host?
Now we get warnings on every lvm command:
# lvs
WARNING: Not using lvmetad because config setting use_lvmetad=0.
WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home vg0 -wi-ao---- 924.00m
lv_root vg0 -wi-ao---- 14.60g
lv_swap vg0 -wi-ao---- 4.00g
Here lvs command as vdsm run it:
# lvs --config 'global { use_lvmetad=0 }'
WARNING: Not using lvmetad because config setting use_lvmetad=0.
WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home vg0 -wi-ao---- 924.00m
lv_root vg0 -wi-ao---- 14.60g
lv_swap vg0 -wi-ao---- 4.00g
WARNING: Not using lvmetad because config setting use_lvmetad=0.
WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
These warnings will be logged to vdsm log, for every lvm command run by vdsm.
If we stop the service, we stop getting the warnings:
# systemctl stop lvm2-lvmetad.service
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
lvm2-lvmetad.socket
# lvs --config 'global { use_lvmetad=0 }'
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home vg0 -wi-ao---- 924.00m
lv_root vg0 -wi-ao---- 14.60g
lv_swap vg0 -wi-ao---- 4.00g
With current oVirt configuration - masking lvm2-lvmetad.*
# lvs --config 'global { use_lvmetad=1 }'
/run/lvm/lvmetad.socket: connect failed: Connection refused
WARNING: Failed to connect to lvmetad. Falling back to device scanning.
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home vg0 -wi-ao---- 924.00m
lv_root vg0 -wi-ao---- 14.60g
lv_swap vg0 -wi-ao---- 4.00g
Looks like there is unneeded dependency on lvmetad socket/service during boot.
Looking in lvm-devel mailing list, this issue may be solved by changing the order
of the services:
https://www.redhat.com/archives/lvm-devel/2017-October/msg00035.htmlhttps://www.redhat.com/archives/lvm-devel/2017-October/msg00039.html
Version-Release number of selected component (if applicable):
Reported in RHEL 7.4.
This issue pops up again when updating to rhel 7.6 beta. During update I get alarming messages like this:
Updating : 7:lvm2-2.02.180-1.el7.x86_64 220/916
Failed to execute operation: Cannot send after transport endpoint shutdown
Failed to start lvm2-lvmetad.socket: Unit is masked.
If you won't correct this for 7.6 maybe write something into the release notes about this error being acceptable during update.
(In reply to Klaas Demter from comment #2)
lvmetad was removed upstream, so at least we have a long term solution:
commit 117160b27e510dceb1ed6acf995115c040acd88d
Author: David Teigland <teigland>
Date: Tue Jul 10 13:39:29 2018 -0500
Remove lvmetad
Native disk scanning is now both reduced and
async/parallel, which makes it comparable in
performance (and often faster) when compared
to lvm using lvmetad.
Autoactivation now uses local temp files to record
online PVs, and no longer requires lvmetad.
There should be no apparent command-level change
in behavior.
I hope that we can have a minimal short term change making it possible to disable
lvmetad without these warnings.
David, do you think this can be improved in 7.x?
lvmetad won't be in rhel8, but it will continue to cause problems for the life of rhel7. The best solution to this whole mess would be for us to move lvmetad into a separate package that could optionally be installed/uninstalled, so if you didn't intstall the lvmetad package then you wouldn't have the lvmetad daemon or systemd files installed on the system. How does it work if you leave the unit files in place and enabled, but move the lvmetad binary somewhere else? Doing that would probably break a dozen rules, but it's basically the correct solution to the problem: we never want to allow lvmetad to run on the system.
As long as the lvmetad binary exists on the system, it will be possible for an admin to start and use it, even if you completely disable all the systemd-related files.
Apart from that, it sounds like it's impossible to completely disable all lvmetad-related systemd files and completely avoid all error messages.
we have this issue on physical server which has 1000+ lun and we are getting this issue.
we have disabled the lvmetad and trying to update the system but it is getting failed with below error. could you please help on this.
Scriptlet output:
1 Failed to execute operation: Cannot send after transport endpoint shutdown
2 Failed to start lvm2-lvmetad.socket: Unit is masked.