1546538 – Provide a way to disable lvm2-lvmetad.service and lvm2-lvmetad.socket on a hypervisor

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1546538 - Provide a way to disable lvm2-lvmetad.service and lvm2-lvmetad.socket on a hypervisor

Summary: Provide a way to disable lvm2-lvmetad.service and lvm2-lvmetad.socket on a hy...

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	LVM and device-mapper development team
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1462792
TreeView+	depends on / blocked

Reported:	2018-02-18 13:09 UTC by Nir Soffer
Modified:	2021-09-03 12:38 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-08-19 22:43:36 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Nir Soffer 2018-02-18 13:09:22 UTC

Description of problem:

lvmdetad service is not compatible with the clustered lvm storage used by RHV.
In the past, RHV used to disable lvmetad for lvm commands ran by vdsm, but it 
was not good enough, and may lead to data corruption in certain cases.
(See bug 1403836, bug 1374545).

On a hypervisor managed by RHV, lvmetad service is harmful, and we *never* want
to run it, or allow any command that enable it.

To avoid the issues, RHV is:

1. Setting global/use_lvmeta to 0 in /etc/lvm/lvmlocal.conf.
2. Disable and mask lvm2-lvmetad.service
3. Disable and mask lvm2-lvmetad.socket. Since we never want to run lvmetad
   service, we don't need socket activation for it.

This works, except this warning in /var/log/messages:

    "systemd: Cannot add dependency job for unit lvm2-lvmetad.socket,
    ignoring: Unit is masked."

This warning is alarming users, see bug 1462792.

We tried to not disable and mask the socket, but you still get warnings if the
service is masked, so the only way to avoid the warnings now is to unmask both
the socket and service, and disable the service.

With this configuration, the service will start once any command is overriding 
global/use_levmetad using --config.


Here is example session, showing what happens after unmasking and enabling
lvm2-lvmetad.service and lvm2-lvmetad.socket.

# systemctl unmask lvm2-lvmetad.service lvm2-lvmetad.socket
Removed symlink /etc/systemd/system/lvm2-lvmetad.service.
Removed symlink /etc/systemd/system/lvm2-lvmetad.socket.


# systemctl enable lvm2-lvmetad.service lvm2-lvmetad.socket


# systemctl start lvm2-lvmetad.socket


# systemctl status lvm2-lvmetad.socket


# systemctl status lvm2-lvmetad.socket
● lvm2-lvmetad.socket - LVM2 metadata daemon socket
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Fri 2017-12-08 16:01:25 IST; 13s ago
     Docs: man:lvmetad(8)
   Listen: /run/lvm/lvmetad.socket (Stream)

Dec 08 16:01:25 voodoo6.tlv.redhat.com systemd[1]: Listening on LVM2 metadata daemon socket.
Dec 08 16:01:25 voodoo6.tlv.redhat.com systemd[1]: Starting LVM2 metadata daemon socket.


# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:lvmetad(8)

Dec 07 04:42:13 voodoo6.tlv.redhat.com systemd[1]: Cannot add dependency job for unit lvm2-lvmetad.service, ignoring: Unit is masked.
...


Now lets run lvs command trying to use lvmetad:

# lvs --config 'global { use_lvmetad = 1 }'
  WARNING: Device for PV bSRiP0-GjWy-IbWV-BuAU-yoRi-XDPQ-HEr1CX not found or rejected by a filter.
  WARNING: Device for PV VblUHM-IuSm-KurY-4wm9-KIJd-FeLY-Ge5Jd0 not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/metadata while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/outbox while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/xleases while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/leases while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/ids while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/inbox while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV 507281e9-0205-444f-aed7-750d062a70e1/master while checking used and assumed devices.
  WARNING: Device for PV nEGdUC-xyqA-jG2V-eCK1-ap5H-ACAN-H85WTh not found or rejected by a filter.
  WARNING: Couldn't find all devices for LV ovirt-local/pool0_tmeta while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV ovirt-local/pool0_tdata while checking used and assumed devices.
  LV                                   VG                                   Attr       LSize   Pool  Origin Data%  Meta%  Move Log Cpy%Sync Convert
  30d786c0-312f-4d0e-856c-67734fb8136b 507281e9-0205-444f-aed7-750d062a70e1 -wi-----p- 128.00m                                                     
  c0a6e665-0f10-42e7-a552-9108c9b26153 507281e9-0205-444f-aed7-750d062a70e1 -wi-----p- 128.00m                                                     
  ids                                  507281e9-0205-444f-aed7-750d062a70e1 -wi-ao--p- 128.00m                                                     
  inbox                                507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 128.00m                                                     
  leases                               507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p-   2.00g                                                     
  master                               507281e9-0205-444f-aed7-750d062a70e1 -wi-ao--p-   1.00g                                                     
  metadata                             507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 512.00m                                                     
  outbox                               507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p- 128.00m                                                     
  xleases                              507281e9-0205-444f-aed7-750d062a70e1 -wi-a---p-   1.00g                                                     
  pool0                                ovirt-local                          twi---tzp-  40.00g              0.00   0.49                            
  test                                 ovirt-local                          Vwi-a-tzp-  10.00g pool0        0.00                                   
  lv_home                              vg0                                  -wi-ao---- 924.00m                                                     
  lv_root                              vg0                                  -wi-ao----  14.60g                                                     
  lv_swap                              vg0                                  -wi-ao----   4.00g                                                     
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).


# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-12-08 16:02:22 IST; 6min ago
     Docs: man:lvmetad(8)
 Main PID: 25128 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─25128 /usr/sbin/lvmetad -f

Dec 08 16:02:22 voodoo6.tlv.redhat.com systemd[1]: Started LVM2 metadata daemon.
Dec 08 16:02:22 voodoo6.tlv.redhat.com systemd[1]: Starting LVM2 metadata daemon...


The service was started - why would we run a service which is harmful on this
host?


Now we get warnings on every lvm command:

# lvs
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
  LV      VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home vg0 -wi-ao---- 924.00m                                                    
  lv_root vg0 -wi-ao----  14.60g                                                    
  lv_swap vg0 -wi-ao----   4.00g                                                    


Here lvs command as vdsm run it:

# lvs --config 'global { use_lvmetad=0 }'
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
  LV      VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home vg0 -wi-ao---- 924.00m                                                    
  lv_root vg0 -wi-ao----  14.60g                                                    
  lv_swap vg0 -wi-ao----   4.00g                                                    
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).

These warnings will be logged to vdsm log, for every lvm command run by vdsm.


If we stop the service, we stop getting the warnings:

# systemctl stop lvm2-lvmetad.service
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
  lvm2-lvmetad.socket

# lvs --config 'global { use_lvmetad=0 }'
  LV      VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home vg0 -wi-ao---- 924.00m                                                    
  lv_root vg0 -wi-ao----  14.60g                                                    
  lv_swap vg0 -wi-ao----   4.00g                                                    


With current oVirt configuration - masking lvm2-lvmetad.*

# lvs --config 'global { use_lvmetad=1 }'
  /run/lvm/lvmetad.socket: connect failed: Connection refused
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  LV      VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_home vg0 -wi-ao---- 924.00m                                                    
  lv_root vg0 -wi-ao----  14.60g                                                    
  lv_swap vg0 -wi-ao----   4.00g  


Looks like there is unneeded dependency on lvmetad socket/service during boot.

Looking in lvm-devel mailing list, this issue may be solved by changing the order
of the services:
https://www.redhat.com/archives/lvm-devel/2017-October/msg00035.html
https://www.redhat.com/archives/lvm-devel/2017-October/msg00039.html


Version-Release number of selected component (if applicable):
Reported in RHEL 7.4.

Comment 2 Klaas Demter 2018-08-23 08:16:58 UTC

This issue pops up again when updating to rhel 7.6 beta. During update I get alarming messages like this:
  Updating   : 7:lvm2-2.02.180-1.el7.x86_64                                                                                                                                               220/916
Failed to execute operation: Cannot send after transport endpoint shutdown
Failed to start lvm2-lvmetad.socket: Unit is masked.

If you won't correct this for 7.6 maybe write something into the release notes about this error being acceptable during update.

Comment 3 Nir Soffer 2018-08-26 14:44:50 UTC

(In reply to Klaas Demter from comment #2)
lvmetad was removed upstream, so at least we have a long term solution:

commit 117160b27e510dceb1ed6acf995115c040acd88d
Author: David Teigland <teigland>
Date:   Tue Jul 10 13:39:29 2018 -0500

    Remove lvmetad
    
    Native disk scanning is now both reduced and
    async/parallel, which makes it comparable in
    performance (and often faster) when compared
    to lvm using lvmetad.
    
    Autoactivation now uses local temp files to record
    online PVs, and no longer requires lvmetad.
    
    There should be no apparent command-level change
    in behavior.

I hope that we can have a minimal short term change making it possible to disable
lvmetad without these warnings.

David, do you think this can be improved in 7.x?

Comment 4 Klaas Demter 2018-08-27 08:07:45 UTC

What does long term mean in this scenario -- is that change going to get into rhel8? If I see this correctly it's not in the latest 2.02 releases

Comment 5 David Teigland 2018-08-27 14:29:17 UTC

lvmetad won't be in rhel8, but it will continue to cause problems for the life of rhel7.  The best solution to this whole mess would be for us to move lvmetad into a separate package that could optionally be installed/uninstalled, so if you didn't intstall the lvmetad package then you wouldn't have the lvmetad daemon or systemd files installed on the system.  How does it work if you leave the unit files in place and enabled, but move the lvmetad binary somewhere else?  Doing that would probably break a dozen rules, but it's basically the correct solution to the problem: we never want to allow lvmetad to run on the system.

As long as the lvmetad binary exists on the system, it will be possible for an admin to start and use it, even if you completely disable all the systemd-related files.

Apart from that, it sounds like it's impossible to completely disable all lvmetad-related systemd files and completely avoid all error messages.

Comment 6 kailas 2019-06-24 11:30:10 UTC

we have this issue on physical server which has 1000+ lun and we are getting this issue.

we have disabled the lvmetad and trying to update the system but it is getting failed with below error. could you please help on this.

Scriptlet output:
   1 Failed to execute operation: Cannot send after transport endpoint shutdown
   2 Failed to start lvm2-lvmetad.socket: Unit is masked.

Comment 7 kailas 2019-06-24 11:50:20 UTC

is this bug related with 3.10.0-957.21.2.el7.x86_64  kernel as we are getting issue with this kernel and  server has 1000+ lun present on the server

Note You need to log in before you can comment on or make changes to this bug.