RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1861459 - [RFE] Update the LVM commands so that volumes managed by the cluster resource LVM-activate w/ systemid are visible on all nodes.
Summary: [RFE] Update the LVM commands so that volumes managed by the cluster resource...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sos
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Pavel Moravec
QA Contact: Miroslav Hradílek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-28 17:04 UTC by Chad Newsom
Modified: 2021-05-18 14:48 UTC (History)
6 users (show)

Fixed In Version: sos-4.0-2.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 14:47:21 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github sosreport sos pull 2202 0 None closed [lvm2] Update LVm commands to support HA-LVM 2021-02-02 17:19:44 UTC

Description Chad Newsom 2020-07-28 17:04:42 UTC
1. Proposed title of this feature request

Update the LVM commands so that volumes managed by the cluster resource LVM-activate w/ systemid are visible on all nodes.


2. Who is the customer behind the request?

N/A

3. What is the nature and description of the request?


In a RHEL 8 HA cluster, we utilize lvm systemid to grant a node exclusive access to a VG, specifivally system_id_source = "uname" in /etc/lvm/lvm.conf. In doing this, standard LVM commands will skip a VG if its systemid does not match the local hostname. In practice this prevents a user from interacting with a VG on a cluster node where it is inactive, which is good. However when troubleshooting cluster issues, it would be helpful to have the LVM commands issued by sos altered so that they show all volume groups (even those whose systemid does not match the local host name and thus are considered foreign). 


Currently the sosreport will skip over a VG if its systemid does not match the local hostname(note the "Skipping foreign volume group cluster_vg" line):

[cnewsom@cnewsom rhel8-node-2.clust-Jul_28_11.06.50]$ cat sos_commands/lvm2/vgs_-v_-o_vg_mda_count_vg_mda_free_vg_mda_size_vg_mda_used_count_vg_tags_--config_global_locking_type_0_metadata_read_only_1 
  Reloading config files
  WARNING: locking_type (0) is deprecated, using --nolocking.
  WARNING: File locking is disabled.
  Skipping foreign volume group cluster_vg
  VG   Attr   Ext   #PV #LV #SN VSize   VFree VG UUID                                VProfile #VMda VMdaFree  VMdaSize  #VMdaUse VG Tags
  rhel wz--n- 4.00m   1   2   0 <14.00g    0  kMAVWQ-FCIP-p0UB-QaEo-U38N-9aha-J714e2              1   507.50k  1020.00k        1        
  Reloading config files



4. Why does the customer need this? (List the business requirements here)

This would make it easier for support to diagnose HA-LVM configuration issues. 

5. How would the customer like to achieve this? (List the functional requirements here)


 - Currently the sos_comands/lvm2/ directory contains the following commands:

lvs_-a_-o_lv_tags_devices_lv_kernel_read_ahead_lv_read_ahead_stripes_stripesize_--config_global_locking_type_0_metadata_read_only_1
pvs_-a_-v_-o_pv_mda_free_pv_mda_size_pv_mda_count_pv_mda_used_count_pe_start_--config_global_locking_type_0_metadata_read_only_1
pvscan_-v_--config_global_locking_type_0_metadata_read_only_1
vgdisplay_-vv_--config_global_locking_type_0_metadata_read_only_1
vgscan_-vvv_--config_global_locking_type_0_metadata_read_only_1
vgs_-v_-o_vg_mda_count_vg_mda_free_vg_mda_size_vg_mda_used_count_vg_tags_--config_global_locking_type_0_metadata_read_only_1


 - The pvs, lvs, and vgdisplay commands will just need to have "--foreign" added. lvs output for example:


BEFORE:

[root@rhel8-node-2 ~]# lvs -a -o+lv_tags,devices,lv_kernel_read_ahead,lv_read_ahead,stripes,stripesize --config 'global {locking_type = 0 metadata_read_only = 1}'
  WARNING: locking_type (0) is deprecated, using --nolocking.
  WARNING: File locking is disabled.
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert LV Tags Devices        KRahead Rahead #Str Stripe
  root rhel -wi-ao---- <12.50g                                                             /dev/vda2(384)   4.00m   auto    1     0 
  swap rhel -wi-ao----   1.50g                                                             /dev/vda2(0)     4.00m   auto    1     0 


AFTER:

[root@rhel8-node-2 ~]# lvs -a -o+lv_tags,devices,lv_kernel_read_ahead,lv_read_ahead,stripes,stripesize --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
  WARNING: locking_type (0) is deprecated, using --nolocking.
  WARNING: File locking is disabled.
  LV         VG         Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert LV Tags Devices        KRahead Rahead #Str Stripe
  cluster_lv cluster_vg -wi------- 1020.00m                                                             /dev/sda(0)         -1   auto    1     0 
  root       rhel       -wi-ao----  <12.50g                                                             /dev/vda2(384)   4.00m   auto    1     0 
  swap       rhel       -wi-ao----    1.50g                                                             /dev/vda2(0)     4.00m   auto    1     0 



 - The vgs command will need to have systemid added to its options in the -o argument and also have "--foreign" added:


BEFORE:

[root@rhel8-node-2 ~]# vgs -v -o+vg_mda_count,vg_mda_free,vg_mda_size,vg_mda_used_count,vg_tags --config 'global {locking_type = 0 metadata_read_only = 1}'
  Reloading config files
  WARNING: locking_type (0) is deprecated, using --nolocking.
  WARNING: File locking is disabled.
  Skipping foreign volume group cluster_vg
  VG   Attr   Ext   #PV #LV #SN VSize   VFree VG UUID                                VProfile #VMda VMdaFree  VMdaSize  #VMdaUse VG Tags
  rhel wz--n- 4.00m   1   2   0 <14.00g    0  kMAVWQ-FCIP-p0UB-QaEo-U38N-9aha-J714e2              1   507.50k  1020.00k        1        
  Reloading config files


AFTER:


[root@rhel8-node-2 ~]# vgs -v -o+vg_mda_count,vg_mda_free,vg_mda_size,vg_mda_used_count,vg_tags,systemid --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
  Reloading config files
  WARNING: locking_type (0) is deprecated, using --nolocking.
  WARNING: File locking is disabled.
  VG         Attr   Ext   #PV #LV #SN VSize   VFree VG UUID                                VProfile #VMda VMdaFree  VMdaSize  #VMdaUse VG Tags System ID         
  cluster_vg wz--n- 4.00m   1   1   0  <2.00g 1.00g 6tukPc-iGPG-1KLf-XheA-19b8-EDbz-Ie3HFt              1   508.00k  1020.00k        1         rhel8-node-1.clust
  rhel       wz--n- 4.00m   1   2   0 <14.00g    0  kMAVWQ-FCIP-p0UB-QaEo-U38N-9aha-J714e2              1   507.50k  1020.00k        1                           
  Reloading config files



6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.

Once changes are made, we can run an updated version of the sosreport on nodes of a cluster to verify that it captures the appropriate data. 

7. Is there already an existing RFE upstream or in Red Hat Bugzilla?

No

8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)?

RHEL 8

9. Is the sales team involved in this request and do they have any additional input?

No

10. List any affected packages or components.

sos

11. Would the customer be able to assist in testing this functionality if implemented?

N/A, Request is from support, and we wouild be able and willing to test this.

Comment 2 Pavel Moravec 2020-07-28 20:18:38 UTC
Few questions - generally "cant be there a situation where adding the param can harm"?

- on systems without LVM cluster, the option does not harm and is basically just ignored, right?

- should we add that parameter every time, or e.g. by default yes and have an option to fall over to current behaviour?

- is the option available for all versions of all the commands (in case the change backports automatically to e.g. CentOS, older Fedora or similar)?

- is that option available on all platforms (I guess so)? In all distros (at least Ubuntu/Debian)?


Thanks in advance for the answers.


(Technically, this is not RFE - no new feature is requested, just amending few commands collected by sosreport. So I updated the Subject and removed the keyword)

Comment 3 Reid Wahl 2020-07-30 06:15:01 UTC
(In reply to Pavel Moravec from comment #2)

Hi, Pavel. Chad asked me to have a look at this.

These are all valid questions. I think the simplest and safest approach would be to add a predicate to test whether the `--foreign` option is available, as we did in https://github.com/sosreport/sos/pull/2127. I don't have any other distros spun up besides Fedora and RHEL.

I believe that the `--foreign` option and the `systemid` field were added throughout LVM2 at the same time. This is in contrast to PR 2127, where we found that `--nolocking` was added to pvs/vgs/lvs before it was added to vgdisplay, etc. :/
  - Add lvmlockd (https://github.com/lvmteam/lvm2/commit/fe70b03d)
  - https://github.com/lvmteam/lvm2/blob/master/tools/lvmcmdline.c#L2471
  - https://github.com/lvmteam/lvm2/blob/master/lib/metadata/metadata.c#L3944

On RHEL, I think it's available starting with RHEL 7.2 (based on the upstream commit's backport tag and on the changelog here: https://access.redhat.com/downloads/content/lvm2/2.02.130-5.el7/x86_64/fd431d51/package-changelog).

---

> Few questions - generally "cant be there a situation where adding the param
> can harm"?
> 
> - on systems without LVM cluster, the option does not harm and is basically
> just ignored, right?

As far as I can tell, that's correct. I tested on my Fedora system, which does NOT have lvm2-lockd (or lvm2-cluster) installed. It worked fine. It didn't change the output or cause any errors.

---

> - should we add that parameter every time, or e.g. by default yes and have
> an option to fall over to current behaviour?

IMO there is no particular reason to create an option to disable the `--foreign` option. But we could do that.

Comment 4 Chad Newsom 2020-08-12 14:23:06 UTC
> on systems without LVM cluster, the option does not harm and is basically just ignored, right?

We may want a storage/LVM subject matter expert to weigh in here, but I do not believe that the additional options will cause any harm, especially since metadata_read_only is set, ensuring that no operations that change on-disk metadata are permitted. I wasn't entirely sure that the options would be recognized by older versions of RHEL, so I tried them out on the other VMs that I have access to right away:


RHEL 7 (with cluster):

[root@z1 ~]# lvs -a -o+lv_tags,devices,lv_kernel_read_ahead,lv_read_ahead,stripes,stripesize --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
  LV       VG           Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert LV Tags Devices        KRahead Rahead #Str Stripe
  ha_lv_01 ha_vg_rh7_01 -wi------- 1020.00m                                                             /dev/sda1(0)        -1   auto    1     0 
  root     rhel         -wi-ao----    9.79g                                                             /dev/vda2(307)   4.00m   auto    1     0 
  swap     rhel         -wi-ao----   <1.20g                                                             /dev/vda2(0)     4.00m   auto    1     0 
[root@z1 ~]# vgs -v -o+vg_mda_count,vg_mda_free,vg_mda_size,vg_mda_used_count,vg_tags,systemid --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
    Reloading config files
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
  VG           Attr   Ext   #PV #LV #SN VSize    VFree VG UUID                                VProfile #VMda VMdaFree  VMdaSize  #VMdaUse VG Tags   System ID
  ha_vg_rh7_01 wz--n- 4.00m   1   1   0 1020.00m    0  pr33Md-RLnU-CMDk-LdWW-mfgR-e0HE-BXzOX3              1   508.00k  1020.00k        1 pacemaker          
  rhel         wz--n- 4.00m   1   2   0  <11.00g 4.00m PnmR5Y-MV0I-Ficf-v0yr-qKhs-iLbg-SzDMP0              1   508.00k  1020.00k        1                    
    Reloading config files



RHEL 7 (no cluster add-on):

[root@testboxcnewsom ~]# lvs -a -o+lv_tags,devices,lv_kernel_read_ahead,lv_read_ahead,stripes,stripesize --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
  LV    VG    Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert LV Tags Devices                   KRahead Rahead #Str Stripe
  home  rhel  -wi-ao---- 174.00g                                                             /dev/sda2(2016)             4.00m   auto    1     0 
  root  rhel  -wi-ao----  50.00g                                                             /dev/sda2(46561)            4.00m   auto    1     0 
  swap  rhel  -wi-ao----  <7.88g                                                             /dev/sda2(0)                4.00m   auto    1     0 
  lv_01 vg_01 -wi-ao---- 149.05g                                                             /dev/sdc1(0),/dev/sdb1(0)   4.00m   auto    2  4.00k
[root@testboxcnewsom ~]# vgs -v -o+vg_mda_count,vg_mda_free,vg_mda_size,vg_mda_used_count,vg_tags,systemid --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
    Reloading config files
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
    Scanning all devices to initialize lvmetad.
  VG    Attr   Ext   #PV #LV #SN VSize    VFree  VG UUID                                VProfile #VMda VMdaFree  VMdaSize  #VMdaUse VG Tags System ID
  rhel  wz--n- 4.00m   1   3   0  231.88g  4.00m i5dDD0-s9iF-FOxd-llSh-d4CW-cjkc-QbZG2D              1        0   1020.00k        1                  
  vg_01 wz--n- 4.00m   2   1   0 <223.54g 74.48g 1t8Sfz-jkaE-Nhiy-P2nX-eqkK-CpXB-g3WH7f              2        0   1020.00k        2                  
    Reloading config files



[root@rhel6-rg-node1 ~]# lvs -a -o+lv_tags,devices,lv_kernel_read_ahead,lv_read_ahead,stripes,stripesize --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
  LV        VG         Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert LV Tags Devices         KRahead Rahead #Str Stripe
  lv_gfs2_1 vg_gfs2    -wi-ao----    1.00g                                                             /dev/sda1(0)    128.00k   auto    1     0 
  lv_gfs2_1 vg_gfs2    -wi-ao----    1.00g                                                             /dev/sdb1(0)    128.00k   auto    1     0 
  lv_gfs2_2 vg_gfs2    -wi-ao---- 1016.00m                                                             /dev/sdb1(1)    128.00k   auto    1     0 
  lv_root   vg_rhel6rg -wi-ao----    8.51g                                                             /dev/vda2(0)    128.00k   auto    1     0 
  lv_swap   vg_rhel6rg -wi-ao----    1.00g                                                             /dev/vda2(2178) 128.00k   auto    1     0 
[root@rhel6-rg-node1 ~]# vgs -v -o+vg_mda_count,vg_mda_free,vg_mda_size,vg_mda_used_count,vg_tags,systemid --config 'global {locking_type = 0 metadata_read_only = 1}' --foreign
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
    Using volume group(s) on command line.
  VG         Attr   Ext   #PV #LV #SN VSize VFree VG UUID                                VProfile #VMda VMdaFree  VMdaSize  #VMdaUse VG Tags System ID
  vg_gfs2    wz--nc 4.00m   2   2   0 1.99g    0  wioGIf-s4mt-RP0E-fmFN-X6C8-jwJA-s1DeBA              2    61.50k   128.00k        2                  
  vg_rhel6rg wz--n- 4.00m   1   2   0 9.51g    0  1yA8Ac-fzMb-obsr-SMBB-fRm6-a1yn-i5e4EM              1   508.00k  1020.00k        1                  
    Reloading config files
    Wiping internal VG cache


> should we add that parameter every time, or e.g. by default yes and have an option to fall over to current behaviour?

I believe we should, if systemid is not set, it will just be blank similar to the Tags option when a tag is not set. Additionally the "--foreign" option may prove to be helpful to sbr-storage as well (i.e. if a VG is no longer visible and the customer doesn't know why)


> is the option available for all versions of all the commands (in case the change backports automatically to e.g. CentOS, older Fedora or similar)?


I've checked with all of the currently supported major releases of RHEL, but cannot immediately verify on older versions of CentOS or Fedora


> is that option available on all platforms (I guess so)? In all distros (at least Ubuntu/Debian)?

Seems to be present in this debian man page:

https://manpages.debian.org/buster/lvm2/vgs.8.en.html

but not in this ubuntu man page from 16.04 LTS, but it seems to still work. I spun up a ubuntu 16.04 live image and "--foreign" seems to work for pvs, vgs, and lvs. and -o+systemid seems to work for vgs:

http://manpages.ubuntu.com/manpages/xenial/man8/vgs.8.html

Comment 5 Pavel Moravec 2020-08-17 07:55:22 UTC
Thanks for detailed feedback. I raised:

https://github.com/sosreport/sos/pull/2202

Please review it if it matches desired plans (I followed #c0 that changes collected commands from:

vgdisplay -vv --config="global{locking_type=0 metadata_read_only=1}"
vgscan -vvv --config="global{locking_type=0 metadata_read_only=1}"
pvscan -v --config="global{locking_type=0 metadata_read_only=1}"
pvs -a -v -o +pv_mda_free,pv_mda_size,pv_mda_count,pv_mda_used_count,pe_start --config="global{locking_type=0 metadata_read_only=1}"
vgs -v -o +vg_mda_count,vg_mda_free,vg_mda_size,vg_mda_used_count,vg_tags --config="global{locking_type=0 metadata_read_only=1}"
lvs -a -o +lv_tags,devices,lv_kernel_read_ahead,lv_read_ahead,stripes,stripesize --config="global{locking_type=0 metadata_read_only=1}"

to:

vgdisplay -vv --config="global{locking_type=0 metadata_read_only=1}" --foreign
vgscan -vvv --config="global{locking_type=0 metadata_read_only=1}"
pvscan -v --config="global{locking_type=0 metadata_read_only=1}"
pvs -a -v -o +pv_mda_free,pv_mda_size,pv_mda_count,pv_mda_used_count,pe_start --config="global{locking_type=0 metadata_read_only=1}" --foreign
vgs -v -o +vg_mda_count,vg_mda_free,vg_mda_size,vg_mda_used_count,vg_tags,systemid --config="global{locking_type=0 metadata_read_only=1}" --foreign
lvs -a -o +lv_tags,devices,lv_kernel_read_ahead,lv_read_ahead,stripes,stripesize --config="global{locking_type=0 metadata_read_only=1}" --foreign

I.e.:
- *scan commands are the same
- vgdisplay, pvs and lvs have --foreign
- vgs has --foreign and systemid

)

Comment 12 errata-xmlrpc 2021-05-18 14:47:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1604


Note You need to log in before you can comment on or make changes to this bug.