sosreport in RHE7 has never shipped vdsm plugin; it will do so since RHEL7.8 where upstream PR https://github.com/sosreport/sos/pull/1205 will supposedly land into. (and there is a request for RHEL8.1 to include it there, see BZ1700780).

I expect if one manually backports the PR to sos-3.7-2 available on brew, the lvm2 plugin data will be collected?

To test it:

yum update http://download-ipv4.eng.brq.redhat.com/brewroot/////packages/sos/3.7/2.el7/noarch/sos-3.7-2.el7.noarch.rpm -y

cd /tmp

wget https://patch-diff.githubusercontent.com/raw/sosreport/sos/pull/1205.patch

cd /usr/lib/python2.7/site-packages

cat /tmp/1205.patch | patch -p1


sosreport ..

Could you test that please?

Anyway, I am bit reluctant to add the plugin to 7.7 in this release cycle phase - is it sufficient to have it in 7.8?

Thanks Pavel, I got errors applying the patch.  

# yum update http://download-ipv4.eng.brq.redhat.com/brewroot/////packages/sos/3.7/2.el7/noarch/sos-3.7-2.el7.noarch.rpm -y
Loaded plugins: enabled_repos_upload, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade
sos-3.7-2.el7.noarch.rpm                                                                                                                                               | 498 kB  00:00:00     
Examining /var/tmp/yum-root-IuhBT6/sos-3.7-2.el7.noarch.rpm: sos-3.7-2.el7.noarch
Marking /var/tmp/yum-root-IuhBT6/sos-3.7-2.el7.noarch.rpm as an update to sos-3.6-17.el7_6.noarch
Resolving Dependencies
--> Running transaction check
---> Package sos.noarch 0:3.6-17.el7_6 will be updated
---> Package sos.noarch 0:3.7-2.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                Arch                                      Version                                      Repository                                                Size
==============================================================================================================================================================================================
Updating:
 sos                                    noarch                                    3.7-2.el7                                    /sos-3.7-2.el7.noarch                                    1.6 M

Transaction Summary
==============================================================================================================================================================================================
Upgrade  1 Package

Total size: 1.6 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : sos-3.7-2.el7.noarch                                                                                                                                                       1/2 
  Cleanup    : sos-3.6-17.el7_6.noarch                                                                                                                                                    2/2 
Uploading Package Profile
  Verifying  : sos-3.7-2.el7.noarch                                                                                                                                                       1/2 
  Verifying  : sos-3.6-17.el7_6.noarch                                                                                                                                                    2/2 

Updated:
  sos.noarch 0:3.7-2.el7                                                                                                                                                                      

Complete!
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager
Unable to upload Enabled Repositories Report
# cd /tmp
# wget https://patch-diff.githubusercontent.com/raw/sosreport/sos/pull/1205.patch
--2019-05-03 10:40:43--  https://patch-diff.githubusercontent.com/raw/sosreport/sos/pull/1205.patch
Resolving patch-diff.githubusercontent.com (patch-diff.githubusercontent.com)... 192.30.253.112
Connecting to patch-diff.githubusercontent.com (patch-diff.githubusercontent.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 200 OK
Cookie coming from patch-diff.githubusercontent.com attempted to set domain to patch-diff.githubusercontent.com
Length: unspecified [text/plain]
Saving to: ‘1205.patch’

    [ <=>                                                                                                                                                 ] 10,335      --.-K/s   in 0.01s   

2019-05-03 10:40:43 (706 KB/s) - ‘1205.patch’ saved [10335]

# cd /usr/lib/python2.7/site-packages
# cat /tmp/1205.patch | patch -p1
patching file sos/plugins/__init__.py
Hunk #1 succeeded at 1437 with fuzz 2 (offset 48 lines).
The next patch would create the file sos/plugins/vdsm.py,
which already exists!  Assume -R? [n] y
patching file sos/plugins/vdsm.py
Hunk #1 FAILED at 1.
File sos/plugins/vdsm.py is not empty after patch; not deleting
1 out of 1 hunk FAILED -- saving rejects to file sos/plugins/vdsm.py.rej
patching file sos/plugins/vdsm.py
Hunk #1 FAILED at 45.
1 out of 1 hunk FAILED -- saving rejects to file sos/plugins/vdsm.py.rej
patching file sos/plugins/vdsm.py
Hunk #1 succeeded at 38 (offset 16 lines).
Hunk #2 succeeded at 51 (offset 16 lines).
patching file sos/plugins/vdsm.py
Hunk #1 FAILED at 72.
1 out of 1 hunk FAILED -- saving rejects to file sos/plugins/vdsm.py.rej
patching file sos/plugins/vdsm.py
Hunk #1 FAILED at 43.
1 out of 1 hunk FAILED -- saving rejects to file sos/plugins/vdsm.py.rej
patching file sos/plugins/vdsm.py
Hunk #1 FAILED at 65.
1 out of 1 hunk FAILED -- saving rejects to file sos/plugins/vdsm.py.rej

Pavel, please try to get this in 7.7 if at all possible.  I know we're only just now opening the bz, but this has been a pain point for the storage group for some time.  Apologies for any inconvenience.  Any progress on the issue would be greatly appreciated.

> The next patch would create the file sos/plugins/vdsm.py,
> which already exists! 

That means /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py is installed from some other source / package. What does

rpm -qf /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py

say?

If sos would replace a file installed by some other package, it could be dangerous. No much idea how rpm will handle that, but you can get undefined behaviour depending on which of the two packages installing the same file would be installed first or second.

Could you ensure VDSM since RHEL7.7 (or any other time/release we agree on) does not install that file? (technically 3 files, vdsm.py, .pyo and .pyc)


Anyway to test the plugin / potential backport:

wget https://raw.githubusercontent.com/sosreport/sos/master/sos/plugins/vdsm.py -O /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py

sosreport ..



About backporting to 7.7: reviewing the plugin, it seems quite OK (though I think json.loads might raise an uncaught plugin exception for some unexpected/erroneous content parsed - sosreport will catch the exception, just the plugin wont work..). So devel_ack+ for 7.7.

To get other acks, namely QE one, please provide business justification and ensure OtherQE (as I doubt our QE will have capacity to verify).

Mirek, if OtherQE will be granted, are you OK to squeeze it into 7.7 ? "it" = new vdsm plugin: https://github.com/sosreport/sos/blob/master/sos/plugins/vdsm.py

(if so, I would do it in the re-spin with 1706059)

# rpm -qf /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py
vdsm-4.19.43-2.3.el7rhgs.x86_64

# pwd
/usr/lib/python2.7/site-packages/sos/plugins
# cp vdsm.py vdsm.py.BAK
# wget https://raw.githubusercontent.com/sosreport/sos/master/sos/plugins/vdsm.py -O /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py
# python -m py_compile vdsm.py

# sosreport -o vdsm,lvm2
[..]
caught exception in plugin method "vdsm.setup()"
writing traceback to sos_logs/vdsm-plugin-errors.txt

# cat sos_logs/vdsm-plugin-errors.txt 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1033, in setup
    plug.setup()
  File "/usr/lib/python2.7/site-packages/sos/plugins/vdsm.py", line 85, in setup
    qemu_pids = self.get_process_pids('qemu-kvm')
AttributeError: 'Vdsm' object has no attribute 'get_process_pids'


 84 
 85         qemu_pids = self.get_process_pids('qemu-kvm')
 86         if qemu_pids:
 87             files = ["cmdline", "status", "mountstats"]
 88             self.add_copy_spec([
 89                 "/proc/%s/%s" % (pid, name)
 90                 for pid in qemu_pids
 91                 for name in files
 92             ])

Looks like we also need this patch: 

https://github.com/sosreport/sos/commit/1b4f8dfb8ac85708441faa3b2c2b9c2624dfa155

After applying commit 1b4f8dfb8ac85708441faa3b2c2b9c2624dfa155 and https://raw.githubusercontent.com/sosreport/sos/master/sos/plugins/vdsm.py behavior appears to be what is expected output from lvm2 plugin and vdsm's lvm output.  

# python -m py_compile __init__.py
# sosreport -o vdsm,lvm2

# cat sos_commands/lvm2/pvs_-a_-v_-o_pv_mda_free_pv_mda_size_pv_mda_count_pv_mda_used_count_pe_start_--config_global_locking_type_0 |grep lvm2
  /dev/md0p1                                         gluster_vg lvm2 a--   <2.00t <1.90t  <2.00t UoR0jd-njZU-Fz32-fVGm-8K99-m4fD-obr4XO        0     <1.47m     1        1  <1.47m
  /dev/sda3                                          vg_root    lvm2 a--  237.22g     0  237.22g rBR76p-NCu2-NLos-M52G-Ko6N-zbcu-LOUGFL        0   1020.00k     1        1   1.00m

# cat sos_commands/lvm2/lvs_-a_-o_lv_tags_devices_--config_global_locking_type_0 
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
  LV                                 VG         Attr       LSize   Pool             Origin          Data%  Meta%  Move Log Cpy%Sync Convert LV Tags Devices                  
  7cd7b8de2fcf43d9bede700149e09ede_0 gluster_vg Vwi-aotz--  10.00g gluster_thinpool gluster_lv_data 2.88                                                                     
  gluster_lv_data                    gluster_vg Vwi-aotz--  10.00g gluster_thinpool                 2.88                                                                     
  gluster_lv_engine                  gluster_vg -wi-a-----   1.00g                                                                                  /dev/md0p1(26112)        
  gluster_thinpool                   gluster_vg twi-aotz-- 100.00g                                  0.33   0.03                                     gluster_thinpool_tdata(0)
  [gluster_thinpool_tdata]           gluster_vg Twi-ao---- 100.00g                                                                                  /dev/md0p1(256)          
  [gluster_thinpool_tmeta]           gluster_vg ewi-ao----   1.00g                                                                                  /dev/md0p1(25856)        
  [lvol0_pmspare]                    gluster_vg ewi-------   1.00g                                                                                  /dev/md0p1(0)            
  home                               vg_root    -wi-ao---- 108.22g                                                                                  /dev/sda3(33024)         
  root                               vg_root    -wi-ao---- 125.00g                                                                                  /dev/sda3(0)             
  swap                               vg_root    -wi-ao----   4.00g                                                                                  /dev/sda3(32000)         


# cat sos_commands/vdsm/lvm_pvs_-v_-o_all_--config_global_locking_type_0_metadata_read_only_1_use_lvmetad_0_devices_preferred_names_.dev.mapper._ignore_suspended_devices_1_write_cache_state_0_disable_after_error_count_3_filter_a_.dev.disk.by-id.dm-uuid-mpath-_r 
    Reloading config files
    Wiping internal VG cache
  WARNING: Not using lvmetad because config setting use_lvmetad=0.
  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).
  WARNING: Locking disabled. Be careful! This could corrupt your metadata.
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Reloading config files
    Wiping internal VG cache

I don't have any PV's matching the vdsm filter so no output.

Thanks for testing and good catch.

So backporting the vdsm.py AND commit https://github.com/sosreport/sos/commit/1b4f8df will resolve this BZ.

Could you please grant OtherQE / to verify the fix in 7.7, by adding OtherQA to Keywords?


Will vdsm package for RHEL7.7 purge away the plugin (worth filing a bug for it)?

(In reply to Pavel Moravec from comment #12)
> Thanks for testing and good catch.
> 
> So backporting the vdsm.py AND commit
> https://github.com/sosreport/sos/commit/1b4f8df will resolve this BZ.
> 
> Could you please grant OtherQE / to verify the fix in 7.7, by adding OtherQA
> to Keywords?
> 
> 
> Will vdsm package for RHEL7.7 purge away the plugin (worth filing a bug for
> it)?

Adding Martin, Dan and Douglas on this.

(In reply to Jon Magrini from comment #19)
> Both yum localinstall and rpm -Uvh return conflict with vdsm package: 
> 
> # yum localinstall sos-3.7-3.el7.noarch.rpm
> 
> --> Running transaction check
> ---> Package sos.noarch 0:3.6-17.el7_6 will be updated
> ---> Package sos.noarch 0:3.7-3.el7 will be an update
> 
> [..]
> 
> Transaction check error:
>   file /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py from install of
> sos-3.7-3.el7.noarch conflicts with file from package
> vdsm-4.19.43-2.3.el7rhgs.x86_64
>   file /usr/lib/python2.7/site-packages/sos/plugins/vdsm.pyc from install of
> sos-3.7-3.el7.noarch conflicts with file from package
> vdsm-4.19.43-2.3.el7rhgs.x86_64
>   file /usr/lib/python2.7/site-packages/sos/plugins/vdsm.pyo from install of
> sos-3.7-3.el7.noarch conflicts with file from package
> vdsm-4.19.43-2.3.el7rhgs.x86_64
> 
> 
> # rpm -Uvh sos-3.7-3.el7.noarch.rpm
> Preparing...                          #################################
> [100%]
> 	file /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py from install of
> sos-3.7-3.el7.noarch conflicts with file from package
> vdsm-4.19.43-2.3.el7rhgs.x86_64
> 	file /usr/lib/python2.7/site-packages/sos/plugins/vdsm.pyc from install of
> sos-3.7-3.el7.noarch conflicts with file from package
> vdsm-4.19.43-2.3.el7rhgs.x86_64
> 	file /usr/lib/python2.7/site-packages/sos/plugins/vdsm.pyo from install of
> sos-3.7-3.el7.noarch conflicts with file from package
> vdsm-4.19.43-2.3.el7rhgs.x86_64
> 
> 
> I didn't use `rpm -U --replacefiles` since that would be non-standard
> practice.

That is due to:

(In reply to Pavel Moravec from comment #12)
> Will vdsm package for RHEL7.7 purge away the plugin (worth filing a bug for
> it)?

That is something vdsm component/engineering must resolve by themselves (I can raise BZ if you wish, or you/reporter who knows the component better than me.

From https://bugzilla.redhat.com/show_bug.cgi?id=1700780#c5 , I should have added:

Conflicts: vdsm < 4.40

but I hadnt.

How the package dependency could be resolved in RHEL7.7 / VDSM 4.19 (or what version will be shipped since 7.7) - from vdsm side? sos package simply assumes anything under /usr/lib/python2.7/site-packages/sos belongs to sos..

(In reply to Pavel Moravec from comment #20)
> How the package dependency could be resolved in RHEL7.7 / VDSM 4.19 (or what
> version will be shipped since 7.7) - from vdsm side? sos package simply
> assumes anything under /usr/lib/python2.7/site-packages/sos belongs to sos..

A gentle reminder of the needinfo, esp. if some change from sos side is expected.

(In reply to Pavel Moravec from comment #21)
> (In reply to Pavel Moravec from comment #20)
> > How the package dependency could be resolved in RHEL7.7 / VDSM 4.19 (or what
> > version will be shipped since 7.7) - from vdsm side? sos package simply
> > assumes anything under /usr/lib/python2.7/site-packages/sos belongs to sos..
> 
> A gentle reminder of the needinfo, esp. if some change from sos side is
> expected.

Just started an email thread about the topic, it will be much easier to discuss it there and only afterwards post result here

Steve,
could you please grant exception+ here?

The VDSM plugin is already in sos-3.7-3 but we need to prevent package conflicts. Current vdsm packages install vdsm.py* into sos directory as "3rd party plugin". That is in conflict with new sos-3.7-3 that will ship the same files on its own.

So we need to add to sos.spec:

Conflicts: vdsm <= 4.30.17

to prevent this conflict. Any vdsm version higher than that will have sos plugin removed, and those vdsm versions will be distributed since RHEL7.7

The small change in spec file is all we require.

Good catch, thanks Miroslav!

John, here is a new version of the plugin / sosreport to test:


A yum repository for the build of sos-3.7-5.el7 (task 22351475) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/5.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/5.el7/sos-3.7-5.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/5.el7/noarch/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/5.el7/noarch/sos-3.7-5.el7.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.7/5.el7/noarch/sos-3.7-5.el7.noarch.rpm

Build output will be available for the next 21 days.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2295