Description of problem: There's exception which is visible in sosreport -v: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1247, in setup plug.setup() File "/usr/lib/python2.7/site-packages/sos/plugins/vdsm.py", line 92, in setup self.addCopySpec("/var/log/vdsm/*", logsize) TypeError: add_copy_spec() takes exactly 2 arguments (3 given) # sed -n '92p' /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py self.addCopySpec("/var/log/vdsm/*", logsize) This causes failure of continuation of vdsm plugin run: # find /tmp/sosreport-dell-r210ii-04.rhev.lab.eng.brq.redhat.com-20170717093653/ -type f | xargs grep 'added cmd output' /tmp/sosreport-dell-r210ii-04.rhev.lab.eng.brq.redhat.com-20170717093653/sos_logs/sos.log:2017-07-17 09:36:53,370 INFO: [plugin:vdsm] added cmd output 'service vdsmd status' ^^ last cmd output # egrep -n "(addCopySpec.*logsize|collectExtOutput)" /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py 69: collectExtOutput = Plugin.add_cmd_output 87: self.collectExtOutput("service vdsmd status") ^^ last cmd definition 92: self.addCopySpec("/var/log/vdsm/*", logsize) ^^ failure 106: self.collectExtOutput("/bin/ls -l /var/log/core") 107: self.collectExtOutput("/bin/ls -ldZ /etc/vdsm") 108: self.collectExtOutput( 110: self.collectExtOutput( 112: self.collectExtOutput("/sbin/lvm vgs -v -o +tags") 113: self.collectExtOutput("/sbin/lvm lvs -v -o +tags") 114: self.collectExtOutput("/sbin/lvm pvs -v -o +all") 115: self.collectExtOutput("/sbin/fdisk -l") 116: self.collectExtOutput("/usr/bin/iostat") 117: self.collectExtOutput("/sbin/iscsiadm -m node") 118: self.collectExtOutput("/sbin/iscsiadm -m session") 119: self.collectExtOutput("/usr/sbin/nodectl info") 120: self.collectExtOutput("/usr/bin/abrt-cli list") 154: self.collectExtOutput("vdsm-tool dump-volume-chains %s" % sd_uuid) ^^ all these are not executed Discovered while being curious why no 'abrt-cli list' output exists from vdsm plugin: # abrt-cli list -n | grep ^reason reason: qemu-kvm killed by SIGSEGV Version-Release number of selected component (if applicable): vdsm-4.19.22-1.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. sosreport -o vdsm -v 2. 3. Actual results: exception Expected results: should pass without issues and all external cmds should have output (eg abrt-cli list) Additional info:
I suspect this relates to new version of sos package - can you specify what version of sos you are using?
(In reply to Yaniv Bronhaim from comment #2) > I suspect this relates to new version of sos package - can you specify what > version of sos you are using? 7.3 (version used): # rpm -q sos vdsm sos-3.3-5.el7_3.noarch vdsm-4.19.22-1.el7ev.x86_64 7.4 sos-3.4-5.el7.noarch vdsm-4.19.22-1.el7ev.x86_64 it seems to work fine on el 7.4: # tar OxJf /var/tmp/sosreport-localhost.localdomain-20170717114525.tar.xz sosreport-localhost.localdomain-20170717114525/sos_logs/sos.log | grep 'added cmd output' 2017-07-17 11:45:25,878 INFO: [plugin:vdsm] added cmd output 'service vdsmd status' 2017-07-17 11:45:25,898 INFO: [plugin:vdsm] added cmd output '/bin/ls -l /var/log/core' 2017-07-17 11:45:25,898 INFO: [plugin:vdsm] added cmd output '/bin/ls -ldZ /etc/vdsm' 2017-07-17 11:45:25,898 INFO: [plugin:vdsm] added cmd output '/bin/su vdsm -s /bin/sh -c '/usr/bin/tree -l /rhev/data-center'' 2017-07-17 11:45:25,898 INFO: [plugin:vdsm] added cmd output '/bin/su vdsm -s /bin/sh -c '/bin/ls -lR /rhev/data-center'' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/sbin/lvm vgs -v -o +tags' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/sbin/lvm lvs -v -o +tags' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/sbin/lvm pvs -v -o +all' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/sbin/fdisk -l' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/usr/bin/iostat' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/sbin/iscsiadm -m node' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/sbin/iscsiadm -m session' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/usr/sbin/nodectl info' 2017-07-17 11:45:25,899 INFO: [plugin:vdsm] added cmd output '/usr/bin/abrt-cli list' 2017-07-17 11:45:26,531 INFO: [plugin:vdsm] added cmd output 'vdsm-tool dump-volume-chains bb1b6832-614a-4edc-95fa-b95e7c01d052'
maybe you have some issues in your 7.3 environment? I have sos-3.3.-5 which works fine with current code - can you reproduce it in clean env?
(In reply to Yaniv Bronhaim from comment #4) > maybe you have some issues in your 7.3 environment? I have sos-3.3.-5 which > works fine with current code - can you reproduce it in clean env? fresh installed EL 7.3 still has same issue (vdsm from d/s from Jul 13): # rpm -q vdsm python sos redhat-release-server vdsm-4.19.22-1.el7ev.x86_64 python-2.7.5-48.el7.x86_64 sos-3.3-5.el7_3.noarch redhat-release-server-7.3-7.el7.x86_64 ... [sos.sosreport:setup] executing 'sosreport -o vdsm -v --debug' Setting up plugins ... [plugin:vdsm] added cmd output 'service vdsmd status' [plugin:vdsm] added copyspec '[]' [plugin:vdsm] added copyspec '[]' [plugin:vdsm] added copyspec '['/etc/vdsm/logrotate', '/etc/vdsm/logger.conf', '/etc/vdsm/mom.conf', '/etc/vdsm/mom.d', '/etc/vdsm/svdsm.logger.conf', '/etc/vdsm/vdsm.conf', '/etc/vdsm/vdsm.conf.d', '/etc/vdsm/vdsm.conf.20170717145821', '/etc/vdsm/vdsm.id']' Traceback (most recent call last): File "/usr/sbin/sosreport", line 25, in <module> main(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 1637, in main sos.execute() TypeError: add_copy_spec() takes exactly 2 arguments (3 given) > /usr/lib/python2.7/site-packages/sos/plugins/vdsm.py(92)setup() -> self.addCopySpec("/var/log/vdsm/*", logsize) (Pdb) ... If you need access to the machine, I'll keep it available.
Based on Comment 4 marking as blocker
Yes, This is a regression we introduced in 4.1 apparently. too bad it was not verified well as part of Bug #1460732. The fix (https://gerrit.ovirt.org/#/c/78285) didn't remove the log limit size, and once it is passed we fail. I'm not sure if the intention of the fix was to remove this limit for good or not. Irit, can you tell why you didn't replace the implementation with something that supports size limit?
Unfortunately every RHEL 7.x contains different SOS version and between those version backward compatibility is broken. So this fix needs to be verified with all RHEL versions which will be supported in 4.1.4
ok, vdsm-4.19.23-1.el7ev.x86_64 * el 7.4 sos-3.4-5.el7.noarch redhat-release-server-7.4-18.el7.x86_64 vdsm-4.19.23-1.el7ev.x86_64 # sosreport -o vdsm -v ... Running 1/1: vdsm... [plugin:vdsm] collecting path '/var/log/vdsm/supervdsm.log' [plugin:vdsm] collecting path '/var/log/vdsm/mom.log.1' [plugin:vdsm] collecting path '/etc/vdsm/mom.d' [plugin:vdsm] collecting path '/usr/libexec/vdsm/hooks' [plugin:vdsm] collecting path '/var/log/vdsm/import' [plugin:vdsm] collecting path '/etc/vdsm/logger.conf' [plugin:vdsm] collecting path '/var/log/vdsm/backup' [plugin:vdsm] collecting path '/etc/vdsm/logrotate' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.conf.20170717114324' [plugin:vdsm] collecting path '/var/run/vdsm/client.log' [plugin:vdsm] collecting path '/var/run/vdsm/trackedInterfaces' [plugin:vdsm] collecting path '/var/run/vdsm/netconf' [plugin:vdsm] collecting path '/var/log/vdsm/mom.log' [plugin:vdsm] collecting path '/etc/vdsm/mom.conf' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.conf' [plugin:vdsm] collecting path '/var/log/vdsm/upgrade.log' [plugin:vdsm] collecting path '/var/run/vdsm/lvm' [plugin:vdsm] collecting path '/var/lib/vdsm' [plugin:vdsm] collecting path '/var/log/sanlock.log' [plugin:vdsm] collecting path '/var/log/vdsm/vdsm.log' [plugin:vdsm] collecting path '/var/run/vdsm/sourceRoutes' [plugin:vdsm] collecting path '/etc/vdsm/svdsm.logger.conf' [plugin:vdsm] collecting path '/var/run/vdsm/nets_restored' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.conf.d' [plugin:vdsm] collecting path '/var/run/vdsm/v2v' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.id' [plugin:vdsm] collecting output of 'service vdsmd status' [plugin:vdsm] collecting output of '/bin/ls -l /var/log/core' [plugin:vdsm] collecting output of '/bin/ls -ldZ /etc/vdsm' [plugin:vdsm] collecting output of '/bin/su vdsm -s /bin/sh -c '/usr/bin/tree -l /rhev/data-center'' [plugin:vdsm] collecting output of '/bin/su vdsm -s /bin/sh -c '/bin/ls -lR /rhev/data-center'' [plugin:vdsm] collecting output of '/sbin/lvm vgs -v -o +tags' [plugin:vdsm] collecting output of '/sbin/lvm lvs -v -o +tags' [plugin:vdsm] collecting output of '/sbin/lvm pvs -v -o +all' [plugin:vdsm] collecting output of '/sbin/fdisk -l' [plugin:vdsm] collecting output of '/usr/bin/iostat' [plugin:vdsm] command '/usr/bin/iostat' not found in / - re-trying in host root [plugin:vdsm] collecting output of '/sbin/iscsiadm -m node' [plugin:vdsm] collecting output of '/sbin/iscsiadm -m session' [plugin:vdsm] collecting output of '/usr/sbin/nodectl info' [plugin:vdsm] command '/usr/sbin/nodectl' not found in / - re-trying in host root [plugin:vdsm] collecting output of '/usr/bin/abrt-cli list' [plugin:vdsm] collecting output of 'vdsm-tool dump-volume-chains bb1b6832-614a-4edc-95fa-b95e7c01d052' ... # tar tvJf /var/tmp/sosreport-localhost.localdomain-20170719110207.tar.xz | egrep '(/var/log/vdsm|sos_commands)' drwx------ root/root 0 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/ drwx------ root/root 0 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/ -rw-r--r-- root/root 2428 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/service_vdsmd_status -rw-r--r-- root/root 8 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/ls_-l_.var.log.core -rw-r--r-- root/root 65 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/ls_-ldZ_.etc.vdsm -rw-r--r-- root/root 2532 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.usr.bin.tree_-l_.rhev.data-center -rw-r--r-- root/root 3934 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.bin.ls_-lR_.rhev.data-center -rw-r--r-- root/root 28 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/lvm_vgs_-v_-o_tags -rw-r--r-- root/root 28 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/lvm_lvs_-v_-o_tags -rw-r--r-- root/root 69 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/lvm_pvs_-v_-o_all -rw-r--r-- root/root 1331 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/fdisk_-l -rw-r--r-- root/root 27 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/iscsiadm_-m_node -rw-r--r-- root/root 30 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/iscsiadm_-m_session -rw-r--r-- root/root 3444 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/abrt-cli_list -rw-r--r-- root/root 624 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/sos_commands/vdsm/vdsm-tool_dump-volume-chains_bb1b6832-614a-4edc-95fa-b95e7c01d052 drwxr-xr-x root/root 0 2017-07-18 11:37 sosreport-localhost.localdomain-20170719110207/var/log/vdsm/ -rw-r--r-- root/root 93416 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/var/log/vdsm/supervdsm.log -rw-r--r-- vdsm/kvm 2097142 2017-07-19 10:47 sosreport-localhost.localdomain-20170719110207/var/log/vdsm/mom.log.1 -rw-r--r-- vdsm/kvm 14090 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/var/log/vdsm/mom.log -rw-r--r-- root/root 3180 2017-07-17 11:44 sosreport-localhost.localdomain-20170719110207/var/log/vdsm/upgrade.log -rw-r--r-- vdsm/kvm 12098049 2017-07-19 11:02 sosreport-localhost.localdomain-20170719110207/var/log/vdsm/vdsm.log * el 7.3 redhat-release-server-7.3-7.el7.x86_64 sos-3.3-5.el7_3.noarch vdsm-4.19.23-1.el7ev.x86_64 # sosreport -o vdsm -v ... Running 1/1: vdsm... [60/1948] [plugin:vdsm] collecting path '/var/log/vdsm/supervdsm.log' [plugin:vdsm] collecting path '/var/log/vdsm/mom.log.1' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.conf.20170717145821' [plugin:vdsm] collecting path '/etc/vdsm/mom.d' [plugin:vdsm] collecting path '/usr/libexec/vdsm/hooks' [plugin:vdsm] collecting path '/var/log/vdsm/import' [plugin:vdsm] collecting path '/etc/vdsm/logger.conf' [plugin:vdsm] collecting path '/var/log/vdsm/backup' [plugin:vdsm] collecting path '/var/run/vdsm/trackedInterfaces' [plugin:vdsm] collecting path '/var/run/vdsm/client.log' [plugin:vdsm] collecting path '/etc/vdsm/logrotate' [plugin:vdsm] collecting path '/var/run/vdsm/netconf' [plugin:vdsm] collecting path '/var/log/vdsm/mom.log' [plugin:vdsm] collecting path '/etc/vdsm/mom.conf' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.conf' [plugin:vdsm] collecting path '/var/log/vdsm/upgrade.log' [plugin:vdsm] collecting path '/var/run/vdsm/lvm' [plugin:vdsm] collecting path '/var/log/vdsm/vdsm.log.1.xz' [plugin:vdsm] collecting path '/var/lib/vdsm' [plugin:vdsm] collecting path '/var/log/sanlock.log' [plugin:vdsm] collecting path '/var/log/vdsm/vdsm.log' [plugin:vdsm] collecting path '/var/run/vdsm/sourceRoutes' [plugin:vdsm] collecting path '/etc/vdsm/svdsm.logger.conf' [plugin:vdsm] collecting path '/var/run/vdsm/nets_restored' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.conf.d' [plugin:vdsm] collecting path '/var/run/vdsm/v2v' [plugin:vdsm] collecting path '/etc/vdsm/vdsm.id' [plugin:vdsm] collecting output of 'service vdsmd status' [plugin:vdsm] collecting output of '/bin/ls -l /var/log/core' [plugin:vdsm] collecting output of '/bin/ls -ldZ /etc/vdsm' [plugin:vdsm] collecting output of '/bin/su vdsm -s /bin/sh -c '/usr/bin/tree -l /rhev/data-center'' [plugin:vdsm] collecting output of '/bin/su vdsm -s /bin/sh -c '/bin/ls -lR /rhev/data-center'' [plugin:vdsm] collecting output of '/sbin/lvm vgs -v -o +tags' [plugin:vdsm] collecting output of '/sbin/lvm lvs -v -o +tags' [plugin:vdsm] collecting output of '/sbin/lvm pvs -v -o +all' [plugin:vdsm] collecting output of '/sbin/fdisk -l' [plugin:vdsm] collecting output of '/usr/bin/iostat' [plugin:vdsm] command '/usr/bin/iostat' not found in / - re-trying in host root [plugin:vdsm] collecting output of '/sbin/iscsiadm -m node' [plugin:vdsm] collecting output of '/sbin/iscsiadm -m session' [plugin:vdsm] collecting output of '/usr/sbin/nodectl info' [plugin:vdsm] command '/usr/sbin/nodectl' not found in / - re-trying in host root [plugin:vdsm] collecting output of '/usr/bin/abrt-cli list' [plugin:vdsm] collecting output of 'vdsm-tool dump-volume-chains bb1b6832-614a-4edc-95fa-b95e7c01d052' ... # tar tvJf /var/tmp/sosreport-localhost.localdomain-20170719110549.tar.xz | egrep '(/var/log/vdsm|sos_commands)' drwx------ root/root 0 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/ drwx------ root/root 0 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/ -rw-r--r-- root/root 2190 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/service_vdsmd_status -rw-r--r-- root/root 8 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/ls_-l_.var.log.core -rw-r--r-- root/root 65 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/ls_-ldZ_.etc.vdsm -rw-r--r-- root/root 2532 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.usr.bin.tree_-l_.rhev.data-center -rw-r--r-- root/root 3934 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/su_vdsm_-s_.bin.sh_-c_.bin.ls_-lR_.rhev.data-center -rw-r--r-- root/root 28 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/lvm_vgs_-v_-o_tags -rw-r--r-- root/root 28 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/lvm_lvs_-v_-o_tags -rw-r--r-- root/root 69 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/lvm_pvs_-v_-o_all -rw-r--r-- root/root 1331 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/fdisk_-l -rw-r--r-- root/root 27 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/iscsiadm_-m_node -rw-r--r-- root/root 30 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/iscsiadm_-m_session -rw-r--r-- root/root 3411 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/abrt-cli_list -rw-r--r-- root/root 624 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/sos_commands/vdsm/vdsm-tool_dump-volume-chains_bb1b6832-614a-4edc-95fa-b95e7c01d052 drwxr-xr-x root/root 0 2017-07-18 11:37 sosreport-localhost.localdomain-20170719110549/var/log/vdsm/ -rw-r--r-- root/root 164226 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/var/log/vdsm/supervdsm.log -rw-r--r-- vdsm/kvm 2097148 2017-07-19 04:29 sosreport-localhost.localdomain-20170719110549/var/log/vdsm/mom.log.1 -rw-r--r-- vdsm/kvm 376977 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/var/log/vdsm/mom.log -rw-r--r-- root/root 3180 2017-07-17 14:59 sosreport-localhost.localdomain-20170719110549/var/log/vdsm/upgrade.log -rw-r--r-- vdsm/kvm 1133756 2017-07-19 08:00 sosreport-localhost.localdomain-20170719110549/var/log/vdsm/vdsm.log.1.xz -rw-r--r-- vdsm/kvm 1291635 2017-07-19 11:05 sosreport-localhost.localdomain-20170719110549/var/log/vdsm/vdsm.log