Bug 1460734

Summary: traceback on hosts sos report failure
Product: [oVirt] ovirt-log-collector Reporter: Sandro Bonazzola <sbonazzo>
Component: CoreAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED CURRENTRELEASE QA Contact: David Necpal <dnecpal>
Severity: high Docs Contact:
Priority: high    
Version: ---CC: bugs, lsvaty
Target Milestone: ovirt-4.2.0Flags: rule-engine: ovirt-4.2+
rule-engine: planning_ack+
sbonazzo: devel_ack+
lsvaty: testing_ack+
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-20 10:51:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1456888    

Description Sandro Bonazzola 2017-06-12 14:36:45 UTC
Due to bug #1460732 sos 3.4 is failing while executed on hosts.
On ovirt-log-collector side, the failure cause a traceback:

About to collect information from 2 hypervisors. Continue? (Y/n): 
INFO: Gathering information from selected hypervisors...
INFO: collecting information from lago-rhv-suite-4-1-host1
DEBUG: calling(['/usr/bin/ssh', '-n', '-p', '22', '-i', '/etc/pki/ovirt-engine/keys/engine_id_rsa', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=600', 'root@lago-rhv-suite-4-1-host1', "\nVERSION=`/bin/rpm -q --qf '[%{VERSION}]' sos | /bin/sed 's/\\.//'`;\nif [ $VERSION -ge 32 ]; then\n    /usr/sbin/sosreport  --batch --all-logs         -o logs,libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm,processor,pci,md,block,scsi,multipath,systemd,sanlock,lvm2,chrony,systemd,ovirt_hosted_engine\nelif [ $VERSION -ge 30 ]; then\n    /usr/sbin/sosreport  --batch -k logs.all_logs=True         -o logs,libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm,processor,pci,md,block,scsi,multipath,systemd,sanlock,lvm2,chrony,systemd\nelif [ $VERSION -ge 22 ]; then\n    /usr/sbin/sosreport  --batch -k general.all_logs=True         -o libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm\nelif [ $VERSION -ge 17 ]; then\n    /usr/sbin/sosreport  --no-progressbar -k general.all_logs=True         -o vdsm,general,networking,hardware,process,yum,filesys\nelse\n    /bin/echo No", 'valid', 'version', 'of', 'sosreport', 'found. 1>&2\n    exit 1\nfi\n'])
INFO: collecting information from lago-rhv-suite-4-1-host0
DEBUG: calling(['/usr/bin/ssh', '-n', '-p', '22', '-i', '/etc/pki/ovirt-engine/keys/engine_id_rsa', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=600', 'root@lago-rhv-suite-4-1-host0', "\nVERSION=`/bin/rpm -q --qf '[%{VERSION}]' sos | /bin/sed 's/\\.//'`;\nif [ $VERSION -ge 32 ]; then\n    /usr/sbin/sosreport  --batch --all-logs         -o logs,libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm,processor,pci,md,block,scsi,multipath,systemd,sanlock,lvm2,chrony,systemd,ovirt_hosted_engine\nelif [ $VERSION -ge 30 ]; then\n    /usr/sbin/sosreport  --batch -k logs.all_logs=True         -o logs,libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm,processor,pci,md,block,scsi,multipath,systemd,sanlock,lvm2,chrony,systemd\nelif [ $VERSION -ge 22 ]; then\n    /usr/sbin/sosreport  --batch -k general.all_logs=True         -o libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm\nelif [ $VERSION -ge 17 ]; then\n    /usr/sbin/sosreport  --no-progressbar -k general.all_logs=True         -o vdsm,general,networking,hardware,process,yum,filesys\nelse\n    /bin/echo No", 'valid', 'version', 'of', 'sosreport', 'found. 1>&2\n    exit 1\nfi\n'])
DEBUG: returncode(0)
DEBUG: STDOUT(
sosreport (version 3.4)

)
DEBUG: STDERR(Warning: Permanently added 'lago-rhv-suite-4-1-host0,192.168.201.3' (ECDSA) to the list of known hosts.
plugin vdsm does not install, skipping: type object 'Plugin' has no attribute 'add_copy_spec_limit'
a non-existing plugin (vdsm) was specified in the command line
)
DEBUG: message(list index out of range)
DEBUG: parse_sosreport_stdout: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_log_collector/__main__.py", line 554, in parse_sosreport_stdout
    if fileAry[0] is not None and len(fileAry) > 0:
IndexError: list index out of range

DEBUG: Traceback (most recent call last):
DEBUG:   File "/usr/lib/python2.7/site-packages/ovirt_log_collector/__main__.py", line 755, in run
DEBUG:     self.parse_sosreport_stdout(stdout)
DEBUG:   File "/usr/lib/python2.7/site-packages/ovirt_log_collector/__main__.py", line 590, in parse_sosreport_stdout
DEBUG:     "Could not parse sosreport output to determine filename"
DEBUG: Exception: Could not parse sosreport output to determine filename
DEBUG: Configuration for lago-rhv-suite-4-1-host0:
DEBUG: returncode(0)
DEBUG: STDOUT(
sosreport (version 3.4)

)
DEBUG: STDERR(Warning: Permanently added 'lago-rhv-suite-4-1-host1,192.168.201.2' (ECDSA) to the list of known hosts.
plugin vdsm does not install, skipping: type object 'Plugin' has no attribute 'add_copy_spec_limit'
a non-existing plugin (vdsm) was specified in the command line
)
DEBUG: {'bc_reports': 'vdsm,general,networking,hardware,process,yum,filesys',
DEBUG: message(list index out of range)
DEBUG:  'cert_file': '/etc/pki/ovirt-engine/ca.pem',
DEBUG: parse_sosreport_stdout: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_log_collector/__main__.py", line 554, in parse_sosreport_stdout
    if fileAry[0] is not None and len(fileAry) > 0:
IndexError: list index out of range

DEBUG:  'conf_file': '/root/ovirt-log-collector.conf',
DEBUG:  'engine': 'engine:443',
DEBUG: Traceback (most recent call last):
DEBUG:  'hostname': 'lago-rhv-suite-4-1-host0',
DEBUG:   File "/usr/lib/python2.7/site-packages/ovirt_log_collector/__main__.py", line 755, in run
DEBUG:     self.parse_sosreport_stdout(stdout)
DEBUG:  'hosts': set([('test-dc',
DEBUG:   File "/usr/lib/python2.7/site-packages/ovirt_log_collector/__main__.py", line 590, in parse_sosreport_stdout
DEBUG:                 <ovirt_log_collector.helper.hypervisors.Cluster object at 0x26bd290>,
DEBUG:     "Could not parse sosreport output to determine filename"
DEBUG:                 'lago-rhv-suite-4-1-host0'),
DEBUG: Exception: Could not parse sosreport output to determine filename
DEBUG:                ('test-dc',
DEBUG: Configuration for lago-rhv-suite-4-1-host1:
DEBUG:                 <ovirt_log_collector.helper.hypervisors.Cluster object at 0x26bd290>,
DEBUG:                 'lago-rhv-suite-4-1-host1')]),
DEBUG: {'bc_reports': 'vdsm,general,networking,hardware,process,yum,filesys',
DEBUG:  'include_sensitive_data': False,
DEBUG:  'cert_file': '/etc/pki/ovirt-engine/ca.pem',
DEBUG:  'insecure': False,
DEBUG:  'conf_file': '/root/ovirt-log-collector.conf',
DEBUG:  'kerberos': False,
DEBUG:  'engine': 'engine:443',
DEBUG:  'key_file': '/etc/pki/ovirt-engine/keys/engine_id_rsa',
DEBUG:  'hostname': 'lago-rhv-suite-4-1-host1',
DEBUG:  'local_scratch_dir': '/dev/shm/log/working/log-collector-data',
DEBUG:  'hosts': set([('test-dc',
DEBUG:  'local_tmp_dir': '/dev/shm/log',
DEBUG:                 <ovirt_log_collector.helper.hypervisors.Cluster object at 0x26bd290>,
DEBUG:  'local_working_dir': '/dev/shm/log/working',
DEBUG:                 'lago-rhv-suite-4-1-host0'),
DEBUG:  'log_file': '/var/log/ovirt-engine/ovirt-log-collector/ovirt-log-collector-20170612082725.log',
DEBUG:                ('test-dc',
DEBUG:  'max_connections': 10,
DEBUG:                 <ovirt_log_collector.helper.hypervisors.Cluster object at 0x26bd290>,
DEBUG:  'no_hypervisor': False,
DEBUG:                 'lago-rhv-suite-4-1-host1')]),
DEBUG:  'no_postgresql': False,
DEBUG:  'output': '/dev/shm',
DEBUG:  'passwd': '123',
DEBUG:  'include_sensitive_data': False,
DEBUG:  'pg_dbhost': 'localhost',
DEBUG:  'insecure': False,
DEBUG:  'pg_dbname': 'engine',
DEBUG:  'kerberos': False,
DEBUG:  'pg_dbport': '5432',
DEBUG:  'key_file': '/etc/pki/ovirt-engine/keys/engine_id_rsa',
DEBUG:  'pg_pass': 'MiQPXNuVwbkiOaPG1zXeNH',
DEBUG:  'local_scratch_dir': '/dev/shm/log/working/log-collector-data',
DEBUG:  'pg_ssh_user': 'root',
DEBUG:  'local_tmp_dir': '/dev/shm/log',
DEBUG:  'pg_user': 'engine',
DEBUG:  'local_working_dir': '/dev/shm/log/working',
DEBUG:  'quiet': False,
DEBUG:  'log_file': '/var/log/ovirt-engine/ovirt-log-collector/ovirt-log-collector-20170612082725.log',
DEBUG:  'reports': 'libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm',
DEBUG:  'reports3': 'processor,pci,md,block,scsi,multipath,systemd,sanlock,lvm2,chrony,systemd',
DEBUG:  'scp_cmd': '/usr/bin/scp -P 22 -i /etc/pki/ovirt-engine/keys/engine_id_rsa -oStrictHostKeyChecking=no -oServerAliveInterval=600 root@lago-rhv-suite-4-1-host0',
DEBUG:  'ssh_cmd': '/usr/bin/ssh -n -p 22 -i /etc/pki/ovirt-engine/keys/engine_id_rsa -oStrictHostKeyChecking=no -oServerAliveInterval=600 root@lago-rhv-suite-4-1-host0',
DEBUG:  'ssh_port': 22,
DEBUG:  'max_connections': 10,
DEBUG:  'time_only': False,
DEBUG:  'no_hypervisor': False,
DEBUG:  'user': 'admin@internal',
DEBUG:  'no_postgresql': False,
DEBUG:  'verbose': True}
DEBUG:  'output': '/dev/shm',
INFO: finished collecting information from lago-rhv-suite-4-1-host0
DEBUG:  'passwd': '123',
DEBUG:  'pg_dbhost': 'localhost',
DEBUG:  'pg_dbname': 'engine',
DEBUG:  'pg_dbport': '5432',
DEBUG:  'pg_pass': 'MiQPXNuVwbkiOaPG1zXeNH',
DEBUG:  'pg_ssh_user': 'root',
DEBUG:  'pg_user': 'engine',
DEBUG:  'quiet': False,
DEBUG:  'reports': 'libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm',
DEBUG:  'reports3': 'processor,pci,md,block,scsi,multipath,systemd,sanlock,lvm2,chrony,systemd',
DEBUG:  'scp_cmd': '/usr/bin/scp -P 22 -i /etc/pki/ovirt-engine/keys/engine_id_rsa -oStrictHostKeyChecking=no -oServerAliveInterval=600 root@lago-rhv-suite-4-1-host1',
DEBUG:  'ssh_cmd': '/usr/bin/ssh -n -p 22 -i /etc/pki/ovirt-engine/keys/engine_id_rsa -oStrictHostKeyChecking=no -oServerAliveInterval=600 root@lago-rhv-suite-4-1-host1',
DEBUG:  'ssh_port': 22,
DEBUG:  'time_only': False,
DEBUG:  'user': 'admin@internal',
DEBUG:  'verbose': True}
INFO: finished collecting information from lago-rhv-suite-4-1-host1

Comment 1 David Necpal 2017-09-04 16:48:05 UTC
Verified on version: 
ovirt-log-collector-4.2.0-0.0.master.20170903141131.gitbd2607f.el7.centos.noarch

Comment 2 Sandro Bonazzola 2017-12-20 10:51:39 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.