Bug 1060383 - vdsm requires a compatible plugin with sos 3
Summary: vdsm requires a compatible plugin with sos 3
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: vdsm
Version: 3.4
Hardware: All
OS: Linux
high
high
Target Milestone: ---
: 3.4.1
Assignee: Douglas Schilling Landgraf
QA Contact: Jiri Belka
URL:
Whiteboard: infra
Depends On: 1088172
Blocks: 1060198
TreeView+ depends on / blocked
 
Reported: 2014-02-01 01:14 UTC by Douglas Schilling Landgraf
Modified: 2014-05-08 13:36 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-05-08 13:36:32 UTC
oVirt Team: ---


Attachments (Terms of Use)
logcollector output (6.67 KB, text/plain)
2014-04-15 15:14 UTC, Jiri Belka
no flags Details


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 25188 None MERGED sos: Support SOS version 3 Never
oVirt gerrit 25553 None MERGED sos: Support SOS version 3 Never

Description Douglas Schilling Landgraf 2014-02-01 01:14:18 UTC
Description of problem:

sos-3 changed API and we need to provide a new plugin for VDSM.

Comment 1 Dan Kenigsberg 2014-02-01 22:27:02 UTC
The fix must include a unit test for the plugin, so catch similar issues earlier.

Comment 2 Itamar Heim 2014-02-02 08:17:04 UTC
Setting target release to current version for consideration and review. please
do not push non-RFE bugs to an undefined target release to make sure bugs are
reviewed for relevancy, fix, closure, etc.

Comment 3 Sandro Bonazzola 2014-03-04 09:19:06 UTC
This is an automated message.
Re-targeting all non-blocker bugs still open on 3.4.0 to 3.4.1.

Comment 4 Jiri Belka 2014-04-15 08:12:13 UTC
I should test this with logcollector?

Comment 5 Douglas Schilling Landgraf 2014-04-15 14:03:10 UTC
(In reply to Jiri Belka from comment #4)
> I should test this with logcollector?

Yes, please.

Comment 6 Jiri Belka 2014-04-15 15:14:18 UTC
fail, vdsm3-4.14.6-0.el7ev.x86_64

# engine-log-collector --no-postgresql -H 10.34.63.222 -v
...snip...
INFO: collecting information from 10.34.63.222
DEBUG: calling(['/usr/bin/ssh', '-n', '-p', '22', '-i', '/etc/pki/ovirt-engine/keys/engine_id_rsa', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=600', 'root@10.34.63.222', "\nVERSION=`/bin/rpm -q --qf '[%{VERSION}]' sos | /bin/sed 's/\\.//'`;\nif [ $VERSION -ge 22 ]; then\n    /usr/sbin/sosreport  --batch -k general.all_logs=True         -o libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,selinux,kernel,memory,rpm\nelif [ $VERSION -ge 17 ]; then\n    /usr/sbin/sosreport  --no-progressbar -k general.all_logs=True         -o vdsm,general,networking,hardware,process,yum,filesys\nelse\n    /bin/echo No", 'valid', 'version', 'of', 'sosreport', 'found. 1>&2\n    exit 1\nfi\n'])
DEBUG: returncode(0)
DEBUG: STDOUT(
sosreport (version 3.0)

)
DEBUG: STDERR(Error reading response length from authentication socket.^M
no such option "all_logs" for plugin (general)
)
DEBUG: message(list index out of range)
DEBUG: parse_sosreport_stdout: Traceback (most recent call last):
  File "/usr/bin/engine-log-collector", line 530, in parse_sosreport_stdout
    if fileAry[0] is not None and len(fileAry) > 0:
IndexError: list index out of range

ERROR: Failed to collect logs from: 10.34.63.222; Could not parse sosreport output to determine filename
DEBUG: Traceback (most recent call last):
DEBUG:   File "/usr/bin/engine-log-collector", line 709, in run
DEBUG:     self.parse_sosreport_stdout(stdout)
DEBUG:   File "/usr/bin/engine-log-collector", line 566, in parse_sosreport_stdout
DEBUG:     "Could not parse sosreport output to determine filename"
DEBUG: Exception: Could not parse sosreport output to determine filename
...snip...
DEBUG: STDOUT(2c81e0afb74f4002eb4708a5c9c62e58  /tmp/sosreport-LogCollector-20140415170959.tar.xz
)
DEBUG: STDERR()
INFO: Log files have been collected and placed in /tmp/sosreport-LogCollector-20140415170959.tar.xz.
The MD5 for this file is 2c81e0afb74f4002eb4708a5c9c62e58 and its size is 43.5M
# echo $?
2

missing data from host:

# tar tJvf /tmp/sosreport-LogCollector-20140415170959.tar.xz | grep '10.34.63.222.*tar.xz'
#

Comment 7 Jiri Belka 2014-04-15 15:14:46 UTC
Created attachment 886516 [details]
logcollector output

Comment 8 Douglas Schilling Landgraf 2014-04-15 17:14:59 UTC
(In reply to Jiri Belka from comment #6)
> fail, vdsm3-4.14.6-0.el7ev.x86_64
> 
> # engine-log-collector --no-postgresql -H 10.34.63.222 -v
> ...snip...
> INFO: collecting information from 10.34.63.222
> DEBUG: calling(['/usr/bin/ssh', '-n', '-p', '22', '-i',
> '/etc/pki/ovirt-engine/keys/engine_id_rsa', '-oStrictHostKeyChecking=no',
> '-oServerAliveInterval=600', 'root@10.34.63.222', "\nVERSION=`/bin/rpm -q
> --qf '[%{VERSION}]' sos | /bin/sed 's/\\.//'`;\nif [ $VERSION -ge 22 ];
> then\n    /usr/sbin/sosreport  --batch -k general.all_logs=True         -o
> libvirt,vdsm,general,networking,hardware,process,yum,filesys,devicemapper,
> selinux,kernel,memory,rpm\nelif [ $VERSION -ge 17 ]; then\n   
> /usr/sbin/sosreport  --no-progressbar -k general.all_logs=True         -o
> vdsm,general,networking,hardware,process,yum,filesys\nelse\n    /bin/echo
> No", 'valid', 'version', 'of', 'sosreport', 'found. 1>&2\n    exit 1\nfi\n'])
> DEBUG: returncode(0)
> DEBUG: STDOUT(
> sosreport (version 3.0)
> 
> )
> DEBUG: STDERR(Error reading response length from authentication socket.^M
> no such option "all_logs" for plugin (general)
> )
> DEBUG: message(list index out of range)
> DEBUG: parse_sosreport_stdout: Traceback (most recent call last):
>   File "/usr/bin/engine-log-collector", line 530, in parse_sosreport_stdout
>     if fileAry[0] is not None and len(fileAry) > 0:
> IndexError: list index out of range
> 
> ERROR: Failed to collect logs from: 10.34.63.222; Could not parse sosreport
> output to determine filename
> DEBUG: Traceback (most recent call last):
> DEBUG:   File "/usr/bin/engine-log-collector", line 709, in run
> DEBUG:     self.parse_sosreport_stdout(stdout)
> DEBUG:   File "/usr/bin/engine-log-collector", line 566, in
> parse_sosreport_stdout
> DEBUG:     "Could not parse sosreport output to determine filename"
> DEBUG: Exception: Could not parse sosreport output to determine filename
> ...snip...
> DEBUG: STDOUT(2c81e0afb74f4002eb4708a5c9c62e58 
> /tmp/sosreport-LogCollector-20140415170959.tar.xz
> )
> DEBUG: STDERR()
> INFO: Log files have been collected and placed in
> /tmp/sosreport-LogCollector-20140415170959.tar.xz.
> The MD5 for this file is 2c81e0afb74f4002eb4708a5c9c62e58 and its size is
> 43.5M
> # echo $?
> 2
> 
> missing data from host:
> 
> # tar tJvf /tmp/sosreport-LogCollector-20140415170959.tar.xz | grep
> '10.34.63.222.*tar.xz'
> #

Seems an issue with engine-log-collector not with vdsm sos plugin.
Sandro, can you please provide your thoughts?

Thanks!

Comment 9 Bryn M. Reeves 2014-04-15 17:45:12 UTC
I started working on an oVirt engine plugin upstream a few weeks back, based on suggestions and patches from Sandro. You can find that work here:

  https://github.com/sosreport/sos/tree/bmr-add-ovirt-plugin

  https://github.com/sosreport/sos/blob/bmr-add-ovirt-plugin/sos/plugins/ovirt.py

Please review and comment on github - I'm just waiting for enough feedback before I merge this (and the dwh/reports bits).

Comment 10 Bryn M. Reeves 2014-04-15 17:47:09 UTC
In reply to comment #8 - it's not really clear what's happening here. Clearly engine-log-collector got something it didn't expect in the output from the remote sos command but without seeing what the output was it's hard to say what may have caused it.

Note that there's been some reorganisation of plugins in 3.0 - e.g. general.all_logs no longer exists as all syslog data is now captured in the 'logs' plugin (there is a corresponding logs.all_logs option now).

Comment 11 Douglas Schilling Landgraf 2014-04-15 17:54:02 UTC
Hello Jiri,

It seems a different bug not related to vdsm sos plugin. Please open a bug on engine-log-collector with the output that Bryn requested in comment #10, we can also add it into Depends On of this bug. I am moving back this bug to ON_QA.

Comment 12 Bryn M. Reeves 2014-04-15 18:34:10 UTC
OK looking at the output in the attachment:

https://bugzilla.redhat.com/attachment.cgi?id=886516

It seems to be the problem I suggested in comment #10 - the "all_logs" option has moved from the general plugin to the logs plugin:

DEBUG: returncode(0)
DEBUG: STDOUT(
sosreport (version 3.0)

)
DEBUG: STDERR(Error reading response length from authentication socket.

no such option "all_logs" for plugin (general)
)

It might be useful to teach the log collector some of the common errors sosreport can spit out so that it can react to this better.

Comment 13 Jiri Belka 2014-04-16 07:50:17 UTC
BZ1088172 create to address the issue reported in #6. Meanwhile making this BZ depending on BZ1088172.

Comment 14 Sandro Bonazzola 2014-04-16 14:30:07 UTC
(In reply to Jiri Belka from comment #13)
> BZ1088172 create to address the issue reported in #6. Meanwhile making this
> BZ depending on BZ1088172.

I'm sorry, I don't understand one thing.
This bug is targeted to oVirt 3.4.1 but log-collector is not expected to support sos 3.0 before oVirt 3.5.0 release.

ovirt-log-collector from master can't generate the issue described in comment #6.

This oVirt bug now depend on a rhev-m bug which has its own life cycle.
We also don't support el7 at all in oVirt 3.4.

Comment 15 Sandro Bonazzola 2014-05-08 13:36:32 UTC
This is an automated message

oVirt 3.4.1 has been released:
 * should fix your issue
 * should be available at your local mirror within two days.

If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.