Sosreport collects various stuff via plug-ins, where each plug-in:

1) is enabled by presence of some package / some of packages.

2) can collect files (config, logs,..)

3) can collect output of commands

4) can scrub confidential info (passwords in configs, or so)


OpenDaylight, a new plug-in should be written, that (please clarify on below details):

1) will be enabled (just) by presence of opendaylight package

2) will collect /opt/opendaylight/data/log/karaf.log file and content of /opt/opendaylight/etc/ directory

3) some cmd output to gather?

4) some password / private key / any other confidential info in collected data we shall obfuscate?

(In reply to Pavel Moravec from comment #2)
> Sosreport collects various stuff via plug-ins, where each plug-in:
> 
> 1) is enabled by presence of some package / some of packages.
> 
> 2) can collect files (config, logs,..)
> 
> 3) can collect output of commands
> 
> 4) can scrub confidential info (passwords in configs, or so)
> 
> 
> OpenDaylight, a new plug-in should be written, that (please clarify on below
> details):
> 
> 1) will be enabled (just) by presence of opendaylight package
> 
> 2) will collect /opt/opendaylight/data/log/karaf.log file and content of
> /opt/opendaylight/etc/ directory
> 
> 3) some cmd output to gather?
> 
> 4) some password / private key / any other confidential info in collected
> data we shall obfuscate?


1) I don't think inspecting if the package is installed is enough to tell if it is enabled.  Better to check systemd to see if 'opendaylight' is running, but then I guess that assumes it didn't crash either ;)  Could also check hieradata to see if the service (OpenDaylightApi) was enabled in deployment.

2) Please collect all logs from /opt/opendaylight/data/log/, and everything in /opt/opendaylight/etc/

3) We need outputs from every node related to openvswitch.  You could check if this hieradata OOO service was enabled on the node (OpenDaylightOvs).  These include:
- ovs-ofctl -O OpenFlow13 show br-int
- ovs-ofctl -O OpenFlow13 dump-flows br-int
- ovs-ofctl -O OpenFlow13 dump-groups br-int
- ovs-ofctl -O OpenFlow13 dump-group-stats br-int
- ovs-vsctl list Open_vSwitch
- ovs-vsctl show

4) I dont think there are any passwords or keys present in the logs or config directory.  The user passwords are stored in a db. Sam can you confirm?

(In reply to Tim Rozet from comment #4)
> (In reply to Pavel Moravec from comment #2)
> > Sosreport collects various stuff via plug-ins, where each plug-in:
> > 
> > 1) is enabled by presence of some package / some of packages.
> > 
> > 2) can collect files (config, logs,..)
> > 
> > 3) can collect output of commands
> > 
> > 4) can scrub confidential info (passwords in configs, or so)
> > 
> > 
> > OpenDaylight, a new plug-in should be written, that (please clarify on below
> > details):
> > 
> > 1) will be enabled (just) by presence of opendaylight package
> > 
> > 2) will collect /opt/opendaylight/data/log/karaf.log file and content of
> > /opt/opendaylight/etc/ directory
> > 
> > 3) some cmd output to gather?
> > 
> > 4) some password / private key / any other confidential info in collected
> > data we shall obfuscate?
> 
> 
> 1) I don't think inspecting if the package is installed is enough to tell if
> it is enabled.  Better to check systemd to see if 'opendaylight' is running,
> but then I guess that assumes it didn't crash either ;)  Could also check
> hieradata to see if the service (OpenDaylightApi) was enabled in deployment.
> 
> 2) Please collect all logs from /opt/opendaylight/data/log/, and everything
> in /opt/opendaylight/etc/
> 
> 3) We need outputs from every node related to openvswitch.  You could check
> if this hieradata OOO service was enabled on the node (OpenDaylightOvs). 
> These include:
> - ovs-ofctl -O OpenFlow13 show br-int
> - ovs-ofctl -O OpenFlow13 dump-flows br-int
> - ovs-ofctl -O OpenFlow13 dump-groups br-int
> - ovs-ofctl -O OpenFlow13 dump-group-stats br-int
> - ovs-vsctl list Open_vSwitch
> - ovs-vsctl show
> 
> 4) I dont think there are any passwords or keys present in the logs or
> config directory.  The user passwords are stored in a db. Sam can you
> confirm?

The credentials shouldn't be in any of the files. I am also double checking upstream.

> ODL RPM installs to /opt/opendaylight.

The /opt subtree is supposed to be for add-on components that are not part of the operating system (inherited in the FHS from SysV). There are various restrictions on the paths we can use there and parts of the tree are reserved for local administrator use (with a requirement that applications not rely on those parts of the tree to function correctly).

In particular, for sos, this means that we do not search any binary paths under /opt.

I hadn't noticed any other OpenStack components using this installation path so it seems to be a break with our own conventions.

> 3) We need outputs from every node related to openvswitch.  You could check if > this hieradata OOO service was enabled on the node (OpenDaylightOvs).  These include:
> - ovs-ofctl -O OpenFlow13 show br-int
> - ovs-ofctl -O OpenFlow13 dump-flows br-int
> - ovs-ofctl -O OpenFlow13 dump-groups br-int
> - ovs-ofctl -O OpenFlow13 dump-group-stats br-int
> - ovs-vsctl list Open_vSwitch
> - ovs-vsctl show

These are all collected by the existing openvswitch plugin; I'm assuming that ovs-vsctl (via the opwnvswitch package) must be installed to use OpenDaylight so there should be no action needed here. If not then we just need to add a new trigger to that plugin.

> 1) I don't think inspecting if the package is installed is enough to tell if it 
> is enabled.

Doesn't matter; sos does not care if something is enabled, or even configured, just that it is present. Data collection is opportunistic, so if something is not there, it is not an error.

We provide package triggers, path triggers, and (in the 3.5 release due in a month or so), executable command triggers based on search of the distribution's standard PATH.

One of these should be adequate for this use.

> > 4) I dont think there are any passwords or keys present in the logs or
> > config directory.  The user passwords are stored in a db. Sam can you
> > confirm?
> 
> The credentials shouldn't be in any of the files. I am also double checking
> upstream.

Upstream confirms that credentials are stored in idmlight.db.mv.db with an encrypted hash. We should be OK that they are not in the logs or config directory.

(In reply to Bryn M. Reeves from comment #7)
> > ODL RPM installs to /opt/opendaylight.
> 
> The /opt subtree is supposed to be for add-on components that are not part
> of the operating system (inherited in the FHS from SysV). There are various
> restrictions on the paths we can use there and parts of the tree are
> reserved for local administrator use (with a requirement that applications
> not rely on those parts of the tree to function correctly).
> 
> In particular, for sos, this means that we do not search any binary paths
> under /opt.
> 
> I hadn't noticed any other OpenStack components using this installation path
> so it seems to be a break with our own conventions.

I'm not an expert on this stuff so you may want to raise a separate BZ against ODL packaging if you feel /opt is the wrong place to put ODL.  From googling it looks like it is ok for binary to be under /opt:

http://www.pathname.com/fhs/pub/fhs-2.3.html#OPTADDONAPPLICATIONSOFTWAREPACKAGES

"Programs to be invoked by users must be located in the directory /opt/<package>/bin"

and it looks like /usr/local is admin only, and /opt is different:
https://unix.stackexchange.com/questions/11544/what-is-the-difference-between-opt-and-usr-local

Also take into consideration that ODL is not part of openstack, it is simply used with openstack for certain deployments.  It is java based, and does not install into python path like openstack packages do.

> From googling it looks like it is ok for binary to be under /opt:

That's not the issue (the whole point of /opt is to put _extra software_ that's not part of the distribution).

It's the fact that this is not in keeping with Red Hat's packaging for RHEL or with prior OSP experience - before we can make a change like this upstream we need to understand if it is the location it is conventionally installed to, or whether this is a quirk of Red Hat packaging that we need to work around some how (it's less problematic since the current request does not include executable commands to be run from the plugin but we still prefer to establish these things before we start making changes).

If this is entirely specific to our packaging then we may prefer to address this via a downstream patch until we have a better picture of how it may be packaged in future Red Hat releases and elsewhere.

> and it looks like /usr/local is admin only, and /opt is different:
> https://unix.stackexchange.com/questions/11544/what-is-the-difference-between-
> opt-and-usr-local

The _whole_ of /usr/local is reserved for the administrator's use. Packaged software must never place files there.

As I said: _parts_ of /opt are reserved - and this places restrictions on the applications that use it - the actual paths ODL is using aren't really a concern, but sos does not include any location in /opt in PATH which is a problem if there is ever a need to run any command from here.

These restrictions are discussed in the fist link you gave, under "Requirements" - the reason that we do not include '/opt/bin' in PATH is the requirement that no packaged software rely on the presence of these paths (since nothing can rely on being able to install files to that location we can't assume anything about its content).

HI Bryn

Would /usr/share be optimal location to install ODL? We can then put logs into /var/log/opendaylight and config files into /etc/opendaylight.

Some questions to decide be reporter:

1) ODL data collection shall be triggered by presence of opendaylight package? Could this package be installed standalone or just together with openvswitch package? (i.e. shall we have separate independent plug-in (my preferred option) or extend the openvswitch plug-in (worth doing only if there is some strong dependency/relationship among openvswitch and ODL))?

1.1) Just presence of the package will trigger the plugin / data collection - it as explained above by Bryn.

2) We shall collect /opt/opendaylight/data/log/karaf.log file and content of /opt/opendaylight/etc/ directory (or rather their equivalents after the movement from /opt)

2.1) optionally, sos shall (also? instead of logfile?) collect systemd service logs of opendaylight unit ("journalctl --no-pager --unit opendaylight" command output)?

3) ovs-* commands are collected by openvswitch plugin, enabled by presence of openvswitch package (that installs the ovs-* commands) - so no need to collect anything, sos_commands/openvswitch directory in sosreport shall contain that data already.


I suggest to wait for decision where the binaries/configs/logs will be placed to per packaging guidelines first, and then add the sos plugin. ODL on OSP 12 (deployed under /opt) wont be captured, ODSL since OSP 13 will be captured.

(alternativelly, we can collect both new locations and also the current under /opt, and drop support of /opt later on, but the 1st option is imho cleaner)


Let me know if I skip something wrt. technical requirements what to collect.

Some answers..

(In reply to Pavel Moravec from comment #16)
> Some questions to decide be reporter:
> 
> 1) ODL data collection shall be triggered by presence of opendaylight
> package? Could this package be installed standalone or just together with
> openvswitch package? (i.e. shall we have separate independent plug-in (my
> preferred option) or extend the openvswitch plug-in (worth doing only if
> there is some strong dependency/relationship among openvswitch and ODL))?

I think it's reasonable to collect based on the presence of the 'opendaylight' package.
It's not necessarily installed alongside openvswitch, so wrt your other questions it makes sense to have it's own SOS plugin.

> 
> 1.1) Just presence of the package will trigger the plugin / data collection
> - it as explained above by Bryn.
> 
> 2) We shall collect /opt/opendaylight/data/log/karaf.log file and content of
> /opt/opendaylight/etc/ directory (or rather their equivalents after the
> movement from /opt)

Sounds right.

> 
> 2.1) optionally, sos shall (also? instead of logfile?) collect systemd
> service logs of opendaylight unit ("journalctl --no-pager --unit
> opendaylight" command output)?

We have a plan to move to journal logging, but for the OSP 12 release it isn't going to happen. For now just the log file mentioned previously is used, and in the future a bug for SOS to collect from the journal should be opened once the logging is moved to the journal.

> 
> 3) ovs-* commands are collected by openvswitch plugin, enabled by presence
> of openvswitch package (that installs the ovs-* commands) - so no need to
> collect anything, sos_commands/openvswitch directory in sosreport shall
> contain that data already.
> 
> 
> I suggest to wait for decision where the binaries/configs/logs will be
> placed to per packaging guidelines first, and then add the sos plugin. ODL
> on OSP 12 (deployed under /opt) wont be captured, ODSL since OSP 13 will be
> captured.
> 
> (alternativelly, we can collect both new locations and also the current
> under /opt, and drop support of /opt later on, but the 1st option is imho
> cleaner)
> 

Is this request going to make it in time for OSP 12 at all?
If so then we need to collect the existing paths, and adjust them once they're changed.

> 
> Let me know if I skip something wrt. technical requirements what to collect.

Hi Pavel,

Will this plugin also add ODL in sosreport when run with TripleO?

There is also a need to gather information about SNAT (routers to NAPT switches).
What do you think?

Hi Pavel

We are in discussion to decide on moving ODL installation out of /opt. Meanwhile, lets implement this with following in mind:

1. Installation path - /opt/opendaylight/
2. Log files - /opt/opendaylight/data/log/
3. Config data - /opt/opendaylight/etc/
4. Presence of ODL package should suffice to trigger data collection
5. ovs related info from ovs commands
- ovs-ofctl -O OpenFlow13 show br-int
- ovs-ofctl -O OpenFlow13 dump-flows br-int
- ovs-ofctl -O OpenFlow13 dump-groups br-int
- ovs-ofctl -O OpenFlow13 dump-group-stats br-int
- ovs-vsctl list Open_vSwitch
- ovs-vsctl show

Let me know if you need more data.

(In reply to Janki from comment #25)
> Hi Pavel
> 
> We are in discussion to decide on moving ODL installation out of /opt.
> Meanwhile, lets implement this with following in mind:
> 

Thanks for clear specification.

> 1. Installation path - /opt/opendaylight/
> 2. Log files - /opt/opendaylight/data/log/

Will add (with default limit of 10MB of newest files and option to increase the limit).

> 3. Config data - /opt/opendaylight/etc/

Will add.

> 4. Presence of ODL package should suffice to trigger data collection

Great.

> 5. ovs related info from ovs commands
> - ovs-ofctl -O OpenFlow13 show br-int
> - ovs-ofctl -O OpenFlow13 dump-flows br-int
> - ovs-ofctl -O OpenFlow13 dump-groups br-int
> - ovs-ofctl -O OpenFlow13 dump-group-stats br-int
> - ovs-vsctl list Open_vSwitch
> - ovs-vsctl show

This shall be collected by openvswitch plug-in. These cmds are already collected - am I right?

ovs-ofctl -O OpenFlow13 dump-flows br-int
ovs-vsctl -t 5 show

(please run "sosreport -o openvswitch --batch --build", go to the directory specified in output, and search for files under sos_commands/openvswitch/ subdirectory; the above 2 shall be included already)

(In reply to Pavel Moravec from comment #26)
> (In reply to Janki from comment #25)
> > Hi Pavel
> > 
> > We are in discussion to decide on moving ODL installation out of /opt.
> > Meanwhile, lets implement this with following in mind:
> > 
> 
> Thanks for clear specification.
> 
> > 1. Installation path - /opt/opendaylight/
> > 2. Log files - /opt/opendaylight/data/log/
> 
> Will add (with default limit of 10MB of newest files and option to increase
> the limit).

ODL creates 2 log files with 10GB size each when used with TripleO. Will these default limit config be needed then?
> 
> > 3. Config data - /opt/opendaylight/etc/
> 
> Will add.
> 
> > 4. Presence of ODL package should suffice to trigger data collection
> 
> Great.
> 
> > 5. ovs related info from ovs commands
> > - ovs-ofctl -O OpenFlow13 show br-int
> > - ovs-ofctl -O OpenFlow13 dump-flows br-int
> > - ovs-ofctl -O OpenFlow13 dump-groups br-int
> > - ovs-ofctl -O OpenFlow13 dump-group-stats br-int
> > - ovs-vsctl list Open_vSwitch
> > - ovs-vsctl show
> 
> This shall be collected by openvswitch plug-in. These cmds are already
> collected - am I right?
> 
> ovs-ofctl -O OpenFlow13 dump-flows br-int
> ovs-vsctl -t 5 show
> 
> (please run "sosreport -o openvswitch --batch --build", go to the directory
> specified in output, and search for files under sos_commands/openvswitch/
> subdirectory; the above 2 shall be included already)

(In reply to Pavel Moravec from comment #26)
> (In reply to Janki from comment #25)
> > Hi Pavel
> > 
> > We are in discussion to decide on moving ODL installation out of /opt.
> > Meanwhile, lets implement this with following in mind:
> > 
> 
> Thanks for clear specification.
> 
> > 1. Installation path - /opt/opendaylight/
> > 2. Log files - /opt/opendaylight/data/log/
> 
> Will add (with default limit of 10MB of newest files and option to increase
> the limit).
> 
> > 3. Config data - /opt/opendaylight/etc/
> 
> Will add.
> 
> > 4. Presence of ODL package should suffice to trigger data collection
> 
> Great.
> 
> > 5. ovs related info from ovs commands
> > - ovs-ofctl -O OpenFlow13 show br-int
> > - ovs-ofctl -O OpenFlow13 dump-flows br-int
> > - ovs-ofctl -O OpenFlow13 dump-groups br-int
> > - ovs-ofctl -O OpenFlow13 dump-group-stats br-int
> > - ovs-vsctl list Open_vSwitch
> > - ovs-vsctl show
> 
> This shall be collected by openvswitch plug-in. These cmds are already
> collected - am I right?
> 
> ovs-ofctl -O OpenFlow13 dump-flows br-int
> ovs-vsctl -t 5 show
> 
> (please run "sosreport -o openvswitch --batch --build", go to the directory
> specified in output, and search for files under sos_commands/openvswitch/
> subdirectory; the above 2 shall be included already)

@Itzik, can you please verify above 2 commands.
@Pavel, these 2 commands are listed in plugin, so they should be good but still verifying with the steps you mentioned.

> > 
> > This shall be collected by openvswitch plug-in. These cmds are already
> > collected - am I right?
> > 
> > ovs-ofctl -O OpenFlow13 dump-flows br-int
> > ovs-vsctl -t 5 show
> > 
> > (please run "sosreport -o openvswitch --batch --build", go to the directory
> > specified in output, and search for files under sos_commands/openvswitch/
> > subdirectory; the above 2 shall be included already)
> 
> @Itzik, can you please verify above 2 commands.
> @Pavel, these 2 commands are listed in plugin, so they should be good but
> still verifying with the steps you mentioned.

These has been verified by Itzik. The ovs plugin needs to be updated to run these commands with -O OpenFlow13 flag.

All required work shall be captured in https://github.com/sosreport/sos/pull/1143

Note to myself: required to backport TWO commits:

https://github.com/sosreport/sos/commit/6013308e2ebfc45d72ef87c3d18fb2edce61d549

https://github.com/sosreport/sos/commit/e7552dd922d262ad13441f050b5223aea68d44ac

Hi

@Itzik, Run this on machine.

@Pavel, We need to include few containarised deployment related settings too like:

1. collect logs from /var/log/containers/opendaylight/*.log like [1]
2. collect config files from /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc like [2].

[1]. https://github.com/stuggi/sos/blob/master/sos/plugins/openstack_ceilometer.py#L38
[2]. https://github.com/stuggi/sos/blob/master/sos/plugins/openstack_ceilometer.py#L47

(In reply to Janki from comment #42)
> Hi
> 
> @Itzik, Run this on machine.
> 
> @Pavel, We need to include few containarised deployment related settings too
> like:
> 
> 1. collect logs from /var/log/containers/opendaylight/*.log like [1]
> 2. collect config files from
> /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc like
> [2].
> 
> [1].
> https://github.com/stuggi/sos/blob/master/sos/plugins/openstack_ceilometer.
> py#L38
> [2].
> https://github.com/stuggi/sos/blob/master/sos/plugins/openstack_ceilometer.
> py#L47

Why you come up with a requirement in the middle of QE phase? :(

I guess both 1 and 2 needs to be collected if and only if opendaylight package is present / within odl sosreport plug-in, am I right?

And you need to collect exactly these two patterns:

/var/log/containers/opendaylight/*.log (files)
/var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc (directory)

Can you pls. confirm?


(If there would be some other requirement to re-spin sos in 7.5, I can add it there, otherwise I dont plan so)

Hi Pavel

So sorry for these late requirements. We are in process of evolving ODL Log mechanism. These can change further in coming few days if/when ODL log mechanism changes.

Would you like these new requirements to be as a separate BZ? Is sos in freeze state for 7.5? If not, what's the deadline for new changes to go in? Can we backport it from "master" to "7.5"?

(In reply to Janki from comment #44)
> Hi Pavel
> 
> So sorry for these late requirements. We are in process of evolving ODL Log
> mechanism. These can change further in coming few days if/when ODL log
> mechanism changes.
> 
> Would you like these new requirements to be as a separate BZ? Is sos in
> freeze state for 7.5? If not, what's the deadline for new changes to go in?
> Can we backport it from "master" to "7.5"?

Pls. come with the requirements within few days so I can re-spin the package before Christmas. Then it should be no problem to add it to 7.5.

Hi Pavel

Please add below functionality to existing ODL Plugin

1. Collect logs from /var/log/containers/opendaylight/*.log (files) and ignore error if the files are not present.
2. Collect config info from /var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc (directory)
3. Collect output of "docker logs opendaylight_api"

Again apologies for last minute editions and thanks for accommodating them.

P.S. I am adding needinfo for you Pavel just so that you get notified so this comment.

(to be) added as PR https://github.com/sosreport/sos/pull/1171 .

Pavel, you also need to backport the last PR to rhel-7.4.z right?

(In reply to Janki from comment #53)
> Pavel, you also need to backport the last PR to rhel-7.4.z right?

I don't plan to backport either of this to RHEL 7.4, due to few reasons:

1) Please use proper flags next time, here "rhel-7.4.z?". Original request "can this be added to 7.4.z?" was responded by mine "first requirements, then implementation, then ACK to 7.5, then .." and since that time one forgets there is such a request..

2) RHEL 7.5 will be released relatively soon, esp. compared to next possible z-stream in 7.4. Is it really worth having it in 7.4 as well? Isn't 7.5 sufficient?

3) this is RFE and RFEs dont get into z-stream. Well, sosreport might be seen as an exception even in RHEL, esp. when just adding a new isolated plug-in. But still these requests are quite exceptionally accepted.

4) Continuation of previous point: currently there is no other z-stream request for sos. I dont think it is worth offering update of a RHEL package in such late z-stream update to _all_ RHEL7.4 customers just due to a new feature that will be utilized by a fraction of them. If there would be e.g. some another important bug to be fixed, _that_ would justify new errata that would be created. But an extra errata just due to a new feature in a late z-stream close to y-stream release?

Sorry, my bad. We need this for RHOSP13 which seems to be based on RHEL7.5. So we are good here.

It was discovered that ODL plugin must be automatically enabled also by puppet-opendaylight package. Thus trivial but critical change pending:

/usr/lib/python2.7/site-packages/sos/plugins/opendaylight.py 

     packages = ('opendaylight',)

change to:

     packages = ('opendaylight', 'puppet-opendaylight')

will fix it.

Janki,
could you please verify the BZ against sos-3.5-5 ?


A yum repository for the build of sos-3.5-5.el7 (task 15272478) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.5/5.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.5/5.el7/sos-3.5-5.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.5/5.el7/noarch/

Ran the following command:
sosreport -o openvswitch -o opendaylight --batch --build --all-logs

Version:
sos-3.5-5.el7.noarch
RHEL OSP12
RHEL7.4

Karaf log is present at:
/var/log/containers/opendaylight/karaf.log.1

ofctl outputs:
# ls -1  sos_commands/openvswitch/ovs-ofctl_-O_OpenFlow13*
sos_commands/openvswitch/ovs-ofctl_-O_OpenFlow13_dump-flows_br-int
sos_commands/openvswitch/ovs-ofctl_-O_OpenFlow13_dump-groups_br-int
sos_commands/openvswitch/ovs-ofctl_-O_OpenFlow13_dump-group-stats_br-int
sos_commands/openvswitch/ovs-ofctl_-O_OpenFlow13_dump-ports-desc_br-int
sos_commands/openvswitch/ovs-ofctl_-O_OpenFlow13_show_br-int

ovs-vsctl outputs:
# ls -1 sos_commands/openvswitch/ovs-vsctl_*
sos_commands/openvswitch/ovs-vsctl_list_bridge_br-ex
sos_commands/openvswitch/ovs-vsctl_list_bridge_br-int
sos_commands/openvswitch/ovs-vsctl_list_bridge_br-isolated
sos_commands/openvswitch/ovs-vsctl_list_Open_vSwitch
sos_commands/openvswitch/ovs-vsctl_-t_5_get_Open_vSwitch_._other_config
sos_commands/openvswitch/ovs-vsctl_-t_5_show

Configuration files:
# tree var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/
var/lib/config-data/puppet-generated/opendaylight/opt/opendaylight/etc/
├── jetty.xml
├── opendaylight
│   └── datastore
│       └── initial
│           └── config
│               └── netvirt-natservice-config.xml
├── org.apache.karaf.features.cfg
└── org.ops4j.pax.logging.cfg

docker logs opendaylight_api output:
sos_commands/opendaylight/docker_logs_opendaylight_api exists and empty (It's the same output as docker logs opendaylight_api)

Janki,
Can you please verify the above outputs are what is needed?

@Itzik, if these files are inside sosreport folder, then yes this is expected output.

Also check if all karaf log files are created. karaf.log.1 and karaf.log

There was only karaf.log.1 om the Hypervisor and in the sos folder.
It's seems OK from my side.
But I can't mark it as verified until it's in d/s repo.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0963