Bug 1560849

Please note:
On tripleo OVN container deployments (OSP13), the host path for log file location is "/var/log/openvswitch/ovn" and for the ovn db files is "/var/lib/openvswitch/ovn".

So we may want to collect the files under these directories when sosreport is run for openvswitch.

Thanks for the report, few points to clarify:

(*) all the mentioned commands and logs should be automatically collected when openvswitch or openvswitch-dpdk package is installed, right?

(*) for `ovn-nbctl list`, how to get the list of tables in the NB database? (i.e. could you provide pseudo-code for this, like:

for table in $(ovn-nbctl show | head -n1); do
  ovn-nbctl list $table
done

(*) the same for `ovn-sbctl list` please - how to get the list of tables?

(*) what did you mean by:
/sosreport-controller-0-20180326122314/var/lib/config-data/
/sosreport-compute-1-20180326122358/sos_commands/

?


(*) "/var/log/openvswitch/ovn" - this is collected when --all-logs is provided; shall we collect content of this dir every time? Or just *.log files from it?

(*) "/var/lib/openvswitch/ovn" - whole dir with some DB files there? How big the DB could be? Can't it have some credentials that would be insecure to share inside a sosreport?

raising needinfo for questions from previous comment

see inline

(In reply to Pavel Moravec from comment #5)
> Thanks for the report, few points to clarify:
> 
> (*) all the mentioned commands and logs should be automatically collected
> when openvswitch or openvswitch-dpdk package is installed, right?
right

> (*) for `ovn-nbctl list`, how to get the list of tables in the NB database?
> (i.e. could you provide pseudo-code for this, like:
> 
> for table in $(ovn-nbctl show | head -n1); do
>   ovn-nbctl list $table
> done
> 
> (*) the same for `ovn-sbctl list` please - how to get the list of tables?

I think it's better to ask OVN DEV. As far as I know, there are more tables that we need to collect.

> (*) what did you mean by:
> /sosreport-controller-0-20180326122314/var/lib/config-data/
> /sosreport-compute-1-20180326122358/sos_commands/
this is the path where the output of the configuration & commands so I think we need to collect this folders. 
> 
> 
> (*) "/var/log/openvswitch/ovn" - this is collected when --all-logs is
> provided; shall we collect content of this dir every time? Or just *.log
> files from it?
yes  we need to collect content of this dir every time

> (*) "/var/lib/openvswitch/ovn" - whole dir with some DB files there? How big
> the DB could be? Can't it have some credentials that would be insecure to
> share inside a sosreport? need to verify with DeV

(In reply to Eran Kuris from comment #7)
> see inline
> 
> (In reply to Pavel Moravec from comment #5)
> > Thanks for the report, few points to clarify:
> > 
> > (*) all the mentioned commands and logs should be automatically collected
> > when openvswitch or openvswitch-dpdk package is installed, right?
> right
> 
> > (*) for `ovn-nbctl list`, how to get the list of tables in the NB database?
> > (i.e. could you provide pseudo-code for this, like:
> > 
> > for table in $(ovn-nbctl show | head -n1); do
> >   ovn-nbctl list $table
> > done
> > 
> > (*) the same for `ovn-sbctl list` please - how to get the list of tables?
> 
> I think it's better to ask OVN DEV. As far as I know, there are more tables
> that we need to collect.

Could you pls. raise needinfo on some OVN devel? I dont know any.

> 
> > (*) what did you mean by:
> > /sosreport-controller-0-20180326122314/var/lib/config-data/
> > /sosreport-compute-1-20180326122358/sos_commands/
> this is the path where the output of the configuration & commands so I think
> we need to collect this folders. 

I dont understand - how these directories are created? (and really with path /sosreport-*, not e.g. /var/tmp/sosreport-* ?) By running some another sosreport (that we should take its output?) Why dont call just one sosreport, then?

Or do you mean the commands output should be stored under sos_commands and configs collected under /var/lib/config-data ? The first is correct all the time, the second almost as well - whatever config or logfile is collected, it is put to the same subtree under the sosreport directory like its origin. I.e. /var/log/openvswitch/ovn will be copied to sosreport-*/var/log/openvswitch/ovn dir.

> > 
> > 
> > (*) "/var/log/openvswitch/ovn" - this is collected when --all-logs is
> > provided; shall we collect content of this dir every time? Or just *.log
> > files from it?
> yes  we need to collect content of this dir every time
Whole dir or "just" *.log files there (that is some common approach to collect only *.log files, until --all-logs option is provided)?

> 
> > (*) "/var/lib/openvswitch/ovn" - whole dir with some DB files there? How big
> > the DB could be? Can't it have some credentials that would be insecure to
> > share inside a sosreport? 
> need to verify with DeV

Again, please provide a devel name or raise needinfo to them directly.

(In reply to Pavel Moravec from comment #8)
> (In reply to Eran Kuris from comment #7)
> > see inline
> > 
> > (In reply to Pavel Moravec from comment #5)
> > > Thanks for the report, few points to clarify:
> > > 
> > > (*) all the mentioned commands and logs should be automatically collected
> > > when openvswitch or openvswitch-dpdk package is installed, right?
> > right
> > 
> > > (*) for `ovn-nbctl list`, how to get the list of tables in the NB database?
> > > (i.e. could you provide pseudo-code for this, like:
> > > 
> > > for table in $(ovn-nbctl show | head -n1); do
> > >   ovn-nbctl list $table
> > > done
> > > 
> > > (*) the same for `ovn-sbctl list` please - how to get the list of tables?
> > 
> > I think it's better to ask OVN DEV. As far as I know, there are more tables
> > that we need to collect.
> 
> Could you pls. raise needinfo on some OVN devel? I dont know any.
> 
> > 
> > > (*) what did you mean by:
> > > /sosreport-controller-0-20180326122314/var/lib/config-data/
> > > /sosreport-compute-1-20180326122358/sos_commands/
> > this is the path where the output of the configuration & commands so I think
> > we need to collect this folders. 
> 
> I dont understand - how these directories are created? (and really with path
> /sosreport-*, not e.g. /var/tmp/sosreport-* ?) By running some another
> sosreport (that we should take its output?) Why dont call just one
> sosreport, then?
> 
> Or do you mean the commands output should be stored under sos_commands and
> configs collected under /var/lib/config-data ? The first is correct all the
> time, the second almost as well - whatever config or logfile is collected,
> it is put to the same subtree under the sosreport directory like its origin.
> I.e. /var/log/openvswitch/ovn will be copied to
> sosreport-*/var/log/openvswitch/ovn dir
yes thats what  I meant.
> 
> > > 
> > > 
> > > (*) "/var/log/openvswitch/ovn" - this is collected when --all-logs is
> > > provided; shall we collect content of this dir every time? Or just *.log
> > > files from it?
> > yes  we need to collect content of this dir every time
> Whole dir or "just" *.log files there (that is some common approach to
> collect only *.log files, until --all-logs option is provided)?
"just" *.log files
> > 
> > > (*) "/var/lib/openvswitch/ovn" - whole dir with some DB files there? How big
> > > the DB could be? Can't it have some credentials that would be insecure to
> > > share inside a sosreport? 
> > need to verify with DeV
> 
> Again, please provide a devel name or raise needinfo to them directly.

@Numan can you help with that ?

(In reply to Eran Kuris from comment #9)
> (In reply to Pavel Moravec from comment #8)
> > (In reply to Eran Kuris from comment #7)
> > > see inline
> > > 
> > > (In reply to Pavel Moravec from comment #5)
> > > > Thanks for the report, few points to clarify:
> > > > 
> > > > (*) all the mentioned commands and logs should be automatically collected
> > > > when openvswitch or openvswitch-dpdk package is installed, right?
> > > right
> > > 
> > > > (*) for `ovn-nbctl list`, how to get the list of tables in the NB database?
> > > > (i.e. could you provide pseudo-code for this, like:
> > > > 
> > > > for table in $(ovn-nbctl show | head -n1); do
> > > >   ovn-nbctl list $table
> > > > done
> > > > 
> > > > (*) the same for `ovn-sbctl list` please - how to get the list of tables?
> > > 
> > > I think it's better to ask OVN DEV. As far as I know, there are more tables
> > > that we need to collect.

Its good to run these commands to get the db contents. Please allow me some time , I will share a script to dump all the required tables in the db.


> > 
> > Could you pls. raise needinfo on some OVN devel? I dont know any.
> > 
> > > 
> > > > (*) what did you mean by:
> > > > /sosreport-controller-0-20180326122314/var/lib/config-data/
> > > > /sosreport-compute-1-20180326122358/sos_commands/
> > > this is the path where the output of the configuration & commands so I think
> > > we need to collect this folders. 
> > 
> > I dont understand - how these directories are created? (and really with path
> > /sosreport-*, not e.g. /var/tmp/sosreport-* ?) By running some another
> > sosreport (that we should take its output?) Why dont call just one
> > sosreport, then?
> > 
> > Or do you mean the commands output should be stored under sos_commands and
> > configs collected under /var/lib/config-data ? The first is correct all the
> > time, the second almost as well - whatever config or logfile is collected,
> > it is put to the same subtree under the sosreport directory like its origin.
> > I.e. /var/log/openvswitch/ovn will be copied to
> > sosreport-*/var/log/openvswitch/ovn dir
> yes thats what  I meant.
> > 
> > > > 
> > > > 
> > > > (*) "/var/log/openvswitch/ovn" - this is collected when --all-logs is
> > > > provided; shall we collect content of this dir every time? Or just *.log
> > > > files from it?
> > > yes  we need to collect content of this dir every time
> > Whole dir or "just" *.log files there (that is some common approach to
> > collect only *.log files, until --all-logs option is provided)?
> "just" *.log files

"collect only *.log files, until --all-logs option is provided" sounds good to me.


> > > 
> > > > (*) "/var/lib/openvswitch/ovn" - whole dir with some DB files there? How big
> > > > the DB could be? Can't it have some credentials that would be insecure to

The DB files can be pretty big if the customer has created lots of neutron resources. 

I think it's good to copy the db files. I don't think it should have any credentials. All the networking-ovn folks seem to be fine with this. I want to get opinion from core OVN team. I will ask Mark Michelson to share his opinion.

The name of the db files are - ovnnb_db.db (i.e /var/lib/openvswitch/ovn/ovnnb_db.db is the path in the host system) and ovnsb_db.db (i.e /var/lib/openvswitch/ovn/ovnsb_db.db)



> > > > share inside a sosreport? 
> > > need to verify with DeV
> > 
> > Again, please provide a devel name or raise needinfo to them directly.
> 
> @Numan can you help with that ?

(In reply to Numan Siddique from comment #10)
> > > > > (*) "/var/lib/openvswitch/ovn" - whole dir with some DB files there? How big
> > > > > the DB could be? Can't it have some credentials that would be insecure to
> 
> The DB files can be pretty big if the customer has created lots of neutron
> resources. 
> 
> I think it's good to copy the db files.

How big it could be? MBs? tens, hundreds of MBs? How much valuable is all the info for some debugging purposes? Cant be just some tables sufficient?

(i.e. how "data efficient" it would be to get those big data?)

(In reply to Pavel Moravec from comment #11)
> (In reply to Numan Siddique from comment #10)
> > > > > > (*) "/var/lib/openvswitch/ovn" - whole dir with some DB files there? How big
> > > > > > the DB could be? Can't it have some credentials that would be insecure to
> > 
> > The DB files can be pretty big if the customer has created lots of neutron
> > resources. 
> > 
> > I think it's good to copy the db files.
> 
> How big it could be? MBs? tens, hundreds of MBs? How much valuable is all
> the info for some debugging purposes? Cant be just some tables sufficient?
> 
> (i.e. how "data efficient" it would be to get those big data?)
I think the OVN nb db (ovnnb_db.db) it can go to tens of MBs. The south db file (ovnsb_db.db) can grow large (could be into hundreds of MBs) due to logical flows.

I think we can copy only ovnnb_db.db and dump required tables for ovnsb_db.db.

I am working on getting the commands to copy the required tables. I will share it shortly

Below commands can be run to list the ovn db tables

**********
if [ -e /var/lib/openvswitch/ovn/ovnnb_db.sock ] && [ -e /var/lib/openvswitch/ovn/ovnsb_db.sock ]; then
    OVN_NB_DB=unix:/var/lib/openvswitch/ovn/ovnnb_db.sock
    OVN_SB_DB=unix:/var/lib/openvswitch/ovn/ovnsb_db.sock

    ovn-nbctl --db=$OVN_NB_DB show
    ovn-sbctl --db=$OVN_SB_DB show
    ovn-sbctl --db=$OVN_SB_DB list port_binding
    ovn-sbctl --db=$OVN_SB_DB list datapath_binding
    ovn-sbctl --db=$OVN_SB_DB list chassis

    ovsdb-client dump $OVN_NB_DB > [TO SOME FILE IF POSSIBLE]
fi
*************

Funny that this should hit my inbox today. Just yesterday I submitted a pull request to the sosreport project for getting OVN configuration added. I'm going to read through the comments on this issue and make sure that everything you want is being collected. If not, I can submit another pull request with the necessary information.

Maybe you guys can help me with this, but I'm not sure what is necessary after the upstream sosreport submission to ensure that the change is present in Red Hat packages. If you could point me towards documentation or walk me through the process, that would be great. Thanks.

All right, I've read through the comments and have more information for you.

First, the sosreport plugin I submitted copies the database contents. There are no credentials at the OVN level, unless neutron is storing them as external-ids somewhere.

As far as content is concerned, it shows everything you have asked for, except it does not perform an `ovn-nbctl show` or `ovn-sbctl show` . Since each database table is listed, you can get all that information from the database listings instead.

You can see the pull request at: https://github.com/sosreport/sos/pull/1259

(In reply to Mark Michelson from comment #15)
> All right, I've read through the comments and have more information for you.
> 
> First, the sosreport plugin I submitted copies the database contents. There
> are no credentials at the OVN level, unless neutron is storing them as
> external-ids somewhere.
> 
> As far as content is concerned, it shows everything you have asked for,
> except it does not perform an `ovn-nbctl show` or `ovn-sbctl show` . Since
> each database table is listed, you can get all that information from the
> database listings instead.
> 
> You can see the pull request at: https://github.com/sosreport/sos/pull/1259

Thanks Mark for your comments and the pull request.
Since for Openstack OSP13 OVN deployments, ovn services are containerized and while starting the containers, we mount the host directory "/var/log/containers/openvswitch" as "/var/log/openvwitch" in the containers.
So we want the sosreport to pull up the log files from the host path - "/var/log/containers/openvswitch"

For the dbs, we mount the host directory "/var/lib/openvswitch/ovn" as "/etc/openvswitch". So we want sosreport to copy the OVN dbs from host directory - "/var/lib/openvswitch/ovn".

To store the ovn runtime files (.ctl, .sock), we mount the host directory "/var/lib/openvswitch/ovn" as "/run/openvswitch". So to run the ovn-nbctl /ovn-sbctl commands  we have to pass "--db=unix://var/lib/openvswitch/ovn/ovn(nb/sb)_db.sock"

So some more work is required in ovn_central.py sosplugin to check for the above mentioned directories.

All right. What do you think is the best approach for picking the proper directory? Have a list of candidate directories to search? Use some well-known environment variable? Something else?

I would say list of candidate directories to search would be good enough and easy to implement as well.

Hi Numan,

After looking into the options a bit further, I found that defining plugin options for the sosreport plugin was likely the best option. I have updated the pull request to define the following plugin options:

db_dir: Directory where the databases can be found. Defaults to /var/lib/openvswitch
log_dir: Directory where the log files are. Defaults to /var/log/openvswitch
runtime_dir: Directory where runtime files are. Defaults to /var/run/openvswitch
schema_dir: Directory where database schema files are. Defaults to /usr/share/openvswitch

You can issue a command like the following:

`sosreport -o ovn_central -k ovn_central.db_dir=/var/lib/openvswitch/ovn -k ovn_central.log_dir=/var/log/containers/openvswitch -k ovn_central.runtime_dir=/var/lib/openvswitch/ovn`

and you should get the behavior you want.

Thanks Mark. From implementation and design point of you what you have done is very clean.

My only concern is educating the customers/admin to pass these options for OVN deployments when generating sosreports. May be we need to do proper documentation.

@Miguel - You have any thoughts on this ? I mean how easy it would be from user perspective to pass these options.

I think if ovn-central sosplugin can figure it out by looking into the file - /var/run/openvswitch/ovnnb_db.sock first and then check for '/var/lib/openvswitch/ovn/ovnnb_db.sock' would be much more convenient in my opinion.

(In reply to Numan Siddique from comment #20)
> Thanks Mark. From implementation and design point of you what you have done
> is very clean.
> 
> My only concern is educating the customers/admin to pass these options for
> OVN deployments when generating sosreports. May be we need to do proper
> documentation.
> 
> @Miguel - You have any thoughts on this ? I mean how easy it would be from
> user perspective to pass these options.
> 
> I think if ovn-central sosplugin can figure it out by looking into the file
> - /var/run/openvswitch/ovnnb_db.sock first and then check for
> '/var/lib/openvswitch/ovn/ovnnb_db.sock' would be much more convenient in my
> opinion.

Agreed. The less options for the operators the better.

Mark,
thanks a lot for the PR.

Just to double-check / confirm: does the PR reflect all above requirements? (i.e. I dont see any .sock string in the PR while it was discussed recently)

@Pavel

There is no .sock string in the PR because I am no passing a "--db=" option to any ovn-nbctl or ovn-sbctl commands.

@Numan and Daniel

I have mixed feelings. On the one hand, I agree that fewer options are better. On the other hand, the suggested method means that any time some project uses a custom directory for OVN data, the sos project has to be patched to deal with it. Would it be possible to supply a script that wraps the call to sosreport to include the correct directory options? This way, rather than having to learn all the plugin options, operators and technicians would just call the script. If you already have other debugging scripts, then this could be folded into those.

(In reply to Mark Michelson from comment #23)
> @Pavel
> 
> There is no .sock string in the PR because I am no passing a "--db=" option
> to any ovn-nbctl or ovn-sbctl commands.
> 
> @Numan and Daniel
> 
> I have mixed feelings. On the one hand, I agree that fewer options are
> better. On the other hand, the suggested method means that any time some
> project uses a custom directory for OVN data, the sos project has to be
> patched to deal with it. Would it be possible to supply a script that wraps
> the call to sosreport to include the correct directory options? This way,
> rather than having to learn all the plugin options, operators and
> technicians would just call the script. If you already have other debugging
> scripts, then this could be folded into those.

@Eran - Do you have any comments from QE perspective on this ? I am not sure whether it is possible or not.

@Pavel - you have any comments on these ?

I was thinking if it is possible to have both the approaches. i.e ovn-central sosreport plugin provides the options to specify the ovn directories. If the ovn sock/runtime files are not present in either the default directory (cases where options are not defined by user) or in the specified directory using the options, ovn-central plugin can additionally try to look in the list of custom directories it maintains.

I think OVN SOS report plugin should figure it out. it might not be pleasant for the admins/users to pass these arguments.
For OVN deployment I am expecting SOS report will know the location of the logs & other setup details.

@Mark, can you confirm that you are working on this and the tool is going to figure out the correct paths for the files?
Thanks!

@Daniel, yes I will have the pull request updated by COB today (US central time, of course ;) )

I have updated the pull request with two new commits that attempt some automatic directory detection. It still allows for plugin variables to be specified on the command line, but it also will attempt some smart looks at directories to try to find the proper location for OVN data.

Great, thanks a lot Mark! Once merged and we will have the sosreport version with it fixed we can flip this to MODIFIED.

The newest commits in the pull request have been nack'ed by the maintainers of sosreport.

It's not a nack to what you're trying to achieve, just on how it is being done: we don't allow plugins to accept an arbitrary path to be collected in full. This is a strict rule: really, if we are resorting to having users supply -k someplugin.somepath=/var/foo/whatever, then we might just as well ask them to make a tarball of the location.

We've a number of mechanisms in sos today for dealing with variations in file and directory locations, from plugin-internal methods (like the var_puppet_gen mechanism used across the openstack plugins, to general mechanisms like the policy framework and plugin tagging classes that are generally available.

It's just a case of finding a way to solve the problem for this plugin that does not involve accepting arbitrary paths on the command line.

Ah, so if I understand correctly, the nack applies to the plugin variables rather than the search of the paths. I will look at the var_puppet_gen and other code in sos to see what the precedents are for this. Thank you very much!

It's the options that are the biggest problem: searching paths isn't ideal, either, but if it is the only workable option then we can consider it (especially if it can be made robust somehow).

The ideal is a mechanism that is fairly unambiguous: we handle this for distribution variations (package, path, and config file names mainly) via the policies and tagging class mechanism. Subclasses set configuration member variables that are used by the generic superclass code.

The various OpenStack deployments have been starting to look more and more like their own distributions for a while now and it may be that the best solution is to try to extend that (for e.g. RedHatOpenStack policy would be a subclass of RHELPolicy, which recognises both RedHatPlugin and some new RedHatOpenStackPlugin (I don't like that name much, but it'll do as an example).

This may be over the top and we may be able to find something more compact that avoids the need for this right now: I'll need to take a deeper look at what's required to get a feel for that.

My understanding this BZ is a dup of 1560845 that tracks the progress more (incl. 7.5.z). Closing this one, then.

*** This bug has been marked as a duplicate of bug 1560845 ***

I don't think closing this bug as a duplicate was correct. This bug is about the lack of OVN reporting, and the linked issue has to do with collecting openvswitch information from containers. To me, they're separate issues.

My pull request at https://github.com/sosreport/sos/pull/1259 has been merged into the upstream sosreport repo. The pull request says that it was closed without merging, but if you look at the sosreport source, my changes are present. I am changing the state to MODIFIED since the change has been merged upstream.

The pull request was initially intended to collect OVN information in sosreport. However, based on feedback from Pavel and Numan, the pull request MAY also fix 1560845 since there were changes made to the openvswitch plugin to check alternate directories for log files. I'll add a note to that issue.

After looking at the comments on 1560845, I don't think I need to add anything there since they are aware of the pull request already.

(In reply to Mark Michelson from comment #37)
> I don't think closing this bug as a duplicate was correct. This bug is about
> the lack of OVN reporting, and the linked issue has to do with collecting
> openvswitch information from containers. To me, they're separate issues.
> 
> My pull request at https://github.com/sosreport/sos/pull/1259 has been
> merged into the upstream sosreport repo. The pull request says that it was
> closed without merging, but if you look at the sosreport source, my changes
> are present. I am changing the state to MODIFIED since the change has been
> merged upstream.
> 
> The pull request was initially intended to collect OVN information in
> sosreport. However, based on feedback from Pavel and Numan, the pull request
> MAY also fix 1560845 since there were changes made to the openvswitch plugin
> to check alternate directories for log files. I'll add a note to that issue.

Thanks for correction; due to PR merged and fatc we will rebase sos to future upstream, the change will appear in 7.6.

This should have been fixed in RHEL7.6 due to previous updates done in sos 3.6 we rebased to in 7.6. Hence closing the BZ as fixed in current release:

package: sos-3.6-11

7.6.0 errata: https://access.redhat.com/errata/RHEA-2018:3144
7.6.0-day errata: https://access.redhat.com/errata/RHBA-2018:3338

(either one contains the fix)