Bug 1250346

It's not missing: it does not exist upstream.

To get into an RH build it first needs to go there.

Leonid, I'm struggling to understand what actually needs to be collected here. Should we just check the default template location for changes and record these or did you have something else in mind?

Hey,Lee. Yes,checking the default template location for changes is good enough.
Thanks.

Moving this to RHEL.  We don't ship sos in OSP.

No pull or patch received upstream so this is not in the sos-3.3 build shipped in 7.3.

RHEL7.3 has been released. Re-scheduling for potential inclusion in 7.4.

Hint: please file upstream PR or at least specify:

- what package presence shall trigger the plugin
- what commands outputs it shall collect
- what files it shall collect
- what secrets might appear in either collected data that shall sos obfuscate

Just a kind reminder - please respond to #c10 in case you wish the plugin in RHEL7.4.

We are now in planning phase and I (iteratively) prioritize bugs according their severity / impact / implementation readiness.

There is already the openstack_instack plugin which collects information from the undercloud. We could extend the RedHatOpenStackRDOManager class to collect further information.

Here some thoughts on what should be collected. 

- what package presence shall trigger the plugin

  openstack-tripleo-heat-templates

- what commands outputs it shall collect

   - check changes to the default templates:
       # rpm -V openstack-tripleo-heat-templates

   - get debug information from the undercloud as user 'stack'
       $ source ~/stackrc 

       collect information as in:
       http://git.openstack.org/cgit/openstack/tripleo-incubator/plain/scripts/undercloud-debug

       We can skip the ovs part from the referenced script as this gets already collected by the 
       ovs plugin.

- what files it shall collect
  - /etc/os-net-config BZ 1365251

- what secrets might appear in either collected data that shall sos obfuscate
  - the output of ironic node-show has the information on the IPMI user/pwd

Thanks for the info, still some pending questions from non-OSP guy:

(In reply to Martin Schuppert from comment #12)
> There is already the openstack_instack plugin which collects information
> from the undercloud. We could extend the RedHatOpenStackRDOManager class to
> collect further information.

You mean "class RedHatRDOManager(OpenStackInstack, RedHatPlugin):", I guess. Just this plugin shall be extended, no new plugin?

> 
> Here some thoughts on what should be collected. 
> 
> - what package presence shall trigger the plugin
> 
>   openstack-tripleo-heat-templates

So the RedHatRDOManager class/plugin will be invoked by presence of either of package:

instack
instack-undercloud
openstack-tripleo-heat-templates

?



> 
> - what commands outputs it shall collect
> 
>    - check changes to the default templates:
>        # rpm -V openstack-tripleo-heat-templates
> 
>    - get debug information from the undercloud as user 'stack'
>        $ source ~/stackrc 

Why the source command is required? Will it provide some output or is it necessary for the undercloud-debug ?

> 
>        collect information as in:
>       
> http://git.openstack.org/cgit/openstack/tripleo-incubator/plain/scripts/
> undercloud-debug
> 
>        We can skip the ovs part from the referenced script as this gets
> already collected by the 
>        ovs plugin.

Does it make sense to call the script directly and store its output file/log undercloud-debug.log ?

(i.e. calling the script directly leaves the necessity to maintain the script, but simplifies updating the info it collects - you need to update your script with shorter release cycle and not sosreport directly)

> 
> - what files it shall collect
>   - /etc/os-net-config BZ 1365251

Will adding that directory just in this plugin sufficient for resolving BZ 1365251 ? Per https://bugzilla.redhat.com/show_bug.cgi?id=1365251#c3 I am not sure if the directory shall not be collected also in other cases (by some other plugin).

> 
> - what secrets might appear in either collected data that shall sos obfuscate
>   - the output of ironic node-show has the information on the IPMI user/pwd

In https://www.rdoproject.org/install/tripleo-cli/ I see ssh_username and ssh_key_contents, is that what you mean? or can you provide better example / ideally the regexp directly?

*** Bug 1250448 has been marked as a duplicate of this bug. ***

sorry for the delay!

(In reply to Pavel Moravec from comment #13)
> Thanks for the info, still some pending questions from non-OSP guy:
> 
> (In reply to Martin Schuppert from comment #12)
> > There is already the openstack_instack plugin which collects information
> > from the undercloud. We could extend the RedHatOpenStackRDOManager class to
> > collect further information.
> 
> You mean "class RedHatRDOManager(OpenStackInstack, RedHatPlugin):", I guess.
> Just this plugin shall be extended, no new plugin?

yes that was my idea

> 
> > 
> > Here some thoughts on what should be collected. 
> > 
> > - what package presence shall trigger the plugin
> > 
> >   openstack-tripleo-heat-templates
> 
> So the RedHatRDOManager class/plugin will be invoked by presence of either
> of package:
> 
> instack
> instack-undercloud
> openstack-tripleo-heat-templates
> 
> ?

yes

> 
> 
> 
> > 
> > - what commands outputs it shall collect
> > 
> >    - check changes to the default templates:
> >        # rpm -V openstack-tripleo-heat-templates
> > 
> >    - get debug information from the undercloud as user 'stack'
> >        $ source ~/stackrc 
> 
> Why the source command is required? Will it provide some output or is it
> necessary for the undercloud-debug ?

We need to source the user auth details to be able to run the commands against 
the OSP services or we can grep them from there and pass them with arguments . 
Otherwise we can not connect.

> 
> > 
> >        collect information as in:
> >       
> > http://git.openstack.org/cgit/openstack/tripleo-incubator/plain/scripts/
> > undercloud-debug
> > 
> >        We can skip the ovs part from the referenced script as this gets
> > already collected by the 
> >        ovs plugin.
> 
> Does it make sense to call the script directly and store its output file/log
> undercloud-debug.log ?
> 
> (i.e. calling the script directly leaves the necessity to maintain the
> script, but simplifies updating the info it collects - you need to update
> your script with shorter release cycle and not sosreport directly)

As we do not do that for the other openstack plugins, not sure if we should do 
that here as well. 

> 
> > 
> > - what files it shall collect
> >   - /etc/os-net-config BZ 1365251
> 
> Will adding that directory just in this plugin sufficient for resolving BZ
> 1365251 ? Per https://bugzilla.redhat.com/show_bug.cgi?id=1365251#c3 I am
> not sure if the directory shall not be collected also in other cases (by
> some other plugin).

yes correct, this could also be on the overcloud nodes. So we should either extend
one we already have or create a new one.

> 
> > 
> > - what secrets might appear in either collected data that shall sos obfuscate
> >   - the output of ironic node-show has the information on the IPMI user/pwd
> 
> In https://www.rdoproject.org/install/tripleo-cli/ I see ssh_username and
> ssh_key_contents, is that what you mean? 

yes that is what I meant. As this is ssh with pubkey auth this should not be
an issue, but we do not support this 

> or can you provide better example / ideally the regexp directly?

for e.g ipmi driver we already obfuscate the pwd. Is it a must to obfuscate the
username or are we good with that output?

[root@ibm-x3630m4-5 ~]# ironic node-show c0b039de-d3e5-4049-9df4-8fabde80c8c8 | more
+------------------------+--------------------------------------------------------------------------+
| Property               | Value                                                                    |
+------------------------+--------------------------------------------------------------------------+
| chassis_uuid           |                                                                          |
| clean_step             | {}                                                                       |
| console_enabled        | False                                                                    |
| created_at             | 2017-01-14T05:07:49+00:00                                                |
| driver                 | pxe_ipmitool                                                             |
| driver_info            | {u'deploy_kernel': u'a326a2f7-fcd1-4f6f-9c4f-4a8aad63ef8d',              |
|                        | u'ipmi_address': u'10.65.176.22', u'deploy_ramdisk': u'b66d14b9-e26c-    |
|                        | 4fff-a19d-965ceafb6b4f', u'ipmi_password': u'******', u'ipmi_username':  |
|                        | u'rcuser'}                                                               |

(In reply to Martin Schuppert from comment #15)
> sorry for the delay!

Sorry for a delay from my side as well :)

> > You mean "class RedHatRDOManager(OpenStackInstack, RedHatPlugin):", I guess.
> > Just this plugin shall be extended, no new plugin?
> 
> yes that was my idea

OK

> > So the RedHatRDOManager class/plugin will be invoked by presence of either
> > of package:
> > 
> > instack
> > instack-undercloud
> > openstack-tripleo-heat-templates
> > 
> > ?
> 
> yes

OK

> > >        $ source ~/stackrc 
> > 
> > Why the source command is required? Will it provide some output or is it
> > necessary for the undercloud-debug ?
> 
> We need to source the user auth details to be able to run the commands
> against 
> the OSP services or we can grep them from there and pass them with arguments
> . 
> Otherwise we can not connect.

OK

> > Does it make sense to call the script directly and store its output file/log
> > undercloud-debug.log ?
> > 
> > (i.e. calling the script directly leaves the necessity to maintain the
> > script, but simplifies updating the info it collects - you need to update
> > your script with shorter release cycle and not sosreport directly)
> 
> As we do not do that for the other openstack plugins, not sure if we should
> do 
> that here as well. 

I am not sure what the answer is - can sosreport just call the undercloud-debug command (and collect output), or do you require retiring the tool in OSP and rewriting the commands from the tool into sosreport?

For the first, it's trivial.

For the second, reviewing what the tool does, it will be tricky to implement sos plugin to call that cmds individually and put everything into one logfile - is it fine if individual commands output would be stored in individual files in sosreport? I.e.:

"ironic node-list" command will be run and output will be stored in sos_commands/openstack_instack/ironic_node-list file.

"for n in ..; do ironic node-port-list $n; done" commands will be run and outputs will be stored in sos_commands/openstack_instack/ironic_node-port-list_* files.

etc.

Please decide if sos can call the tool directly, or re-write the cmds with above change in output file(s).

> > > - what files it shall collect
> > >   - /etc/os-net-config BZ 1365251
> > 
> > Will adding that directory just in this plugin sufficient for resolving BZ
> > 1365251 ? Per https://bugzilla.redhat.com/show_bug.cgi?id=1365251#c3 I am
> > not sure if the directory shall not be collected also in other cases (by
> > some other plugin).
> 
> yes correct, this could also be on the overcloud nodes. So we should either
> extend
> one we already have or create a new one.

OK, it will be better to deal with os-net-config separately in that BZ - will raise needinfo there.


> 
> > 
> > > 
> > > - what secrets might appear in either collected data that shall sos obfuscate
> > >   - the output of ironic node-show has the information on the IPMI user/pwd
> > 
> > In https://www.rdoproject.org/install/tripleo-cli/ I see ssh_username and
> > ssh_key_contents, is that what you mean? 
> 
> yes that is what I meant. As this is ssh with pubkey auth this should not be
> an issue, but we do not support this 

OK so I understood no need to obfuscate.

> for e.g ipmi driver we already obfuscate the pwd. Is it a must to obfuscate
> the
> username or are we good with that output?

If the command/tool already obfuscates the credentials, I would let it as is. Until knowing also standalone username is sort of vulnerable information that some customers might be afraid to share via sosreport.

(In reply to Pavel Moravec from comment #16)
> (In reply to Martin Schuppert from comment #15)
> > sorry for the delay!
> 
> Sorry for a delay from my side as well :)
> 
> > > You mean "class RedHatRDOManager(OpenStackInstack, RedHatPlugin):", I guess.
> > > Just this plugin shall be extended, no new plugin?
> > 
> > yes that was my idea
> 
> OK
> 
> > > So the RedHatRDOManager class/plugin will be invoked by presence of either
> > > of package:
> > > 
> > > instack
> > > instack-undercloud
> > > openstack-tripleo-heat-templates
> > > 
> > > ?
> > 
> > yes
> 
> OK
> 
> > > >        $ source ~/stackrc 
> > > 
> > > Why the source command is required? Will it provide some output or is it
> > > necessary for the undercloud-debug ?
> > 
> > We need to source the user auth details to be able to run the commands
> > against 
> > the OSP services or we can grep them from there and pass them with arguments
> > . 
> > Otherwise we can not connect.
> 
> OK
> 
> > > Does it make sense to call the script directly and store its output file/log
> > > undercloud-debug.log ?
> > > 
> > > (i.e. calling the script directly leaves the necessity to maintain the
> > > script, but simplifies updating the info it collects - you need to update
> > > your script with shorter release cycle and not sosreport directly)
> > 
> > As we do not do that for the other openstack plugins, not sure if we should
> > do 
> > that here as well. 
> 
> I am not sure what the answer is - can sosreport just call the
> undercloud-debug command (and collect output), or do you require retiring
> the tool in OSP and rewriting the commands from the tool into sosreport?
> 
> For the first, it's trivial.
> 
> For the second, reviewing what the tool does, it will be tricky to implement
> sos plugin to call that cmds individually and put everything into one
> logfile - is it fine if individual commands output would be stored in
> individual files in sosreport? I.e.:
> 
> "ironic node-list" command will be run and output will be stored in
> sos_commands/openstack_instack/ironic_node-list file.
> 
> "for n in ..; do ironic node-port-list $n; done" commands will be run and
> outputs will be stored in
> sos_commands/openstack_instack/ironic_node-port-list_* files.
> 
> etc.
> 
> Please decide if sos can call the tool directly, or re-write the cmds with
> above change in output file(s).

It is not needed to have everything in a single log. But as we ship it, lets
use the tool at /usr/libexec/openstack-tripleo/undercloud-debug . It writes
everything to stdout and to undercloud-debug.log .

A call could be:
su -s /bin/sh -c "cd /var/tmp && source ~/stackrc && /usr/libexec/openstack-tripleo/undercloud-debug" stack

> 
> > > > - what files it shall collect
> > > >   - /etc/os-net-config BZ 1365251
> > > 
> > > Will adding that directory just in this plugin sufficient for resolving BZ
> > > 1365251 ? Per https://bugzilla.redhat.com/show_bug.cgi?id=1365251#c3 I am
> > > not sure if the directory shall not be collected also in other cases (by
> > > some other plugin).
> > 
> > yes correct, this could also be on the overcloud nodes. So we should either
> > extend
> > one we already have or create a new one.
> 
> OK, it will be better to deal with os-net-config separately in that BZ -
> will raise needinfo there.
> 
> 
> > 
> > > 
> > > > 
> > > > - what secrets might appear in either collected data that shall sos obfuscate
> > > >   - the output of ironic node-show has the information on the IPMI user/pwd
> > > 
> > > In https://www.rdoproject.org/install/tripleo-cli/ I see ssh_username and
> > > ssh_key_contents, is that what you mean? 
> > 
> > yes that is what I meant. As this is ssh with pubkey auth this should not be
> > an issue, but we do not support this 
> 
> OK so I understood no need to obfuscate.

Yes, right now I do not have info that we need to obfuscate.

> 
> > for e.g ipmi driver we already obfuscate the pwd. Is it a must to obfuscate
> > the
> > username or are we good with that output?
> 
> If the command/tool already obfuscates the credentials, I would let it as
> is. Until knowing also standalone username is sort of vulnerable information
> that some customers might be afraid to share via sosreport.

Ok

See https://github.com/sosreport/sos/pull/936

Additionally to agreed above, RSA private key in ironic node-show outputs.

Leonid, Martin,
it was agreed in upstream that whole undercloud-debug wont be collected due to several reasons, and rather its commands (partially collected by several sos plugins, already) shall be collected by relevant plugins.

While Lee is under load and deadline for sos 3.4 is approaching, and some OSP person shall help here, could you please contribute to:

https://github.com/sosreport/sos/pull/936

?

Submitted 

https://github.com/sosreport/sos/pull/955

also a fix for the gnocchi plugin

https://github.com/sosreport/sos/pull/956

POSTed to upstream in https://github.com/sosreport/sos/commit/4f7ea5d46f7531e8fdb8a58738a747f79a9c4841

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2203