Bug 1541100

Summary: SOS report should collect Octavia info
Product: Red Hat Enterprise Linux 7 Reporter: Alexander Stafeyev <astafeye>
Component: sosAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: urgent Docs Contact: Apurva Bhide <abhide>
Priority: high    
Version: 7.4CC: abhide, agk, amuller, astafeye, bcafarel, bmr, cgoncalves, cww, gavin, ihrachys, lpeer, majopela, mhradile, nyechiel, oblaut, plambri, pmoravec, sbradley
Target Milestone: pre-dev-freezeKeywords: OtherQA, Triaged, ZStream
Target Release: 7.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: GSSApproved
Fixed In Version: sos-3.6-2.el7 Doc Type: If docs needed, set a value
Doc Text:
With this update, the "openstack_octavia" plug-in has been added. As a result, the sosreport now collects data related to OpenStack Octavia network load balancer.
Story Points: ---
Clone Of:
: 1568882 (view as bug list) Environment:
Last Closed: 2018-10-30 10:31:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1433523, 1568380, 1568882    
Attachments:
Description Flags
initial version of the plugin
none
openstack_octavia.py.new
none
openstack_octavia.py.diff none

Description Alexander Stafeyev 2018-02-01 17:09:21 UTC
[root@controller-0 ~]# sosreport -l | grep neutron
 openstack_neutron    OpenStack Networking
[root@controller-0 ~]# sosreport -l | grep octav
[root@controller-0 ~]# 


The SOS report should collect logs and needed info about Octavia as well

Comment 3 Assaf Muller 2018-02-02 16:45:51 UTC
Moving to sosreport component under RHEL because that is where we expect the code to land. The first step would be to send a patch to upstream sosreport.

Comment 6 Assaf Muller 2018-02-02 16:49:33 UTC
Carlos, can you please list all files we'd like to collect for Octavia? You can gain inspiration from other openstack sosreport plugins:
https://github.com/sosreport/sos/tree/master/sos/plugins

Comment 7 Pavel Moravec 2018-02-03 16:02:38 UTC
Given the fact there is no work done, reviewing PR will take some time and proximity of 7.5.0 date, I dont think we will be able to incorporate the plugin into 7.5.0 errata. So updating flags.

If 7.5.z is required, please raise it.

I can help with the plugin (to some extent), let me know via mail if needed.

Comment 9 Carlos Goncalves 2018-02-04 10:35:00 UTC
profiles = ('openstack', 'openstack_controller')


RPM packages:
openstack-octavia-amphora-agent
openstack-octavia-api
openstack-octavia-common
openstack-octavia-diskimage-create
openstack-octavia-health-manager
openstack-octavia-housekeeping
openstack-octavia-ui
openstack-octavia-worker
puppet-octavia
python-octavia
python-octaviaclient


Log files:
/var/log/containers/httpd/octavia-api/*.log
/var/log/containers/octavia/*.log
/var/log/octavia/*.log


etc files:
/etc/logrotate.d/openstack-octavia
/etc/octavia/*


Puppet files:
/var/lib/octavia
/var/lib/config-data/puppet-generated/octavia/etc/octavia/
/var/lib/config-data/puppet-generated/octavia/etc/my.cnf.d/tripleo.cnf



def check_enabled(self):
        return self.is_installed("openstack-octavia-common")



add_cmd_output:
openstack loadbalancer list
openstack loadbalancer amphora list
openstack loadbalancer healthmonitor list
openstack loadbalancer l7policy list
openstack loadbalancer pool list
openstack loadbalancer quota list




Command requiring argument:

openstack loadbalancer l7rule list <l7policy>
openstack loadbalancer member list <pool>


Retrieve information from each load balancer (similar to https://github.com/sosreport/sos/blob/master/sos/plugins/openstack_nova.py#L90)


for lb_id in $(openstack loadbalancer list -f id);
do
    openstack loadbalancer loadbalancer show $ld_id
done

Comment 10 Pavel Moravec 2018-02-25 09:23:10 UTC
Created attachment 1400538 [details]
initial version of the plugin

Copy it under

/usr/lib/python2.7/site-packages/sos/plugins/

directory and test it.

Few questions:

(0) I added effect of --all-logs cmdline argument - using it, all files (not only *.log) from the relevant dirs will be collected.

Also worth to know the default limit 10MB is applied here.

Let me know if either of above is a problem.


(1)
> Command requiring argument:
> 
> openstack loadbalancer l7rule list <l7policy>
> openstack loadbalancer member list <pool>

Where to take the argument from? Parsing some output of cmds:

openstack loadbalancer l7policy list
openstack loadbalancer pool list

(or iterate over those two lists and take cmds for each member, like done for "openstack loadbalancer loadbalancer show $ld_id" ?)


(2) I know OSP is often containerized, what implies some changes in configs/logs/cmds. Wont this happen here as well?


(3) Can't be there some password/certificate/secret in data collected we shall obfuscate?



(once we have some stable version of the plugin, I will create PR in upstream)

Comment 11 Pavel Moravec 2018-02-25 09:24:41 UTC
needinfo for the plugin testing

Comment 12 Carlos Goncalves 2018-03-05 14:32:37 UTC
I am still waiting for a RDO Queens/OSP 13 deployment to test with your octavia sosreport plugin.


(1) I'll do it.

(2) Overcloud is containerized from OSP 13 onward. I based my input in comment #9 based on existing sosreport plugins, which I assume cover configs/logs/cmds on containerized envs. If I missed something, could you please be more detailed? Thank you.

(3) I'll have a look once I get a deployment running and report back.

Comment 15 Carlos Goncalves 2018-03-27 09:00:21 UTC
Created attachment 1413620 [details]
openstack_octavia.py.new

Comment 16 Carlos Goncalves 2018-03-27 09:00:57 UTC
Created attachment 1413631 [details]
openstack_octavia.py.diff

Comment 17 Carlos Goncalves 2018-03-27 09:01:19 UTC
Pavel, 

(1) Completed the missing l7rule and member list commands.

(2) the plugin is collecting the config and log files in a containerized deployment.

(3) ofuscated couple of passwords/key-phrases.

(4) added a verify step to log installed RPMs (rpm -V), although it does only log whether the packages are installed or not and that is consistent with other plugins. Could we log the package NVR even if breaks consistency?


Attaching updated file and diff file.

Comment 18 Pavel Moravec 2018-04-02 13:25:36 UTC
Thanks, fixed one minor formatting stuff and create upstream PR after review.

devel_ack+ for 7.6

for 7.5.z if needed, please provide business justification and get PMApproved.

Comment 34 Pavel Moravec 2018-04-18 10:37:43 UTC
GSSApproved for 7.5.z

Comment 35 Pavel Moravec 2018-04-18 11:06:08 UTC
posted to dist-git

Comment 48 errata-xmlrpc 2018-10-30 10:31:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3144