Bug 1282921 - Please collect haproxy overview page in OSP sosreport
Please collect haproxy overview page in OSP sosreport
Status: ON_QA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sos (Show other bugs)
7.3
x86_64 Linux
medium Severity medium
: rc
: 7.4
Assigned To: Pavel Moravec
Miroslav Hradílek
https://github.com/sosreport/sos/pull...
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-17 14:39 EST by John Fulton
Modified: 2017-11-15 03:10 EST (History)
13 users (show)

See Also:
Fixed In Version: sos-3.5-1.el7
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description John Fulton 2015-11-17 14:39:23 EST
Description of problem:

Please add `curl http://$provision_ip:1993/\;csv -o haproxy_overview.txt` to the OSP sosreport. This will collect haproxy overview information. 

A vanilla install of OSP-D configures the controller nodes with haproxy statistics. For example:  
~~~
listen haproxy.stats
  bind 172.16.0.33:1993 
  mode http
  stats enable
  stats uri /
~~~
Having haproxy_overview.txt (as defined above) would give Support more insight into haproxy.
Comment 2 Bryn M. Reeves 2015-11-18 05:54:26 EST
How do you obtain the value of "$provision_ip"?

Currently only the haproxy plugin deals with haproxy data (this is the ideal); putting this into one of the OpenStack plugins classes gets pretty ugly from that point of view but so does adding unrelated knowledge to the haproxy class in order to obtain the provisioning address.

Is it possible to obtain the in-use address directly from the haproxy configuration?
Comment 3 Joe Talerico 2015-11-18 09:46:08 EST
Pulling from haproxy.cfg we could pull the bind address...

> grep -A1 "haproxy.stats" /etc/haproxy/haproxy.cfg | grep bind | awk '{print $2}
Comment 4 Lee Yarwood 2015-11-19 11:35:04 EST
(In reply to Bryn M. Reeves from comment #2)
> How do you obtain the value of "$provision_ip"?
> 
> Currently only the haproxy plugin deals with haproxy data (this is the
> ideal); putting this into one of the OpenStack plugins classes gets pretty
> ugly from that point of view but so does adding unrelated knowledge to the
> haproxy class in order to obtain the provisioning address.

Agreed but I think John was just looking for this to be in the resulting archive not collected by a specific Openstack plugin. Clearing the needinfo given Joe's reply below.

(In reply to Joe Talerico from comment #3)
> Pulling from haproxy.cfg we could pull the bind address...
> 
> > grep -A1 "haproxy.stats" /etc/haproxy/haproxy.cfg | grep bind | awk '{print $2}

Looking into this now.
Comment 5 Peter Portante 2015-11-19 17:30:45 EST
Is there an haproxy plugin already?  If so, does that already collect the bind address?
Comment 6 Bryn M. Reeves 2015-11-20 06:08:14 EST
commit 8022663faad91d1c6e2346a3b710718616c162be
Author: Bryn M. Reeves <bmr@redhat.com>
Date:   Tue Jun 10 21:47:51 2014 +0100

    [haproxy] new plugin
    
    Add a plugin for HAProxy, based on a suggestion from Brandon
    Perkins in rhbz#1107865.
    
    Fixes #303.
    
    Signed-off-by: Bryn M. Reeves <bmr@redhat.com>

We collect /etc/haproxy/haproxy.cfg, /var/log/haproxy.log and "haproxy -f /etc/haproxy/haproxy.cfg -c" (not much use; just a syntax check).

The bind address for stats (if configured) is present in haproxy.cfg but is not parsed as a separate item.

Adding parsing for this is not impossible but it would be preferable if haproxy itself could report the value it is currently using (we don't really want plugins to end up growing their own configuration parsers at this time).
Comment 9 Pavel Moravec 2017-01-19 04:48:41 EST
Please clarify me some points:

- what particular openstack plugins in sos shall collect it? (or what package presence on the system shall trigger that command execution?)

- is haproxy plugin run on such systems / is haproxy package installed on such systems? (if so, does it make sense to add the cmd to haproxy directly, or would that redundantly call the cmd on non-OSP systems as well?)

- shall not be curl used with -s (suppress progress bar on stdout) and -S (but print errors)?

- shall not be curl used with options e.g. -m or --retry=0 to prevent possible hangs (if e.g. haproxy is not responding or there is some network issue or so)?
Comment 11 John Fulton 2017-01-27 17:45:47 EST
Hi Pavel,

Adding those curl options, as you describe in your last two questions,
sound good to me.

Regarding your first two questions, OSPd installs RH-OSP by putting
the same image on all servers. That image has haproxy installed [1].
However, haproxy is not running on all OpenStack servers, though it
is installed, thus if you want to increase the chances of getting the
status, you might wish to verify the service is running [2] before you
trigger the command as opposed to checking for the presence of the
package alone. As to whether this should be for OpenStack collection
or for haproxy collection alone, I can't say. I feel confident that
OSPd will configure it with stats enabled, as reported in #1, because
configurations are managed by Puppet. When I think of haproxy alone,
I think the user could use whatever configs they want. I don't know
enough about the details of sosplugins so this may not be what you're
looking for. I think Lee understands both well enough so if this isn't
what you are after you might want to try a needinfo from him. 

  John

[1] 
# rpm -q haproxy
haproxy-1.5.18-3.el7_3.1.x86_64
# 

[2] 
# netstat -an | grep 1993
tcp        0      0 192.168.24.3:1993       0.0.0.0:*               LISTEN     
#
Comment 12 Pavel Moravec 2017-02-12 16:04:17 EST
(In reply to Pavel Moravec from comment #9)
> Please clarify me some points:
> 
> - what particular openstack plugins in sos shall collect it? (or what
> package presence on the system shall trigger that command execution?)
> 
> - is haproxy plugin run on such systems / is haproxy package installed on
> such systems? (if so, does it make sense to add the cmd to haproxy directly,
> or would that redundantly call the cmd on non-OSP systems as well?)
> 

Hi Lee,
could you please answer these questions? I am uncertain from #c11 where / to what plugin(s) the curl command to add.
Comment 13 Lee Yarwood 2017-03-14 10:29:20 EDT
(In reply to Pavel Moravec from comment #12)
> (In reply to Pavel Moravec from comment #9)
> > Please clarify me some points:
> > 
> > - what particular openstack plugins in sos shall collect it? (or what
> > package presence on the system shall trigger that command execution?)

This shouldn't be an openstack plugin thing, we just need to extend the HAProxy plugin Bryn highlighted in c#6.

> > - is haproxy plugin run on such systems / is haproxy package installed on
> > such systems? (if so, does it make sense to add the cmd to haproxy directly,
> > or would that redundantly call the cmd on non-OSP systems as well?)
> > 

Always installed, only used in the control plane. Again, we should just extend the HAProxy plugin itself to search for the bind_address and if found use it to collect this page.
Comment 14 Pavel Moravec 2017-03-15 17:25:18 EDT
Ah, sorry for confusion on my side.

I filed https://github.com/sosreport/sos/pull/961 but I feel there is something wrong there:

(In reply to John Fulton from comment #0)

> statistics. For example:  
> ~~~
> listen haproxy.stats
>   bind 172.16.0.33:1993 
>   mode http
>   stats enable
>   stats uri /
> ~~~

See the $provision_ip contains port also. While:


(In reply to John Fulton from comment #0)
> Please add `curl http://$provision_ip:1993/\;csv -o haproxy_overview.txt` to

has the port as well.

Must be the haproxy service listen on 1993 every time? If "bind" line contains the port, shall we use that one? And only if no port is given, use 1993?

(I guess the address could be IPv6 address containing ':' so then detecting port presence is even more fun..)

What is the procedure to identify the IP + port?

(reply here or better in the upstream PR)
Comment 15 John Fulton 2017-03-16 10:00:59 EDT
Hi Pavel,

(In reply to Pavel Moravec from comment #14)
> Must be the haproxy service listen on 1993 every time?

No, it will listen on the port that the user sets in the conf
file. Normally the user will use the default or whatever the config
tool (e.g. puppet uses) but if you want to be more complete, then
don't assume that. Extract it from the conf file.

> If "bind" line contains the port, shall we use that one? And only if
> no port is given, use 1993?

Sounds fine to me. 

> (I guess the address could be IPv6 address containing ':' so then detecting
> port presence is even more fun..)

Sure, you could extract the IP and use a regex to determine if it's
IPv6 or IPv4. 

> What is the procedure to identify the IP + port?

It's your implementation so you should probably decide how you want to
do that. One idea is:

Find lines containing "stats" and "bind" from /etc/haproxy/haproxy.cfg
If there is no stats, don't curl. Else, parse the bind line for the
port and IP address. Use a regex to determine if you have an IPv4 or
IPv6 address. Based on that info you should be able to separate the
address from the port and curl accordingly.

You might just be able to assume that if it's to the right of "bind",
that you can curl it. You could look check the haproxy config parser
to see if that's a safe assumption.

Pick your favorite from the above or use another method. It's your call.
Comment 16 Pavel Moravec 2017-03-16 11:27:37 EDT
(In reply to John Fulton from comment #15)
> Hi Pavel,
> 
> (In reply to Pavel Moravec from comment #14)
> > Must be the haproxy service listen on 1993 every time?
> 
> No, it will listen on the port that the user sets in the conf
> file. Normally the user will use the default or whatever the config
> tool (e.g. puppet uses) but if you want to be more complete, then
> don't assume that. Extract it from the conf file.
> 
> > If "bind" line contains the port, shall we use that one? And only if
> > no port is given, use 1993?
> 
> Sounds fine to me. 
> 
> > (I guess the address could be IPv6 address containing ':' so then detecting
> > port presence is even more fun..)
> 
> Sure, you could extract the IP and use a regex to determine if it's
> IPv6 or IPv4. 
> 
> > What is the procedure to identify the IP + port?
> 
> It's your implementation so you should probably decide how you want to
> do that. One idea is: ..

Thanks for provided clarification. Yeah I meant what is the theoretical procedure "if bind line isnt there, then ..; else if port isnt there, then .. else .." and so, but I understood it now:

- if bind line is missing -> no curl
- detect if port is in the bind URL - if so, use that one
- if no port is there, use default 1993 one
Comment 17 Pavel Moravec 2017-03-16 14:43:25 EDT
I rebased my PR but appreciate if somebody can test it in a sosreport:

https://github.com/sosreport/sos/pull/961

Note You need to log in before you can comment on or make changes to this bug.