Bug 1506866 - [3.9] haproxy lb install failed when haproxy-1.7.9-1.el7.x86_64 is installed.
[3.9] haproxy lb install failed when haproxy-1.7.9-1.el7.x86_64 is installed.
Status: VERIFIED
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
3.9.0
Unspecified Unspecified
high Severity high
: ---
: 3.9.0
Assigned To: Russell Teague
Johnny Liu
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-10-26 23:39 EDT by Johnny Liu
Modified: 2018-01-26 02:29 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
haproxy.cfg template in the load-balancer role was not updated to reflect changes in new versions of haproxy causing the service to fail to start. The config file template was updated to work for newer versions of haproxy.
Story Points: ---
Clone Of:
: 1538789 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Johnny Liu 2017-10-26 23:39:13 EDT
Description of problem:
See the following detailed

Version-Release number of the following components:
3.7/2017-10-26.4_v3.7.0-0.181.0
openshift-ansible-3.7.0-0.181.0.git.0.34f6e3e.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. create ocp repo on haproxy lb host 
2. trigger a multiple master rpm install with haproxy lb
3.

Actual results:
installation failed.
TASK [openshift_master : Wait for API to become available] *********************
Friday 27 October 2017  02:47:22 +0000 (0:00:00.121)       0:13:12.100 ******** 
<--snip-->
FAILED - RETRYING: Wait for API to become available (1 retries left).
 [WARNING]: Consider using get_url or uri module rather than running curl

fatal: [qe-jialiu1-khis-master-etcd-1.1027-31k.qe.rhcloud.com]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "--silent", "--tlsv1.2", "--cacert", "/etc/origin/master/ca-bundle.crt", "https://qe-jialiu1-khis-lb-1.1027-31k.qe.rhcloud.com:8443/healthz/ready"], "delta": "0:00:00.012922", "end": "2017-10-26 22:51:39.902043", "failed": true, "msg": "non-zero return code", "rc": 7, "start": "2017-10-26 22:51:39.889121", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Go to master, master api service is running well.
Go to haproxy lb host, haproxy service is NOT started.
# service haproxy status
Redirecting to /bin/systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/haproxy.service.d
           └─limits.conf
   Active: failed (Result: exit-code) since Thu 2017-10-26 23:13:20 EDT; 3s ago
  Process: 16854 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 16854 (code=exited, status=1/FAILURE)

Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: Starting HAProxy Load Balancer...
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : parsing [/etc/haproxy/haproxy.cfg:37] : 'listen' cannot handle unexpected a... ':9000'.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : parsing [/etc/haproxy/haproxy.cfg:37] : please use the 'bind' keyword for l...ddresses.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : Fatal errors found in configuration.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: haproxy-systemd-wrapper: exit, haproxy RC=1
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: Unit haproxy.service entered failed state.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: haproxy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.



Expected results:
haproxy lb should be installed successfully.

Additional info:
1. downgrade haproxy to haproxy-1.5.18-6.el7.x86_64, restart haproxy successfully.
2. update haproxy to haproxy-1.7.9-1.el7.x86_64 again, haproxy still failed, then removing the following lines from /etc/haproxy/haproxy.cfg:

#listen stats :9000
#    mode http
#    stats enable
#    stats uri /

Restart haproxy, this time it succeed. So seem like the new haproxy does NOT support the old syntax.

And this haproxy rpm is installed from OCP puddle.
Comment 1 Scott Dodson 2017-10-27 11:22:35 EDT
Workaround, downgrade to haproxy 1.5.x.

long term we should configure the installer to either force 1.5 or we need to lay down configuration that's specific to 1.7 or 1.5 based on the version installed.
Comment 3 Ben Bennett 2017-10-27 12:01:00 EDT
If we change the config to:

listen stats
    bind :9000
    mode http
    stats enable
    stats uri /

It should work with 1.5 and 1.7.  BUT the larger issue of the RPM being being named haproxy and conflicting still applies.
Comment 8 Gaoyun Pei 2018-01-22 04:04:18 EST
So this issue is blocking containerized haproxy service start now.

[root@ip-172-18-7-218 ~]# docker images
REPOSITORY                                                         TAG                 IMAGE ID            CREATED             SIZE
registry.x.x.com:443/openshift3/ose-haproxy-router   v3.9.0              5ce9aed6c36c        2 days ago          1.243 GB

[root@ip-172-18-7-218 ~]# docker run --entrypoint rpm registry.x.x.com:443/openshift3/ose-haproxy-router:v3.9.0 -qa |grep haproxy
haproxy18-1.8.1-5.el7.x86_64



Trigger a containerized ha-master 3.9 cluster, haproxy service failed to start on containerized lb host.

[root@ip-172-18-7-218 ~]# journalctl --no-pager -u haproxy.service

Jan 22 02:17:11 ip-172-18-7-218.ec2.internal systemd[1]: Starting haproxy.service...
Jan 22 02:17:11 ip-172-18-7-218.ec2.internal docker[5745]: Error response from daemon: No such container: openshift_loadbalancer
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : parsing [/etc/haproxy/haproxy.cfg:30] : 'listen' cannot handle unexpected argument ':9000'.
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : parsing [/etc/haproxy/haproxy.cfg:30] : please use the 'bind' keyword for listening addresses.
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : Fatal errors found in configuration.
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE

[root@ip-172-18-7-218 ~]# grep -n ".*" /etc/haproxy/haproxy.cfg
1:# Global settings
2:#---------------------------------------------------------------------
...
30:listen stats :9000
31:    mode http
32:    stats enable
33:    stats uri /
Comment 9 Scott Dodson 2018-01-22 08:29:45 EST
Ben,

Is the fix in https://bugzilla.redhat.com/show_bug.cgi?id=1506866#c3 still valid now that we're on haproxy 1.8?
Comment 10 Ben Bennett 2018-01-23 11:04:54 EST
@scott: Yes... but the current haproxy.conf has the fix in it... where is the one for that container coming from?
Comment 11 Scott Dodson 2018-01-23 13:15:06 EST
This is an openshift-ansible specific problem, we re-use the haproxy image during containerized installs to serve as an API server load balancer and we generate our own config.

we just need to update https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_loadbalancer/templates/haproxy.cfg.j2 with the changes suggested in comment 3.
Comment 12 Russell Teague 2018-01-23 13:30:22 EST
Proposed: https://github.com/openshift/openshift-ansible/pull/6839
Comment 13 Russell Teague 2018-01-24 13:03:35 EST
Merged
Comment 14 Gaoyun Pei 2018-01-25 04:12:19 EST
As PR#6839 was already merged, and it should be included in openshift-ansible 3.9.0-0.24.0+ version, so gave it a try on master branch.

$ git describe
openshift-ansible-3.9.0-0.24.0-8-g2487fa8


Containerized haproxy service could run well. 

-bash-4.2# docker run --entrypoint rpm openshift3/ose-haproxy-router:v3.9.0 -qa |grep haproxy
haproxy18-1.8.1-5.el7.x86_64

-bash-4.2# sed -n '30,35p' /etc/haproxy/haproxy.cfg
listen stats
    bind :9000
    mode http
    stats enable
    stats uri /
Comment 15 Gaoyun Pei 2018-01-26 02:29:05 EST
Move this bug to verified with openshift-ansible-3.9.0-0.24.0.git.0.735690f.el7.noarch.rpm according to Comment 14.

Note You need to log in before you can comment on or make changes to this bug.