Bug 1506866 - [3.9] haproxy lb install failed when haproxy-1.7.9-1.el7.x86_64 is installed.
Summary: [3.9] haproxy lb install failed when haproxy-1.7.9-1.el7.x86_64 is installed.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.9.0
Assignee: Russell Teague
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-27 03:39 UTC by Johnny Liu
Modified: 2018-03-28 14:09 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
haproxy.cfg template in the load-balancer role was not updated to reflect changes in new versions of haproxy causing the service to fail to start. The config file template was updated to work for newer versions of haproxy.
Clone Of:
: 1538789 (view as bug list)
Environment:
Last Closed: 2018-03-28 14:08:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1483579 0 medium CLOSED Create an haproxy-1.8 package 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2018:0489 0 None None None 2018-03-28 14:09:21 UTC

Internal Links: 1483579

Description Johnny Liu 2017-10-27 03:39:13 UTC
Description of problem:
See the following detailed

Version-Release number of the following components:
3.7/2017-10-26.4_v3.7.0-0.181.0
openshift-ansible-3.7.0-0.181.0.git.0.34f6e3e.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. create ocp repo on haproxy lb host 
2. trigger a multiple master rpm install with haproxy lb
3.

Actual results:
installation failed.
TASK [openshift_master : Wait for API to become available] *********************
Friday 27 October 2017  02:47:22 +0000 (0:00:00.121)       0:13:12.100 ******** 
<--snip-->
FAILED - RETRYING: Wait for API to become available (1 retries left).
 [WARNING]: Consider using get_url or uri module rather than running curl

fatal: [qe-jialiu1-khis-master-etcd-1.1027-31k.qe.rhcloud.com]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "--silent", "--tlsv1.2", "--cacert", "/etc/origin/master/ca-bundle.crt", "https://qe-jialiu1-khis-lb-1.1027-31k.qe.rhcloud.com:8443/healthz/ready"], "delta": "0:00:00.012922", "end": "2017-10-26 22:51:39.902043", "failed": true, "msg": "non-zero return code", "rc": 7, "start": "2017-10-26 22:51:39.889121", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Go to master, master api service is running well.
Go to haproxy lb host, haproxy service is NOT started.
# service haproxy status
Redirecting to /bin/systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/haproxy.service.d
           └─limits.conf
   Active: failed (Result: exit-code) since Thu 2017-10-26 23:13:20 EDT; 3s ago
  Process: 16854 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 16854 (code=exited, status=1/FAILURE)

Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: Starting HAProxy Load Balancer...
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : parsing [/etc/haproxy/haproxy.cfg:37] : 'listen' cannot handle unexpected a... ':9000'.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : parsing [/etc/haproxy/haproxy.cfg:37] : please use the 'bind' keyword for l...ddresses.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : Fatal errors found in configuration.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: haproxy-systemd-wrapper: exit, haproxy RC=1
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: Unit haproxy.service entered failed state.
Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: haproxy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.



Expected results:
haproxy lb should be installed successfully.

Additional info:
1. downgrade haproxy to haproxy-1.5.18-6.el7.x86_64, restart haproxy successfully.
2. update haproxy to haproxy-1.7.9-1.el7.x86_64 again, haproxy still failed, then removing the following lines from /etc/haproxy/haproxy.cfg:

#listen stats :9000
#    mode http
#    stats enable
#    stats uri /

Restart haproxy, this time it succeed. So seem like the new haproxy does NOT support the old syntax.

And this haproxy rpm is installed from OCP puddle.

Comment 1 Scott Dodson 2017-10-27 15:22:35 UTC
Workaround, downgrade to haproxy 1.5.x.

long term we should configure the installer to either force 1.5 or we need to lay down configuration that's specific to 1.7 or 1.5 based on the version installed.

Comment 3 Ben Bennett 2017-10-27 16:01:00 UTC
If we change the config to:

listen stats
    bind :9000
    mode http
    stats enable
    stats uri /

It should work with 1.5 and 1.7.  BUT the larger issue of the RPM being being named haproxy and conflicting still applies.

Comment 8 Gaoyun Pei 2018-01-22 09:04:18 UTC
So this issue is blocking containerized haproxy service start now.

[root@ip-172-18-7-218 ~]# docker images
REPOSITORY                                                         TAG                 IMAGE ID            CREATED             SIZE
registry.x.x.com:443/openshift3/ose-haproxy-router   v3.9.0              5ce9aed6c36c        2 days ago          1.243 GB

[root@ip-172-18-7-218 ~]# docker run --entrypoint rpm registry.x.x.com:443/openshift3/ose-haproxy-router:v3.9.0 -qa |grep haproxy
haproxy18-1.8.1-5.el7.x86_64



Trigger a containerized ha-master 3.9 cluster, haproxy service failed to start on containerized lb host.

[root@ip-172-18-7-218 ~]# journalctl --no-pager -u haproxy.service

Jan 22 02:17:11 ip-172-18-7-218.ec2.internal systemd[1]: Starting haproxy.service...
Jan 22 02:17:11 ip-172-18-7-218.ec2.internal docker[5745]: Error response from daemon: No such container: openshift_loadbalancer
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : parsing [/etc/haproxy/haproxy.cfg:30] : 'listen' cannot handle unexpected argument ':9000'.
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : parsing [/etc/haproxy/haproxy.cfg:30] : please use the 'bind' keyword for listening addresses.
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : Fatal errors found in configuration.
Jan 22 02:17:14 ip-172-18-7-218.ec2.internal systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE

[root@ip-172-18-7-218 ~]# grep -n ".*" /etc/haproxy/haproxy.cfg
1:# Global settings
2:#---------------------------------------------------------------------
...
30:listen stats :9000
31:    mode http
32:    stats enable
33:    stats uri /

Comment 9 Scott Dodson 2018-01-22 13:29:45 UTC
Ben,

Is the fix in https://bugzilla.redhat.com/show_bug.cgi?id=1506866#c3 still valid now that we're on haproxy 1.8?

Comment 10 Ben Bennett 2018-01-23 16:04:54 UTC
@scott: Yes... but the current haproxy.conf has the fix in it... where is the one for that container coming from?

Comment 11 Scott Dodson 2018-01-23 18:15:06 UTC
This is an openshift-ansible specific problem, we re-use the haproxy image during containerized installs to serve as an API server load balancer and we generate our own config.

we just need to update https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_loadbalancer/templates/haproxy.cfg.j2 with the changes suggested in comment 3.

Comment 12 Russell Teague 2018-01-23 18:30:22 UTC
Proposed: https://github.com/openshift/openshift-ansible/pull/6839

Comment 13 Russell Teague 2018-01-24 18:03:35 UTC
Merged

Comment 14 Gaoyun Pei 2018-01-25 09:12:19 UTC
As PR#6839 was already merged, and it should be included in openshift-ansible 3.9.0-0.24.0+ version, so gave it a try on master branch.

$ git describe
openshift-ansible-3.9.0-0.24.0-8-g2487fa8


Containerized haproxy service could run well. 

-bash-4.2# docker run --entrypoint rpm openshift3/ose-haproxy-router:v3.9.0 -qa |grep haproxy
haproxy18-1.8.1-5.el7.x86_64

-bash-4.2# sed -n '30,35p' /etc/haproxy/haproxy.cfg
listen stats
    bind :9000
    mode http
    stats enable
    stats uri /

Comment 15 Gaoyun Pei 2018-01-26 07:29:05 UTC
Move this bug to verified with openshift-ansible-3.9.0-0.24.0.git.0.735690f.el7.noarch.rpm according to Comment 14.

Comment 18 errata-xmlrpc 2018-03-28 14:08:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.