Description of problem: See the following detailed Version-Release number of the following components: 3.7/2017-10-26.4_v3.7.0-0.181.0 openshift-ansible-3.7.0-0.181.0.git.0.34f6e3e.el7.noarch How reproducible: Always Steps to Reproduce: 1. create ocp repo on haproxy lb host 2. trigger a multiple master rpm install with haproxy lb 3. Actual results: installation failed. TASK [openshift_master : Wait for API to become available] ********************* Friday 27 October 2017 02:47:22 +0000 (0:00:00.121) 0:13:12.100 ******** <--snip--> FAILED - RETRYING: Wait for API to become available (1 retries left). [WARNING]: Consider using get_url or uri module rather than running curl fatal: [qe-jialiu1-khis-master-etcd-1.1027-31k.qe.rhcloud.com]: FAILED! => {"attempts": 120, "changed": false, "cmd": ["curl", "--silent", "--tlsv1.2", "--cacert", "/etc/origin/master/ca-bundle.crt", "https://qe-jialiu1-khis-lb-1.1027-31k.qe.rhcloud.com:8443/healthz/ready"], "delta": "0:00:00.012922", "end": "2017-10-26 22:51:39.902043", "failed": true, "msg": "non-zero return code", "rc": 7, "start": "2017-10-26 22:51:39.889121", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Go to master, master api service is running well. Go to haproxy lb host, haproxy service is NOT started. # service haproxy status Redirecting to /bin/systemctl status haproxy.service ● haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/haproxy.service.d └─limits.conf Active: failed (Result: exit-code) since Thu 2017-10-26 23:13:20 EDT; 3s ago Process: 16854 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=1/FAILURE) Main PID: 16854 (code=exited, status=1/FAILURE) Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: Starting HAProxy Load Balancer... Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : parsing [/etc/haproxy/haproxy.cfg:37] : 'listen' cannot handle unexpected a... ':9000'. Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : parsing [/etc/haproxy/haproxy.cfg:37] : please use the 'bind' keyword for l...ddresses. Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: [ALERT] 298/231320 (16855) : Fatal errors found in configuration. Oct 26 23:13:20 qe-jialiu1-khis-lb-1 haproxy-systemd-wrapper[16854]: haproxy-systemd-wrapper: exit, haproxy RC=1 Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: Unit haproxy.service entered failed state. Oct 26 23:13:20 qe-jialiu1-khis-lb-1 systemd[1]: haproxy.service failed. Hint: Some lines were ellipsized, use -l to show in full. Expected results: haproxy lb should be installed successfully. Additional info: 1. downgrade haproxy to haproxy-1.5.18-6.el7.x86_64, restart haproxy successfully. 2. update haproxy to haproxy-1.7.9-1.el7.x86_64 again, haproxy still failed, then removing the following lines from /etc/haproxy/haproxy.cfg: #listen stats :9000 # mode http # stats enable # stats uri / Restart haproxy, this time it succeed. So seem like the new haproxy does NOT support the old syntax. And this haproxy rpm is installed from OCP puddle.
Workaround, downgrade to haproxy 1.5.x. long term we should configure the installer to either force 1.5 or we need to lay down configuration that's specific to 1.7 or 1.5 based on the version installed.
If we change the config to: listen stats bind :9000 mode http stats enable stats uri / It should work with 1.5 and 1.7. BUT the larger issue of the RPM being being named haproxy and conflicting still applies.
So this issue is blocking containerized haproxy service start now. [root@ip-172-18-7-218 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.x.x.com:443/openshift3/ose-haproxy-router v3.9.0 5ce9aed6c36c 2 days ago 1.243 GB [root@ip-172-18-7-218 ~]# docker run --entrypoint rpm registry.x.x.com:443/openshift3/ose-haproxy-router:v3.9.0 -qa |grep haproxy haproxy18-1.8.1-5.el7.x86_64 Trigger a containerized ha-master 3.9 cluster, haproxy service failed to start on containerized lb host. [root@ip-172-18-7-218 ~]# journalctl --no-pager -u haproxy.service Jan 22 02:17:11 ip-172-18-7-218.ec2.internal systemd[1]: Starting haproxy.service... Jan 22 02:17:11 ip-172-18-7-218.ec2.internal docker[5745]: Error response from daemon: No such container: openshift_loadbalancer Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : parsing [/etc/haproxy/haproxy.cfg:30] : 'listen' cannot handle unexpected argument ':9000'. Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : parsing [/etc/haproxy/haproxy.cfg:30] : please use the 'bind' keyword for listening addresses. Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg Jan 22 02:17:14 ip-172-18-7-218.ec2.internal docker[5750]: [ALERT] 021/071714 (1) : Fatal errors found in configuration. Jan 22 02:17:14 ip-172-18-7-218.ec2.internal systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE [root@ip-172-18-7-218 ~]# grep -n ".*" /etc/haproxy/haproxy.cfg 1:# Global settings 2:#--------------------------------------------------------------------- ... 30:listen stats :9000 31: mode http 32: stats enable 33: stats uri /
Ben, Is the fix in https://bugzilla.redhat.com/show_bug.cgi?id=1506866#c3 still valid now that we're on haproxy 1.8?
@scott: Yes... but the current haproxy.conf has the fix in it... where is the one for that container coming from?
This is an openshift-ansible specific problem, we re-use the haproxy image during containerized installs to serve as an API server load balancer and we generate our own config. we just need to update https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_loadbalancer/templates/haproxy.cfg.j2 with the changes suggested in comment 3.
Proposed: https://github.com/openshift/openshift-ansible/pull/6839
Merged
As PR#6839 was already merged, and it should be included in openshift-ansible 3.9.0-0.24.0+ version, so gave it a try on master branch. $ git describe openshift-ansible-3.9.0-0.24.0-8-g2487fa8 Containerized haproxy service could run well. -bash-4.2# docker run --entrypoint rpm openshift3/ose-haproxy-router:v3.9.0 -qa |grep haproxy haproxy18-1.8.1-5.el7.x86_64 -bash-4.2# sed -n '30,35p' /etc/haproxy/haproxy.cfg listen stats bind :9000 mode http stats enable stats uri /
Move this bug to verified with openshift-ansible-3.9.0-0.24.0.git.0.735690f.el7.noarch.rpm according to Comment 14.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489