Bug 1441977

Summary: [UPDATES] Update of mod_ssl package prevents haproxy from starting
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-tripleo-heat-templatesAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 11.0 (Ocata)CC: augol, bperkins, jcoufal, jschluet, lbezdick, mandreou, mburns, mcornea, rhel-osp-director-maint, sathlang
Target Milestone: rcKeywords: Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-6.0.0-7.el7ost puppet-tripleo-6.3.0-11.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1441982 (view as bug list) Environment:
Last Closed: 2017-05-17 20:22:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1441982, 1446289, 1446292, 1446293, 1450825    

Description Yurii Prokulevych 2017-04-13 08:46:44 UTC
Description of problem:
-----------------------
Minor update of RHOS-11 fails cause haproxy is not running.
Looks like mod_ssl package is updated and pulls in  /etc/httpd/conf.d/ssl.conf, which has 'Listen 443' directive uncommented.
Apache gets restarted, binds to port and causes haproxy to fail:

Apr 13 08:27:26 controller-0.localdomain systemd[1]: Started Cluster Controlled haproxy.
Apr 13 08:27:26 controller-0.localdomain systemd[1]: Starting Cluster Controlled haproxy...
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: [WARNING] 102/082726 (168083) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy aodh started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy ceilometer started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy cinder started.
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: [ALERT] 102/082726 (168083) : Starting proxy horizon: cannot bind socket [2620:52:0:13b8:5054:ff:fe3e:1:443]
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: [ALERT] 102/082726 (168083) : Starting proxy horizon: cannot bind socket [fd00:fd00:fd00:2000::16:443]
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy glance_api started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy gnocchi started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy haproxy.stats started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy heat_api started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy heat_cfn started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy heat_cloudwatch started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy keystone_admin started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy keystone_public started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy mysql started.
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: haproxy-systemd-wrapper: exit, haproxy RC=1
Apr 13 08:27:26 controller-0.localdomain systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Apr 13 08:27:26 controller-0.localdomain systemd[1]: Unit haproxy.service entered failed state.
Apr 13 08:27:26 controller-0.localdomain systemd[1]: haproxy.service failed.

ss -anp | grep 443
u_dgr  UNCONN     0      0      /run/systemd/cgroups-agent 1443                  * 0                   users:(("systemd",pid=1,fd=23))
tcp    LISTEN     0      128      :::443                  :::*                   users:(("httpd",pid=172103,fd=13),("httpd",pid=172102,fd=13),("httpd",pid=172101,fd=13),("httpd",pid=172100,fd=13),("httpd",pid=172099,fd=13),("httpd",pid=172098,fd=13),("httpd",pid=172097,fd=13),("httpd",pid=172096,fd=13),("httpd",pid=172050,fd=13))
tcp    SYN-SENT   0      1       fd00:fd00:fd00:2000::21:44350               fd00:fd00:fd00:2000::16:3306                users:(("neutron-server",pid=140764,fd=41))


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
mod_ssl-2.4.6-45.el7_3.4.x86_64
openstack-tripleo-heat-templates-6.0.0-4.el7ost.noarch

Steps to Reproduce:
-------------------
1. Deploy RHOS-11 (2017-03-30.4)
2. Setup repos(2017-04-12.4)
3. Update UC
4. Try update OC

Comment 2 Lukas Bezdicka 2017-04-13 10:28:53 UTC
mod_ssl should not be present and if yet it should have its ssl.conf file changed but not removed. Most likely image issue.

Comment 3 Marios Andreou 2017-04-13 12:45:52 UTC
o/ folks, fyi my OSP10/11 *upgrade* environment definitely has mod_ssl package on all the overcloud nodes (mod_ssl-2.4.6-45.el7_3.4.x86_64 - which was updated as part of today's upgrade 10->11). Doesn't tell us much but I also tried a pcs resource restart haproxy to see if comment #0 reproduces in this upgraded env and it does not.

So are we sure mod_ssl is the issue here? From comment #0 I see that there was some problem starting the horizon proxy as 443 is occupied. Can we get the journal/logs from the node (assuming this must be one of the controllers, and likely the first controller to be updated)? 

Even if 'yum remove mod_ssl' fixes this are we really sure we don't need that package (do we know enough about what it is what if you *do* have ssl in your env is it not needed?)

Comment 4 Lukas Bezdicka 2017-04-13 14:16:28 UTC
/var/lib/heat-config/deployed/b5c7d010-4a88-43f5-86f2-b9d44f2b9d12.notify.json:\u001b[0;36mDebug: /Stage[main]/Apache/File[/etc/httpd/conf.d/ssl.conf]: Removing existing file for replacement with absent\u001b[0m

Comment 5 Lukas Bezdicka 2017-04-13 17:27:52 UTC
Second commit will be touch in tripleo-heat-templates

Comment 10 Yurii Prokulevych 2017-05-04 11:25:27 UTC
Verified with openstack-tripleo-heat-templates-6.0.0-10.el7ost.noarch and puppet-tripleo-6.3.0-12.el7ost.noarch

Comment 11 errata-xmlrpc 2017-05-17 20:22:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245