Bug 1446289

Summary:	[UPDATES] Update of mod_ssl package prevents haproxy from starting
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	openstack-tripleo-heat-templates	Assignee:	Marios Andreou <mandreou>
Status:	CLOSED NOTABUG	QA Contact:	Yurii Prokulevych <yprokule>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	9.0 (Mitaka)	CC:	achernet, bperkins, emacchi, jcoufal, lbezdick, mandreou, mburns, mcornea, rhel-osp-director-maint, sasha, sathlang, yprokule
Target Milestone:	async	Keywords:	Reopened, Triaged, ZStream
Target Release:	9.0 (Mitaka)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1441982	Environment:
Last Closed:	2017-05-09 13:34:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1441977, 1441982, 1446292, 1446293
Bug Blocks:

Description Marius Cornea 2017-04-27 15:20:14 UTC

This is the clone for OSP9.

+++ This bug was initially created as a clone of Bug #1441982 +++

+++ This bug was initially created as a clone of Bug #1441977 +++

Description of problem:
-----------------------
Minor update of RHOS-11 fails cause haproxy is not running.
Looks like mod_ssl package is updated and pulls in  /etc/httpd/conf.d/ssl.conf, which has 'Listen 443' directive uncommented.
Apache gets restarted, binds to port and causes haproxy to fail:

Apr 13 08:27:26 controller-0.localdomain systemd[1]: Started Cluster Controlled haproxy.
Apr 13 08:27:26 controller-0.localdomain systemd[1]: Starting Cluster Controlled haproxy...
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: [WARNING] 102/082726 (168083) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy aodh started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy ceilometer started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy cinder started.
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: [ALERT] 102/082726 (168083) : Starting proxy horizon: cannot bind socket [2620:52:0:13b8:5054:ff:fe3e:1:443]
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: [ALERT] 102/082726 (168083) : Starting proxy horizon: cannot bind socket [fd00:fd00:fd00:2000::16:443]
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy glance_api started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy gnocchi started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy haproxy.stats started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy heat_api started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy heat_cfn started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy heat_cloudwatch started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy keystone_admin started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy keystone_public started.
Apr 13 08:27:26 controller-0.localdomain haproxy[168083]: Proxy mysql started.
Apr 13 08:27:26 controller-0.localdomain haproxy-systemd-wrapper[168082]: haproxy-systemd-wrapper: exit, haproxy RC=1
Apr 13 08:27:26 controller-0.localdomain systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Apr 13 08:27:26 controller-0.localdomain systemd[1]: Unit haproxy.service entered failed state.
Apr 13 08:27:26 controller-0.localdomain systemd[1]: haproxy.service failed.

ss -anp | grep 443
u_dgr  UNCONN     0      0      /run/systemd/cgroups-agent 1443                  * 0                   users:(("systemd",pid=1,fd=23))
tcp    LISTEN     0      128      :::443                  :::*                   users:(("httpd",pid=172103,fd=13),("httpd",pid=172102,fd=13),("httpd",pid=172101,fd=13),("httpd",pid=172100,fd=13),("httpd",pid=172099,fd=13),("httpd",pid=172098,fd=13),("httpd",pid=172097,fd=13),("httpd",pid=172096,fd=13),("httpd",pid=172050,fd=13))
tcp    SYN-SENT   0      1       fd00:fd00:fd00:2000::21:44350               fd00:fd00:fd00:2000::16:3306                users:(("neutron-server",pid=140764,fd=41))


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
mod_ssl-2.4.6-45.el7_3.4.x86_64
openstack-tripleo-heat-templates-6.0.0-4.el7ost.noarch

Steps to Reproduce:
-------------------
1. Deploy RHOS-11 (2017-03-30.4)
2. Setup repos(2017-04-12.4)
3. Update UC
4. Try update OC

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-04-13 04:49:45 EDT ---

This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

--- Additional comment from Marius Cornea on 2017-04-13 04:50:42 EDT ---

The same issue happens with OSP10 minor update.

Comment 1 Red Hat Bugzilla Rules Engine 2017-04-27 15:20:30 UTC

This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 2 Sofer Athlan-Guyot 2017-04-28 13:03:53 UTC

This bug is triggered when we update mod_ssl which come with a new default configuration with Listen 443.

It happens when we yum upgrade on platform with mod_ssl installed.

But we had https://bugzilla.redhat.com/show_bug.cgi?id=1445886 which shows that mod_ssl is not in osp9 base image.

Bottom line, It would be nice to have confirmation that this bug is valid for osp9.

Comment 4 Sofer Athlan-Guyot 2017-05-02 10:26:28 UTC

Closing it as notabug as update is successful without the backport of the "mod_ssl patch"[1] as mod_ssl isn't installed.

[1] https://review.openstack.org/#/q/Ic5a0719f67d3795a9edca25284d1cf6f088073e8,n,z

Comment 5 Marios Andreou 2017-05-09 09:54:13 UTC

re-opening this BZ because of BZ 1448420 which is OSP9 ->10 upgrade - the environment there does have mod_ssl and is displaying the (98)Address already in use: AH00073: make_sock: unable to listen for connections on address [::]:443 as the ssl.conf does have a Listen 443.

I think we should backport the 'touch ssl.conf' workaround for mitaka OSP9 too what do others think? It should otherwise be harmless no?

Comment 6 Marios Andreou 2017-05-09 10:17:46 UTC

Update - may just close this again - looks like the  BZ 1448420  case is related indeed but we will need to do the touch ssl.conf workaround before installing mod_ssl during the 9-10 upgrade - see comment #8 at that BZ.

Comment 7 Marios Andreou 2017-05-09 13:34:57 UTC

moving back to closed as per comments at https://bugzilla.redhat.com/show_bug.cgi?id=1448420#c10 we can use that BZ to track the upgrade case. There should be no mod_ssl on OSP9 as discussed here already so we currently don't need to track something further here.