Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1687944

Summary:	keepalived container fails to restart during minor upgrade, hanging docker and stalling upgrade.
Product:	Red Hat OpenStack	Reporter:	Alistair Tonner <atonner>
Component:	openstack-containers	Assignee:	Jose Luis Franco <jfrancoa>
Status:	CLOSED DUPLICATE	QA Contact:	Marius Cornea <mcornea>
Severity:	high	Docs Contact:	Andrew Burden <aburden>
Priority:	high
Version:	14.0 (Rocky)	CC:	bperkins, jfrancoa, lbezdick, m.andre, mburns, sgolovat
Target Milestone:	---	Keywords:	Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-03 08:55:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alistair Tonner 2019-03-12 17:43:11 UTC

Description of problem:
 During minor update of OSP 14 on RHEL7.6 during pre-upgrade tasks keepalived goes down and does not restart.  Docker hangs.  Upgrade stalls and times out.


Version-Release number of selected component (if applicable):
Installed repositories (rhel-7.6):
  14
  ceph-3
  ceph-osd-3
  rhel-7.6
docker-1.13.1-91.git07f3374.el7.x86_64
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 290a33602b16ff2d1cc5339bc0297f0e094462ce (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)

192.168.24.1:8787/rhosp14/openstack-keepalived                   2019-03-06.1        d2f2329685cf        8 days ago          535 MB



How reproducible:

Build RHOS14 undercloud:1,controller:3,compute:1,ceph:3,ironic:2 and run a minor update against the stack.


Steps to Reproduce:

Actual results:

ar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Unexpected argument(s): | tee /var/log/keepalived.log
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Stopped
Mar 12 10:58:44 undercloud-0.redhat.local Keepalived[38986]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Unexpected argument(s): | tee /var/log/keepalived.log
Mar 12 10:58:44 undercloud-0.redhat.local Keepalived_vrrp[39033]: Stopped
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: time="2019-03-12T10:58:44.164609694-04:00" level=debug msg="attach: stderr: begin"
 


Expected results:

Update completes normally

Additional info:

Comment 1 Jose Luis Franco 2019-03-18 14:13:13 UTC

Having a look at the docker version this looks to me very similar to the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1671861 . Could we have more logs please? Was there a  restart of the docker service? if that could be confirmed then we can ensure this is a duplicate of 1671861.

Comment 2 Jose Luis Franco 2019-03-18 14:14:44 UTC

A good way to verify if we're facing the same issue is to login in some of the overcloud controller nodes and perform a "docker ps", if the command hangs then it's exactly the same issue described in 1671861.

Comment 3 Lukas Bezdicka 2019-05-03 08:55:13 UTC


*** This bug has been marked as a duplicate of bug 1671861 ***