Bug 1687944 - keepalived container fails to restart during minor upgrade, hanging docker and stalling upgrade.
Summary: keepalived container fails to restart during minor upgrade, hanging docker an...
Keywords:
Status: CLOSED DUPLICATE of bug 1671861
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Jose Luis Franco
QA Contact: Marius Cornea
Andrew Burden
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-12 17:43 UTC by Alistair Tonner
Modified: 2019-11-01 18:11 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-03 08:55:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Alistair Tonner 2019-03-12 17:43:11 UTC 
Description of problem:
 During minor update of OSP 14 on RHEL7.6 during pre-upgrade tasks keepalived goes down and does not restart.  Docker hangs.  Upgrade stalls and times out.


Version-Release number of selected component (if applicable):
Installed repositories (rhel-7.6):
  14
  ceph-3
  ceph-osd-3
  rhel-7.6
docker-1.13.1-91.git07f3374.el7.x86_64
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 290a33602b16ff2d1cc5339bc0297f0e094462ce (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)

192.168.24.1:8787/rhosp14/openstack-keepalived                   2019-03-06.1        d2f2329685cf        8 days ago          535 MB



How reproducible:

Build RHOS14 undercloud:1,controller:3,compute:1,ceph:3,ironic:2 and run a minor update against the stack.


Steps to Reproduce:

Actual results:

ar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Unexpected argument(s): | tee /var/log/keepalived.log
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Stopped
Mar 12 10:58:44 undercloud-0.redhat.local Keepalived[38986]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: Unexpected argument(s): | tee /var/log/keepalived.log
Mar 12 10:58:44 undercloud-0.redhat.local Keepalived_vrrp[39033]: Stopped
Mar 12 10:58:44 undercloud-0.redhat.local dockerd-current[516710]: time="2019-03-12T10:58:44.164609694-04:00" level=debug msg="attach: stderr: begin"
 


Expected results:

Update completes normally

Additional info:
Comment 1 Jose Luis Franco 2019-03-18 14:13:13 UTC 
Having a look at the docker version this looks to me very similar to the issue described in https://bugzilla.redhat.com/show_bug.cgi?id=1671861 . Could we have more logs please? Was there a  restart of the docker service? if that could be confirmed then we can ensure this is a duplicate of 1671861.
Comment 2 Jose Luis Franco 2019-03-18 14:14:44 UTC 
A good way to verify if we're facing the same issue is to login in some of the overcloud controller nodes and perform a "docker ps", if the command hangs then it's exactly the same issue described in 1671861.
Comment 3 Lukas Bezdicka 2019-05-03 08:55:13 UTC 

*** This bug has been marked as a duplicate of bug 1671861 ***
Note You need to log in before you can comment on or make changes to this bug.