Bug 1549454 - Etcd scale-up failed when running as system container on RHEL
Summary: Etcd scale-up failed when running as system container on RHEL
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.9.0
Assignee: Vadim Rutkovsky
QA Contact: Gaoyun Pei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-27 06:57 UTC by Gaoyun Pei
Modified: 2018-06-18 18:14 UTC (History)
5 users (show)

Fixed In Version: openshift-ansible-3.9.2-1.git.0.1a855b3.el7.noarch
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-18 18:14:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2013 normal SHIPPED_LIVE Important: OpenShift Container Platform 3.9 security, bug fix, and enhancement update 2018-06-27 22:01:43 UTC

Description Gaoyun Pei 2018-02-27 06:57:06 UTC
Description of problem:
Etcd scale-up playbook failed when trying to check cluster healthy on the first new etcd host:

TASK [Verify cluster is stable] ************************************************
FAILED - RETRYING: Verify cluster is stable (3 retries left).
FAILED - RETRYING: Verify cluster is stable (2 retries left).
FAILED - RETRYING: Verify cluster is stable (1 retries left).
fatal: [qe-gpei-auto-atqd-rhel-1.0227-r5a.qe.rhcloud.com]: FAILED! => {"attempts": 3, "changed": false, "cmd": "/usr/bin/etcdctl --cert-file /etc/etcd/peer.crt --key-file /etc/etcd/peer.key --ca-file /etc/etcd/ca.crt -C https://qe-gpei-auto-anwg-master-etcd-1:2379 cluster-health", "msg": "[Errno 2] No such file or directory", "rc": 2}

No etcd package installed on etcd hosts, so no etcdctl command available.


Version-Release number of the following components:
openshift-ansible-3.9.0-0.53.0.git.0.f8f01ef.el7.noarch.rpm
ansible-2.4.3.0-1.el7ae.noarch

How reproducible:
Always

Steps to Reproduce:
1.Add two new etcd hosts under [new_etcd] group, run etcd scale-up playbook
ansible-playbook playbooks/openshift-etcd/scaleup.yml -v


Actual results:

Expected results:

Additional info:
The full output will be attached.

Comment 2 Vadim Rutkovsky 2018-02-28 12:08:19 UTC
etcd playbooks use `etcdctl` when the installation is not atomic, I'll prepare a fix

Comment 3 Vadim Rutkovsky 2018-03-01 12:32:17 UTC
Created https://github.com/openshift/openshift-ansible/pull/7340

Comment 4 Vadim Rutkovsky 2018-03-05 11:03:17 UTC
Fix is available in openshift-ansible-3.9.2-1.git.0.1a855b3.el7

Comment 5 Gaoyun Pei 2018-03-06 03:11:56 UTC
Verify this bug with openshift-ansible-3.9.2-1.git.0.1a855b3.el7.noarch.rpm

Verify cluster step passed during the scale-up. 

TASK [Verify cluster is stable] ************************************************
changed: [qe-gpei-auto-yunr-rhel-1.0305-t81.qe.rhcloud.com] => {"attempts": 1, "changed": true, "cmd": ["runc", "exec", "etcd", "etcdctl", "--cert-file", "/etc/etcd/peer.crt", "--key-file", "/etc/etcd/peer.key", "--ca-file", "/etc/etcd/ca.crt", "-C", "https://qe-gpei-auto-atfs-master-etcd-1:2379", "cluster-health"], "delta": "0:00:00.088975", "end": "2018-03-05 10:24:28.002649", "failed": false, "rc": 0, "start": "2018-03-05 10:24:27.913674", "stderr": "", "stderr_lines": [], "stdout": "member 3b2cb16416a55304 is healthy: got healthy result from https://10.240.0.215:2379\nmember eff658dbd0dd1181 is healthy: got healthy result from https://10.240.0.213:2379\ncluster is healthy", "stdout_lines": ["member 3b2cb16416a55304 is healthy: got healthy result from https://10.240.0.215:2379", "member eff658dbd0dd1181 is healthy: got healthy result from https://10.240.0.213:2379", "cluster is healthy"]}


Note You need to log in before you can comment on or make changes to this bug.