Red Hat Bugzilla – Bug 1549454
Etcd scale-up failed when running as system container on RHEL
Last modified: 2018-06-18 14:14:23 EDT
Description of problem: Etcd scale-up playbook failed when trying to check cluster healthy on the first new etcd host: TASK [Verify cluster is stable] ************************************************ FAILED - RETRYING: Verify cluster is stable (3 retries left). FAILED - RETRYING: Verify cluster is stable (2 retries left). FAILED - RETRYING: Verify cluster is stable (1 retries left). fatal: [qe-gpei-auto-atqd-rhel-1.0227-r5a.qe.rhcloud.com]: FAILED! => {"attempts": 3, "changed": false, "cmd": "/usr/bin/etcdctl --cert-file /etc/etcd/peer.crt --key-file /etc/etcd/peer.key --ca-file /etc/etcd/ca.crt -C https://qe-gpei-auto-anwg-master-etcd-1:2379 cluster-health", "msg": "[Errno 2] No such file or directory", "rc": 2} No etcd package installed on etcd hosts, so no etcdctl command available. Version-Release number of the following components: openshift-ansible-3.9.0-0.53.0.git.0.f8f01ef.el7.noarch.rpm ansible-2.4.3.0-1.el7ae.noarch How reproducible: Always Steps to Reproduce: 1.Add two new etcd hosts under [new_etcd] group, run etcd scale-up playbook ansible-playbook playbooks/openshift-etcd/scaleup.yml -v Actual results: Expected results: Additional info: The full output will be attached.
etcd playbooks use `etcdctl` when the installation is not atomic, I'll prepare a fix
Created https://github.com/openshift/openshift-ansible/pull/7340
Fix is available in openshift-ansible-3.9.2-1.git.0.1a855b3.el7
Verify this bug with openshift-ansible-3.9.2-1.git.0.1a855b3.el7.noarch.rpm Verify cluster step passed during the scale-up. TASK [Verify cluster is stable] ************************************************ changed: [qe-gpei-auto-yunr-rhel-1.0305-t81.qe.rhcloud.com] => {"attempts": 1, "changed": true, "cmd": ["runc", "exec", "etcd", "etcdctl", "--cert-file", "/etc/etcd/peer.crt", "--key-file", "/etc/etcd/peer.key", "--ca-file", "/etc/etcd/ca.crt", "-C", "https://qe-gpei-auto-atfs-master-etcd-1:2379", "cluster-health"], "delta": "0:00:00.088975", "end": "2018-03-05 10:24:28.002649", "failed": false, "rc": 0, "start": "2018-03-05 10:24:27.913674", "stderr": "", "stderr_lines": [], "stdout": "member 3b2cb16416a55304 is healthy: got healthy result from https://10.240.0.215:2379\nmember eff658dbd0dd1181 is healthy: got healthy result from https://10.240.0.213:2379\ncluster is healthy", "stdout_lines": ["member 3b2cb16416a55304 is healthy: got healthy result from https://10.240.0.215:2379", "member eff658dbd0dd1181 is healthy: got healthy result from https://10.240.0.213:2379", "cluster is healthy"]}