Bug 1533783
| Summary: | Waiting for ES to be ready always fail when redeploy logging | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Anping Li <anli> |
| Component: | Logging | Assignee: | ewolinet |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Anping Li <anli> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.9.0 | CC: | anli, aos-bugs, ewolinet, jcantril, rmeggins, smunilla |
| Target Milestone: | --- | Keywords: | TestBlocker |
| Target Release: | 3.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openshift-ansible-3.9.0-0.19.0.git.0.de168fd.el7.noarch | Doc Type: | No Doc Update |
| Doc Text: |
As part of an enhancement we no longer use a config trigger for ES and instead rollout in a handler. The handler was watching the old ES pod to see if the ES cluster was green/yellow
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-06-13 14:42:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Logging redeploy/upgrade testing are blocked The bug is fixed in openshift-ansible-3.9.0-0.19.0.git.0.de168fd.el7.noarch |
Description of problem: The task "Waiting for ES to be ready" failed for running command in the deleted pod. It should run command in the new deployed pod. For example: Before redeploy, the elasticsearch pod name is logging-es-data-master-7lrz5fma-5-qrml6 During redeploy, oc rollout latest deleted logging-es-data-master-7lrz5fma-5-qrml6 and start a new pod logging-es-data-master-7lrz5fma-6-5wblt. But the playbook is still use the pod logging-es-data-master-7lrz5fma-5-qrml6. Version-Release number of selected component (if applicable): openshift-ansible-3.9.0-0.16.0.git.0.9f19afc.el7.noarch How reproducible: always Steps to Reproduce: 1. Deploy logging openshift_logging_install_logging=true openshift_logging_es_cluster_size=3 openshift_logging_es_memory_limit=1Gi openshift_logging_es_ops_memory_limit=1Gi openshift_logging_use_ops=true openshift_logging_es_ops_cluster_size=3 2. check the pod list logging-es-data-master-61e2sm7z-1-cls5n 2/2 Running 0 3h logging-es-data-master-7lrz5fma-5-qrml6 2/2 Running 0 8m logging-es-data-master-jqky6qen-1-k66bn 2/2 Running 0 3h 3. Redpeloy logging again. openshift_logging_install_logging=true openshift_logging_es_cluster_size=3 openshift_logging_es_memory_limit=1Gi openshift_logging_es_ops_memory_limit=1Gi openshift_logging_use_ops=true openshift_logging_es_ops_cluster_size=3 openshift_logging_es_number_of_replicas=3 openshift_logging_es_number_of_shards=3 4. Get pod list again [root@openshift-181 ~]# oc get pods logging-es-data-master-61e2sm7z-1-cls5n 2/2 Running 0 4h logging-es-data-master-7lrz5fma-6-5wblt 2/2 Running 0 18m logging-es-data-master-jqky6qen-1-k66bn 2/2 Running 0 4h Actual results: RUNNING HANDLER [openshift_logging_elasticsearch : command] ******************************************************************************************************************************************************* changed: [openshift-181.lab.eng.nay.redhat.com] => {"changed": true, "cmd": ["oc", "get", "pods", "-l", "deploymentconfig=logging-es-data-master-7lrz5fma", "-n", "logging", "-o", "jsonpath={.items[*].metadata.name}"], "delta": "0:00:00.331319", "end": "2018-01-12 02:09:51.158892", "failed": false, "rc": 0, "start": "2018-01-12 02:09:50.827573", "stderr": "", "stderr_lines": [], "stdout": "logging-es-data-master-7lrz5fma-5-qrml6", "stdout_lines": ["logging-es-data-master-7lrz5fma-5-qrml6"]} RUNNING HANDLER [openshift_logging_elasticsearch : command] ******************************************************************************************************************************************************* FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (60 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (59 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (58 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (57 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (56 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (55 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (54 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (53 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (52 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (51 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (50 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (49 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (48 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (47 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (46 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (45 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (44 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (43 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (42 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (41 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (40 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (39 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (38 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (37 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (36 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (35 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (34 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (33 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (32 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (31 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (30 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (29 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (28 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (27 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (26 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (25 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (24 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (23 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (22 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (21 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (20 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (19 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (18 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (17 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (16 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (15 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (14 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (13 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (12 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (11 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (10 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (9 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (8 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (7 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (6 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (5 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (4 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (3 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (2 retries left). FAILED - RETRYING: Waiting for ES to be ready for logging-es-data-master-7lrz5fma (1 retries left). failed: [openshift-181.lab.eng.nay.redhat.com] (item=logging-es-data-master-7lrz5fma-5-qrml6) => {"_pod": "logging-es-data-master-7lrz5fma-5-qrml6", "attempts": 60, "changed": false, "cmd": "oc exec \"logging-es-data-master-7lrz5fma-5-qrml6\" -c elasticsearch -n \"logging\" -- curl -s --cacert /etc/elasticsearch/secret/admin-ca --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/_cat/health | cut -d' ' -f4", "delta": "0:00:00.287207", "end": "2018-01-12 02:15:35.871834", "failed": true, "rc": 0, "start": "2018-01-12 02:15:35.584627", "stderr": "Error from server (NotFound): pods \"logging-es-data-master-7lrz5fma-5-qrml6\" not found", "stderr_lines": ["Error from server (NotFound): pods \"logging-es-data-master-7lrz5fma-5-qrml6\" not found"], "stdout": "", "stdout_lines": []} included: /usr/share/ansible/openshift-ansible/roles/openshift_logging_elasticsearch/tasks/restart_es_node.yml for openshift-181.lab.eng.nay.redhat.com RUNNING HANDLER [openshift_logging_elasticsearch : command] ******************************************************************************************************************************************************* RUNNING HANDLER [openshift_logging_elasticsearch : oc_obj] ******************************************************************************************************************************************************** RUNNING HANDLER [openshift_logging_elasticsearch : command] ******************************************************************************************************************************************************* Expected results: Redeploy succeed. Waiting for ES to be ready should run health checking command in new pod logging-es-data-master-7lrz5fma-6-5wblt. Additional info: