Bug 1848454
| Summary: | OpenShift logging upgrade from 3.11.161 to 3.11.219 fails | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Devendra Kulkarni <dkulkarn> | ||||
| Component: | Logging | Assignee: | Sergey Yedrikov <syedriko> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 3.11.0 | CC: | aos-bugs, einar.martinius.furuholmen, erich, ikarpukh, jharding, mburke, periklis, syedriko | ||||
| Target Milestone: | --- | Keywords: | DeliveryBlocker | ||||
| Target Release: | 3.11.z | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Linux | ||||||
| Whiteboard: | logging-core | ||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
Cause:
Missing check for the matching number of Elasticsearch DCs, PVCs and indices.
Consequence:
Ansible pads missing elements with Nones leading to cryptic Cluster Logging playbook failure.
Fix:
Added the missing check.
Result:
Cluster logging playbook issues a diagnostic if the number of Elasticsearch DCs, PVCs and indices do not match.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-10-22 11:02:22 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Please provide: * The version of ansible * The version of openshift-ansible Hello @Jeff, # rpm -qa | grep openshift-ansible openshift-ansible-roles-3.11.219-1.git.0.8845382.el7.noarch openshift-ansible-playbooks-3.11.219-1.git.0.8845382.el7.noarch openshift-ansible-3.11.219-1.git.0.8845382.el7.noarch openshift-ansible-docs-3.11.219-1.git.0.8845382.el7.noarch #rpm -qa | grep ^ansible ansible-2.9.6-1.el7ae.noarch Please confirm the attached inventory is the correct one. The content would indicate it's not to install logging at all: # Aggregated logging openshift_logging_install_logging=False Hello Jeff, I have already pointed out this to the customer and they are using extra vars (openshift_logging_install_logging=True) while running the playbook. Regards, Devendra Kulkarni Put UpcomingSprint label, not likely to land this until EOS. Moving to UpcomingSprint *** Bug 1842608 has been marked as a duplicate of this bug. *** Moving to UpcomingSprint for future evaluation Moving to UpcomingSprint awaiting for PRs to merge, etc. The upgrade failed when ES DC count = ES PVC count.
Sunday 11 October 2020 15:53:05 +0000 (0:00:00.148) 0:01:04.922 ********
fatal: [ec2-54-89-6-220.compute-1.amazonaws.com]: FAILED! => {
"changed": false,
"msg": "There must be the same number of ES DeploymentConfigs, ES PVCs and ES indices. Found ES DeploymentConfigs - \"[u'logging-es-data-master-5cg1if76', u'logging-es-data-master-2i7wgnxz', u'logging-es-data-master-r1ijyoz5', u'logging-es-data-master-v3pbddal']\", ES DC count - \"4\", ES PVCs - \"{}\", ES PVC length - \"0\" and ES indices - \"[0, 1, 2, 3]\", ES indices length - \"4\""
}
#ES PVC length =0 when pvc number =4
$oc get dc
NAME REVISION DESIRED CURRENT TRIGGERED BY
logging-es-data-master-2i7wgnxz 3 1 1
logging-es-data-master-5cg1if76 3 1 1
logging-es-data-master-r1ijyoz5 3 1 1
logging-es-data-master-v3pbddal 3 1 1
logging-kibana 2 1 1 config
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
es-0 Bound logginges1 10G RWO 57m
es-1 Bound logginges2 10G RWO 56m
es-2 Bound logginges0 10G RWO 55m
es-3 Bound logginges3 10G RWO 55m
Created attachment 1720679 [details]
Playbook logs
#######Logging-Variables##########
openshift_logging_install_logging=true
openshift_logging_es_cluster_size=4
openshift_logging_es_number_of_replicas=0
openshift_logging_es_allows_cluster_reader=True
openshift_logging_es_nodeselector={"role": "node"}
openshift_logging_elasticsearch_storage_type=pvc
openshift_logging_es_pvc_size=5Gi
#openshift_logging_es_pvc_storage_class_name=''
openshift_logging_es_pvc_dynamic=true
openshift_logging_es_pvc_prefix=es
openshift_logging_es_pv_selector={'logging-infra':'es'}
openshift_logging_es_memory_limit=2Gi
The PR pull in new issue. The upgrade always failed using the new code. Workaround: Please don't use openshift-ansible:v3.11.306. use openshift-ansible:v3.11.286 if you need to upgrade logging. @Sergey, openshift_logging_es_pvc_prefix=logging-es pass. Only hit comment 18 when we use customized pvc prefix logging_es_pvc_prefix=es. Revert the PR as it cause regression. the support case behind BZ 1848454 got closed, there's no customer waiting for that fix atm. so close it. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.306 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4170 |
Description of problem: Openshift Logging upgrade fails from 3.11.161 to 3.11.219 on the ansible task: ~~~ TASK [openshift_logging_elasticsearch : set_fact] ******************************************************************** fatal: [svvtocp1mastr01.vegvesen.no]: FAILED! => {"msg": "The conditional check 'openshift_logging_elasticsearch_deployment_name == \"\"' failed. The error was: error while evaluating conditional (openshift_logging_elasticsearch_deployment_name == \"\"): 'None' has no attribute 'name'\n\nThe error appears to be in '/usr/share/ansible/openshift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml': line 470, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- set_fact:\n ^ here\n"} ******************************************************************** ~~~ - Logging stack is currently running fine with 3.11.161 version. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Playbook fails at the TASK:[openshift_logging_elasticsearch : set_fact] Expected results: Upgrade should be successful. Additional info: