Bug 1616047
| Summary: | [3.11.0-0.14.0] logging installation using openshift ansible logging playbook fails. | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Siva Reddy <schituku> | ||||||
| Component: | Logging | Assignee: | ewolinet | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 3.11.0 | CC: | aos-bugs, ewolinet, jokerman, mmccomas, rmeggins, schituku | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 3.11.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||||
| Doc Text: |
As part of figuring out which nodes to create a sysctl for as part of installing an ES5 cluster, we weren't excluding etcd hosts, which may not be nodes and therefore would be missing a fact that we were using to evaluate if a node matched an ansible host in the inventory.
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-10-11 07:24:57 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Eric - any ideas? Can you provide the full output of running the logging playbook with -vvv? Created attachment 1476156 [details]
The ansible log with -vvv
The fact collection looks to have those structures in them... Siva, What does your inventory group section look like? ([masters], [nodes], etc)? Created attachment 1476167 [details]
inventory file
I think that task is failing when its evaluating the etcd group since it is not a node. If so then the fix would be to only iterate over the list of nodes, not groups['OSEv3']. Will try to recreate this locally. Eric, Yes that looks like the case. I made the etcd group empty and re-ran the playbook. it finished successfully. Siva, that's what I observed locally as well. The plan is to update the loop to only iterate over the list of node hosts; I will open up a PR with the fix afterwards. Verified this bug on
openshift v3.11.0-0.20.0
oc v3.11.0-0.19.0
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO
openshift-ansible git log
# cd openshift-ansible
# git log
commit d003bdf30c23a47f485f9f1f394a3d247c88da06
Merge: ac09fc2 921df2b
Author: OpenShift Merge Robot <openshift-merge-robot.github.com>
Date: Thu Aug 23 08:56:20 2018 -0700
The test was performed on a OCP cluster which had split etcd and master. And etcd was not a node. THe installation completed successfully.
# oc project
Using project "openshift-logging" on server
# oc get pods
NAME READY STATUS RESTARTS AGE
logging-es-data-master-lzw28btf-1-5xpt7 2/2 Running 0 6m
logging-fluentd-4lz76 1/1 Running 0 6m
logging-fluentd-gl8hm 1/1 Running 0 6m
logging-fluentd-n4dfw 1/1 Running 0 6m
logging-fluentd-qqn2j 1/1 Running 0 6m
logging-kibana-1-vc9v7 2/2 Running 0 7m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652 |
Description of problem: The install fails when trying to install logging on a 3.11 OCP cluster. Version-Release number of selected component (if applicable): oc v3.11.0-0.14.0 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO openshift v3.11.0-0.14.0 kubernetes v1.11.0+d4cacc0 How reproducible: Always Steps to Reproduce: 1. Install logging using openshift-ansible playbook. ansible-playbook -i inventory openshift-ansible/playbooks/openshift-logging/config.yml -- inventory file [OSEv3:vars] deployment_type=openshift-enterprise openshift_deployment_type=openshift-enterprise openshift_release=v3.11 openshift_docker_additional_registries=registry.reg-aws.openshift.com oreg_url=registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version} openshift_logging_image_version=v3.11 openshift_logging_install_logging=true openshift_logging_es_cluster_size=1 openshift_logging_es_pvc_dynamic=true openshift_logging_es_pvc_size=50Gi openshift_logging_es_pvc_storage_class_name=gp2 openshift_logging_fluentd_read_from_head=false openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"} openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"} openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"} openshift_logging_es_memory_limit=8Gi openshift_logging_install_eventrouter=true Actual results: The install fails at task TASK [Evaluate oo_elasticsearch_nodes] ************************************************************************************************************************************** skipping: [ec2-34-217-107-181.us-west-2.compute.amazonaws.com] => (item=ec2-34-217-107-181.us-west-2.compute.amazonaws.com) fatal: [ec2-34-217-107-181.us-west-2.compute.amazonaws.com]: FAILED! => {"msg": "The conditional check 'hostvars[item]['openshift']['common']['ip'] in openshift_logging_elas ticsearch_hosts' failed. The error was: error while evaluating conditional (hostvars[item]['openshift']['common']['ip'] in openshift_logging_elasticsearch_hosts): 'ansible.v ars.hostvars.HostVarsVars object' has no attribute 'openshift'\n\nThe error appears to have been in '/root/openshift-ansible/playbooks/openshift-logging/private/config.yml': line 82, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n # files for specific nodes based on <node>.status.addresses[@.type==InternalIP].address\n - name: Evaluate oo_elasticsearch_nodes\n ^ here\n"} Description of problem: Version-Release number of the following components: rpm -q openshift-ansible rpm -q ansible ansible --version How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag