Red Hat Bugzilla – Bug 1468113
metrics 3.6, metricsPublicURL is only configured to first master with ansible metrics installer
Last modified: 2017-08-16 15:51 EDT
Created attachment 1294828 [details] ansible inventory file Description of problem: Same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1465275 metricsPublicURL is configured to first master only with HA masters setup Version-Release number of selected component (if applicable): # openshift version openshift v3.6.135 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 # rpm -qa | grep openshift-ansible openshift-ansible-playbooks-3.6.135-1.git.0.5533fe3.el7.noarch openshift-ansible-callback-plugins-3.6.135-1.git.0.5533fe3.el7.noarch openshift-ansible-roles-3.6.135-1.git.0.5533fe3.el7.noarch openshift-ansible-docs-3.6.135-1.git.0.5533fe3.el7.noarch openshift-ansible-filter-plugins-3.6.135-1.git.0.5533fe3.el7.noarch openshift-ansible-lookup-plugins-3.6.135-1.git.0.5533fe3.el7.noarch openshift-ansible-3.6.135-1.git.0.5533fe3.el7.noarch How reproducible: Always Steps to Reproduce: 1. Deploy metrics 3.6 on HA env. ansible-playbook -vvv -i ${inventory_file} playbooks/byo/openshift-cluster/openshift-metrics.yml 2. 3. Actual results: metricsPublicURL is configured to first master only Expected results: metricsPublicURL should be configured to all masters Additional info: Attached ansible inventory file and running log Description of problem: Version-Release number of the following components: rpm -q openshift-ansible rpm -q ansible ansible --version How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
Created attachment 1294829 [details] ansible running log
Issue is not fixed, it is changed to ON_QA automatically by add this defect to advisory by Errata System. Testing env: # openshift version openshift v3.6.140 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 # rpm -qa | grep openshift-ansible openshift-ansible-callback-plugins-3.6.140-1.git.0.4a02427.el7.noarch openshift-ansible-playbooks-3.6.140-1.git.0.4a02427.el7.noarch openshift-ansible-3.6.140-1.git.0.4a02427.el7.noarch openshift-ansible-lookup-plugins-3.6.140-1.git.0.4a02427.el7.noarch openshift-ansible-roles-3.6.140-1.git.0.4a02427.el7.noarch openshift-ansible-docs-3.6.140-1.git.0.4a02427.el7.noarch openshift-ansible-filter-plugins-3.6.140-1.git.0.4a02427.el7.noarch
The code change is here. It is not merged as of this morning. https://github.com/openshift/openshift-ansible/pull/4701
The changes have merged now.
I found a strange phenomenon, sometimes it only came to step "Adding metrics route information to metricsPublicURL", and then the ansible work stopped without [openshift_metrics : restart master], [openshift_metrics : restart master api],[openshift_metrics : restart master controllers] steps, the metrics route could not be accessed, and showed "Application is not available" error on https://hawkular-metrics.${SUB_DOAMIN}/hawkular/metrics page. Attached the ansible running log which is this scenario Most of time, the ansible showed logs about restart master,master api and master controllers,metrics route could be accessed
Created attachment 1297426 [details] ansible log, which don't have log about restart master, master api, master controller
Created attachment 1297429 [details] ansible log, it have log about restart master, master api, master controller
run the ansible script by using the following commands # cd /usr/share/ansible/openshift-ansible/ # ansible-playbook -vvv -i ${INVENTORY_FILE} playbooks/byo/openshift-cluster/openshift-metrics.yml
Unless we're sure that the playbooks are doing something wrong I think we should treat that as a router error. Does the problem eventually go away with no further action via ansible?
Tested today, used the following commands, the strange issue in Comment 7 was not happen today, could find [openshift_metrics : restart master], [openshift_metrics : restart master api],[openshift_metrics : restart master controllers] steps, and metricsPublicURL were configured to all masters. # cd /usr/share/ansible/openshift-ansible/ # ansible-playbook -vvv -i ${INVENTORY_FILE} playbooks/byo/openshift-cluster/openshift-metrics.yml openshift-ansible packages # rpm -qa | grep openshift-ansible openshift-ansible-3.6.153-1.git.0.5a6bf7d.el7.noarch openshift-ansible-roles-3.6.153-1.git.0.5a6bf7d.el7.noarch openshift-ansible-docs-3.6.153-1.git.0.5a6bf7d.el7.noarch openshift-ansible-callback-plugins-3.6.153-1.git.0.5a6bf7d.el7.noarch openshift-ansible-lookup-plugins-3.6.153-1.git.0.5a6bf7d.el7.noarch openshift-ansible-playbooks-3.6.153-1.git.0.5a6bf7d.el7.noarch openshift-ansible-filter-plugins-3.6.153-1.git.0.5a6bf7d.el7.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716