Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1468113

Summary: metrics 3.6, metricsPublicURL is only configured to first master with ansible metrics installer
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: InstallerAssignee: Kenny Woodson <kwoodson>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, jokerman, mmccomas, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:29:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ansible inventory file
none
ansible running log
none
ansible log, which don't have log about restart master, master api, master controller
none
ansible log, it have log about restart master, master api, master controller none

Description Junqi Zhao 2017-07-06 06:06:47 UTC
Created attachment 1294828 [details]
ansible inventory file

Description of problem:
Same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1465275
metricsPublicURL is configured to first master only with HA masters setup

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.6.135
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

# rpm -qa | grep openshift-ansible
openshift-ansible-playbooks-3.6.135-1.git.0.5533fe3.el7.noarch
openshift-ansible-callback-plugins-3.6.135-1.git.0.5533fe3.el7.noarch
openshift-ansible-roles-3.6.135-1.git.0.5533fe3.el7.noarch
openshift-ansible-docs-3.6.135-1.git.0.5533fe3.el7.noarch
openshift-ansible-filter-plugins-3.6.135-1.git.0.5533fe3.el7.noarch
openshift-ansible-lookup-plugins-3.6.135-1.git.0.5533fe3.el7.noarch
openshift-ansible-3.6.135-1.git.0.5533fe3.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.6 on HA env.
ansible-playbook -vvv -i ${inventory_file} playbooks/byo/openshift-cluster/openshift-metrics.yml
2.
3.

Actual results:
metricsPublicURL is configured to first master only

Expected results:
metricsPublicURL should be configured to all masters

Additional info:
Attached ansible inventory file and running log

Description of problem:

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Junqi Zhao 2017-07-06 06:08:14 UTC
Created attachment 1294829 [details]
ansible running log

Comment 4 Junqi Zhao 2017-07-11 08:13:24 UTC
Issue is not fixed, it is changed to ON_QA automatically by add this defect to advisory by Errata System.

Testing env:
# openshift version
openshift v3.6.140
kubernetes v1.6.1+5115d708d7
etcd 3.2.1


# rpm -qa | grep openshift-ansible
openshift-ansible-callback-plugins-3.6.140-1.git.0.4a02427.el7.noarch
openshift-ansible-playbooks-3.6.140-1.git.0.4a02427.el7.noarch
openshift-ansible-3.6.140-1.git.0.4a02427.el7.noarch
openshift-ansible-lookup-plugins-3.6.140-1.git.0.4a02427.el7.noarch
openshift-ansible-roles-3.6.140-1.git.0.4a02427.el7.noarch
openshift-ansible-docs-3.6.140-1.git.0.4a02427.el7.noarch
openshift-ansible-filter-plugins-3.6.140-1.git.0.4a02427.el7.noarch

Comment 5 Kenny Woodson 2017-07-11 13:30:48 UTC
The code change is here.  It is not merged as of this morning.

https://github.com/openshift/openshift-ansible/pull/4701

Comment 6 Kenny Woodson 2017-07-11 13:39:30 UTC
The changes have merged now.

Comment 7 Junqi Zhao 2017-07-13 06:02:13 UTC
I found a strange phenomenon, sometimes it only came to step "Adding metrics route information to metricsPublicURL", and then the ansible work stopped without [openshift_metrics : restart master], [openshift_metrics : restart master api],[openshift_metrics : restart master controllers] steps, the metrics route could not be accessed, and showed "Application is not available" error on https://hawkular-metrics.${SUB_DOAMIN}/hawkular/metrics page.

Attached the ansible running log which is this scenario


Most of time, the ansible showed logs about restart master,master api and master controllers,metrics route could be accessed

Comment 8 Junqi Zhao 2017-07-13 06:03:30 UTC
Created attachment 1297426 [details]
ansible log, which don't have log about restart master, master api, master controller

Comment 9 Junqi Zhao 2017-07-13 06:05:47 UTC
Created attachment 1297429 [details]
ansible log, it have log about restart master, master api, master controller

Comment 10 Junqi Zhao 2017-07-13 06:11:37 UTC
run the ansible script by using the following commands
# cd /usr/share/ansible/openshift-ansible/
# ansible-playbook -vvv -i ${INVENTORY_FILE}  playbooks/byo/openshift-cluster/openshift-metrics.yml

Comment 12 Scott Dodson 2017-07-17 15:14:05 UTC
Unless we're sure that the playbooks are doing something wrong I think we should treat that as a router error. Does the problem eventually go away with no further action via ansible?

Comment 13 Junqi Zhao 2017-07-19 05:57:34 UTC
Tested today, used the following commands, the strange issue in Comment 7 was not happen today, could find [openshift_metrics : restart master], [openshift_metrics : restart master api],[openshift_metrics : restart master controllers] steps, and metricsPublicURL were configured to all masters.

# cd /usr/share/ansible/openshift-ansible/
# ansible-playbook -vvv -i ${INVENTORY_FILE}  playbooks/byo/openshift-cluster/openshift-metrics.yml

openshift-ansible packages
# rpm -qa | grep openshift-ansible
openshift-ansible-3.6.153-1.git.0.5a6bf7d.el7.noarch
openshift-ansible-roles-3.6.153-1.git.0.5a6bf7d.el7.noarch
openshift-ansible-docs-3.6.153-1.git.0.5a6bf7d.el7.noarch
openshift-ansible-callback-plugins-3.6.153-1.git.0.5a6bf7d.el7.noarch
openshift-ansible-lookup-plugins-3.6.153-1.git.0.5a6bf7d.el7.noarch
openshift-ansible-playbooks-3.6.153-1.git.0.5a6bf7d.el7.noarch
openshift-ansible-filter-plugins-3.6.153-1.git.0.5a6bf7d.el7.noarch

Comment 15 errata-xmlrpc 2017-08-10 05:29:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716