Bug 1441915

Temporary workaround that should be reverted as soon as we fix this bug :

Add a group named [oo_first_master] with your first master and then run playbooks/common/openshift-cluster/openshift_metrics.yml

Proposed: https://github.com/openshift/openshift-ansible/pull/3948
Will need to backport this to 3.5.

Merged https://github.com/openshift/openshift-ansible/pull/3948

Russell,

I cloned the repo "openshift-ansible" to local test server, then use following inventory for metrics:
[root@host-8-175-224 ~]# cat inventory_metrics 
[OSEv3:children]
masters

[masters]
host-8-175-206.host.centralci.eng.rdu2.redhat.com openshift_public_hostname=host-8-175-206.host.centralci.eng.rdu2.redhat.com

[OSEv3:vars]
ansible_ssh_user=root
ansible_ssh_private_key_file="~/libra.pem"
deployment_type=openshift-enterprise

openshift_metrics_install_metrics=true
openshift_metrics_hawkular_hostname=hawkular-metrics.0418-bzl.qe.rhcloud.com
openshift_metrics_project=openshift-infra

openshift_metrics_image_prefix=registry.ops.openshift.com/openshift3/
openshift_metrics_image_version=3.5.0

openshift_metrics_cassandra_storage_type=emptydir

Then setup metrics by:
# ansible-playbook -i /root/inventory_metrics openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml

Th setup seems works well, but the cassandra pod finally failed to start with the status "CrashLoopBackOff".
You can find the detail logs "logs_1441915_CrashLoopBackOff.txt" from the attachment.
Do you know what's wrong cause the failure?

Thanks,
Nicholas

Created attachment 1272461 [details]
logs for cassandra pod failed to start.

Second issue I found is if I use the template mentioned in OCP 3.5 doc "https://docs.openshift.com/container-platform/3.5/install_config/cluster_metrics.html#deploying-the-metrics-components", it will failed, see:
# ansible-playbook -i /root/inventory_metrics openshift-ansible/playbooks/common/openshift-cluster/openshift_metrics.yml

PLAY [Populate config host groups] *********************************************

TASK [fail] ********************************************************************
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true
}

MSG:

This playbook requires g_etcd_hosts to be set

	to retry, use: --limit @/root/openshift-ansible/playbooks/common/openshift-cluster/openshift_metrics.retry

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1   

So should we update the doc for deploy metrics like:
FROM:
$ ansible-playbook <OPENSHIFT_ANSIBLE_DIR>/common/openshift-cluster/openshift_metrics.yml \
   -e openshift_metrics_install_metrics=True \
   -e openshift_metrics_hawkular_hostname=hawkular-metrics.example.com \
   -e openshift_metrics_cassandra_storage_type=pv
TO:
$ ansible-playbook <OPENSHIFT_ANSIBLE_DIR>/byo/openshift-cluster/openshift-metrics.yml \
   -e openshift_metrics_install_metrics=True \
   -e openshift_metrics_hawkular_hostname=hawkular-metrics.example.com \
   -e openshift_metrics_cassandra_storage_type=pv
OR TO(to keep consistency with deploy logging):
$ ansible-playbook [-i </path/to/inventory>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml

If yes, I will open another PR for the doc changes.

Nicholas,
The first issue with Cassandra appears to be unrelated to the change in entry point playbooks.  If that problem continues open another issue.

You are correct about the second issue in the docs.  The correct playbooks to run are from playbooks/byo/openshift-cluster.

Proposed fix for 3.5 (release-1.5 branch) https://github.com/openshift/openshift-ansible/pull/3953
This PR is different from the commits to master due to the playbook refactoring which has happened on master for 3.6.

Liming,
  FYI, the correct playbook is under the byo directory, the QE automation has this 

https://github.com/openshift/cucushift/blob/v3/features/step_definitions/logging_metrics.rb#L280

Russell,
  Looks like there is an issue wit the byo playbook for metrics as it is complaining about the errors with the ansible generated cert 

 <openshift-127.lab.sjc.redhat.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/tmp/tmp/libra.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/opt/app-root/src/.ansible/cp/ansible-ssh-%h-%p-%r -tt openshift-127.lab.sjc.redhat.com '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1492629881.83-254965767611179/command.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1492629881.83-254965767611179/" > /dev/null 2>&1 && sleep 0'"'"''
      fatal: [openshift-127.lab.sjc.redhat.com]: FAILED! => {
          "changed": true,
          "cmd": [
              "oadm",
              "ca",
              "create-server-cert",
              "--config=/tmp/openshift-metrics-ansible-HDBPf7/admin.kubeconfig",
              "--key=/tmp/openshift-metrics-ansible-HDBPf7/heapster.key",
              "--cert=/tmp/openshift-metrics-ansible-HDBPf7/heapster.cert",
              "--hostnames=heapster",
              "--signer-cert=/tmp/openshift-metrics-ansible-HDBPf7/ca.crt",
              "--signer-key=/tmp/openshift-metrics-ansible-HDBPf7/ca.key",
              "--signer-serial=/tmp/openshift-metrics-ansible-HDBPf7/ca.serial.txt"
          ],
          "delta": "0:00:00.260178",
          "end": "2017-04-19 15:24:42.338261",
          "failed": true,
          "invocation": {
              "module_args": {
                  "_raw_params": "oadm ca create-server-cert --config=/tmp/openshift-metrics-ansible-HDBPf7/admin.kubeconfig --key='/tmp/openshift-metrics-ansible-HDBPf7/heapster.key' --cert='/tmp/openshift-metrics-ansible-HDBPf7/heapster.cert' --hostnames=heapster --signer-cert='/tmp/openshift-metrics-ansible-HDBPf7/ca.crt' --signer-key='/tmp/openshift-metrics-ansible-HDBPf7/ca.key' --signer-serial='/tmp/openshift-metrics-ansible-HDBPf7/ca.serial.txt'",
                  "_uses_shell": false,
                  "chdir": null,
                  "creates": null,
                  "executable": null,
                  "removes": null,
                  "warn": true
              },
              "module_name": "command"
          },
          "rc": 1,
          "start": "2017-04-19 15:24:42.078083",
          "stderr": "error: --signer-cert, \"/tmp/openshift-metrics-ansible-HDBPf7/ca.crt\" must be a valid certificate file\nSee 'oadm ca create-server-cert -h' for help and examples.",
          "stdout": "",
          "stdout_lines": [],
          "warnings": []
      }
        to retry, use: --limit @/tmp/tmp/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.retry
 
      PLAY RECAP *********************************************************************
      localhost                  : ok=1    changed=0    unreachable=0    failed=0
      openshift-127.lab.sjc.redhat.com : ok=28   changed=0    unreachable=0    failed=1

Peter,

What's the inventory look like? I cannot reproduce that.

Russell,

Got it, thanks.
I will do test again to see if the cassandra issues exist or not, will open BUG for it if still exist.

By the way, will the openshift doc update per this bug or should I open another BUG for the doc to update?

Thanks,
Nicholas

Russell,

About the first issue mentioned in this bug which the cassandra failed startup, Per the info in link "https://github.com/openshift/openshift-ansible/pull/3953", I ran "git clone https://github.com/openshift/openshift-ansible -b release-1.5" to clone to the code to local, then deploy metrics with template under byo:
# ansible-playbook -i inventory_metrics openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml 
All works well.

And I compare the file under branch "master" and branch "release-1.5", confirmed the updates in "release-1.5" still not exist in branch "master", and verified use the template file in branch "master" will cause the cassandra failed to start. 
So do you know when the update will push to branch "master"?

The test env version is here:
[root@host-8-175-108 ~]# openshift version
openshift v3.5.5.5
kubernetes v1.5.2+43a9be4
etcd 3.1.0
[root@host-8-175-108 ~]# rpm -qa|grep openshift-ansible
openshift-ansible-3.5.55-1.git.0.29802ee.el7.noarch
openshift-ansible-roles-3.5.55-1.git.0.29802ee.el7.noarch
openshift-ansible-docs-3.5.55-1.git.0.29802ee.el7.noarch
openshift-ansible-lookup-plugins-3.5.55-1.git.0.29802ee.el7.noarch
openshift-ansible-callback-plugins-3.5.55-1.git.0.29802ee.el7.noarch
openshift-ansible-playbooks-3.5.55-1.git.0.29802ee.el7.noarch
openshift-ansible-filter-plugins-3.5.55-1.git.0.29802ee.el7.noarch

Thanks,
Nicholas

@Russell, we should build a 3.5 openshift-ansible rpm package to include that fix, and attach the rpm to advisory, then move this bug to ON_QA, is that our work flow, right?

@Russell,

I changed the status to "MODIFIED", when the rpm ready, you change to "ON_QA" so we can start to do test with rpm pkgs.

Thanks,
Nicholas

@Scott,
  I've filed another bug https://bugzilla.redhat.com/show_bug.cgi?id=1443741 since it seems to be inventory file related.

The patch for #3953 is in 3.5.56.

$ git tag --contains 8ba61987ca972c8fbe186cd12ef04394a311c2dd
openshift-ansible-3.5.56-1
openshift-ansible-3.5.57-1

Russell,

During latest OCP testing, the openshift ansible version is 3.5.57-1, the test of deploy metrics with following command:
ansible-playbook -i ./ansible_inventory_file -v /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml 

The deploy works well, all the services and pods are up and running.
So this can verify the fix.
So I'd like to change the status to verified. Let me know if anything more need to do per this bug.

Thanks,
Nicholas

Russell,

I saw the 3.5 doc still use "playbooks/common":
https://docs.openshift.com/container-platform/3.5/install_config/cluster_metrics.html#deploying-the-metrics-components

So should I open another bug for the doc need to update to use "playbooks/byo" instead of "playbooks/common"?

Thanks,
Nicholas

Yep, docs should point to playbooks/byo.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1140