Bug 1424981 - [IntService_public_324] Logging upgrade failed at TASK [openshift_logging : Applying /tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml]
Summary: [IntService_public_324] Logging upgrade failed at TASK [openshift_logging : A...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: ewolinet
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-20 09:07 UTC by Xia Zhao
Modified: 2017-07-24 14:11 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-04-12 19:02:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
ansible upgrade log (496.48 KB, text/plain)
2017-02-20 09:10 UTC, Xia Zhao
no flags Details
the inventory file used for logging upgrade (1.08 KB, text/plain)
2017-02-20 09:11 UTC, Xia Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0903 0 normal SHIPPED_LIVE OpenShift Container Platform atomic-openshift-utils bug fix and enhancement 2017-04-12 22:45:42 UTC

Description Xia Zhao 2017-02-20 09:07:13 UTC
Description of problem:
Upgrade logging from 3.3.1 to 3.5.0, failed at TASK [openshift_logging : Applying /tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml]:

TASK [openshift_logging : Applying /tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml] ***
task path: /root/openshift-ansible/roles/openshift_logging/tasks/oc_apply.yaml:13
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<$master> ESTABLISH SSH CONNECTION FOR USER: root
<$master> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/libra.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r $master '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
fatal: [$master]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "--config=/tmp/openshift-logging-ansible-1PQ3bY/admin.kubeconfig", 
        "apply", 
        "-f", 
        "/tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml", 
        "-n", 
        "logging"
    ], 
    "delta": "0:00:00.125620", 
    "end": "2017-02-20 03:48:33.265045", 
    "failed": true, 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oc --config=/tmp/openshift-logging-ansible-1PQ3bY/admin.kubeconfig apply -f /tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml -n logging", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2017-02-20 03:48:33.139425", 
    "warnings": []
}

STDERR:

error: unable to decode "/tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml": [pos 1595]: json: expect char '"' but got char 't'


PLAY RECAP *********************************************************************
$master : ok=149  changed=3    unreachable=0    failed=1   


Version-Release number of selected component (if applicable):
https://github.com/openshift/openshift-ansible -b master

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging 3.3.1 stacks (on OCP 3.5.0) with journald log driver enabled and node selectors defined in configmap:
"use-journal": "true"
"curator-nodeselector": "logging-infra-fluentd=true"
"es-nodeselector": "logging-infra-fluentd=true"
"kibana-nodeselector": "logging-infra-fluentd=true"

Also bound es with hostPV storage on es node, wait until log entries shown on kibana UI.

2.Upgrade to logging 3.5.0 stacks by using ansible, specifying these parameters in inventory file (as in the attachment):
openshift_logging_fluentd_use_journal=true

openshift_logging_es_nodeselector={'logging-infra-fluentd':'true'}
openshift_logging_kibana_nodeselector={'logging-infra-fluentd':'true'}
openshift_logging_curator_nodeselector={'logging-infra-fluentd':'true'}
openshift_logging_fluentd_nodeselector={'logging-infra-fluentd':'true'}

3.Check upgrade result

Actual results:
Upgrade failed 

Expected results:
Upgrade should be successful

Additional info:
Ansible upgrade log attached
inventory file for the upgrade attached
Repro env attached

Comment 1 Xia Zhao 2017-02-20 09:10:23 UTC
Created attachment 1255622 [details]
ansible upgrade log

Comment 2 Xia Zhao 2017-02-20 09:11:01 UTC
Created attachment 1255623 [details]
the inventory file used for logging upgrade

Comment 7 Junqi Zhao 2017-02-24 03:55:05 UTC
Verified on OCP 3.5 with the latest playbooks from openshift-ansible, this error does not happen now.

Set it to verified and close it.

Comment 9 errata-xmlrpc 2017-04-12 19:02:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903


Note You need to log in before you can comment on or make changes to this bug.