Bug 1424981

Description of problem:
Upgrade logging from 3.3.1 to 3.5.0, failed at TASK [openshift_logging : Applying /tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml]:

TASK [openshift_logging : Applying /tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml] ***
task path: /root/openshift-ansible/roles/openshift_logging/tasks/oc_apply.yaml:13
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<$master> ESTABLISH SSH CONNECTION FOR USER: root
<$master> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/libra.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r $master '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
fatal: [$master]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "--config=/tmp/openshift-logging-ansible-1PQ3bY/admin.kubeconfig", 
        "apply", 
        "-f", 
        "/tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml", 
        "-n", 
        "logging"
    ], 
    "delta": "0:00:00.125620", 
    "end": "2017-02-20 03:48:33.265045", 
    "failed": true, 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oc --config=/tmp/openshift-logging-ansible-1PQ3bY/admin.kubeconfig apply -f /tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml -n logging", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2017-02-20 03:48:33.139425", 
    "warnings": []
}

STDERR:

error: unable to decode "/tmp/openshift-logging-ansible-1PQ3bY/templates/logging-curator-dc.yaml": [pos 1595]: json: expect char '"' but got char 't'


PLAY RECAP *********************************************************************
$master : ok=149  changed=3    unreachable=0    failed=1   


Version-Release number of selected component (if applicable):
https://github.com/openshift/openshift-ansible -b master

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging 3.3.1 stacks (on OCP 3.5.0) with journald log driver enabled and node selectors defined in configmap:
"use-journal": "true"
"curator-nodeselector": "logging-infra-fluentd=true"
"es-nodeselector": "logging-infra-fluentd=true"
"kibana-nodeselector": "logging-infra-fluentd=true"

Also bound es with hostPV storage on es node, wait until log entries shown on kibana UI.

2.Upgrade to logging 3.5.0 stacks by using ansible, specifying these parameters in inventory file (as in the attachment):
openshift_logging_fluentd_use_journal=true

openshift_logging_es_nodeselector={'logging-infra-fluentd':'true'}
openshift_logging_kibana_nodeselector={'logging-infra-fluentd':'true'}
openshift_logging_curator_nodeselector={'logging-infra-fluentd':'true'}
openshift_logging_fluentd_nodeselector={'logging-infra-fluentd':'true'}

3.Check upgrade result

Actual results:
Upgrade failed 

Expected results:
Upgrade should be successful

Additional info:
Ansible upgrade log attached
inventory file for the upgrade attached
Repro env attached