Bug 1420684 - [intservice_public_324] Upgrade logging stack from 3.4.1 to 3.5.0 failed due to take the wrong namespace by ansible
Summary: [intservice_public_324] Upgrade logging stack from 3.4.1 to 3.5.0 failed due ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.5.z
Assignee: Jeff Cantrill
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1416210
TreeView+ depends on / blocked
 
Reported: 2017-02-09 10:10 UTC by Junqi Zhao
Modified: 2017-12-14 21:01 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-12-14 21:01:20 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3438 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-15 01:58:11 UTC

Description Junqi Zhao 2017-02-09 10:10:14 UTC
Description of problem:
After deploying logging 3.4.1 successfully by using deployer pod and then upgrade to logging 3.5.0 via ansible, the upgrade is failed due to take the wrong namespace. In this defect, my namespace is juzhao, but ansible take "/tmp/openshift-logging-ansible-x00dCB/admin.kubeconfig" as namespace


Version-Release number of selected component (if applicable):
# openshift version
openshift v3.5.0.18+9a5d1aa
kubernetes v1.5.2+43a9be4
etcd 3.1.0


Image id:
openshift3/logging-elasticsearch    d715f4d34ad4
openshift3/logging-kibana    e0ab09c2cbeb
openshift3/logging-fluentd    47057624ecab
openshift3/logging-auth-proxy    139f7943475e
openshift3/logging-curator    7f034fdf7702

How reproducible:
Always


Steps to Reproduce:
1. Deploy logging 3.4.1 first, make sure all the pods are running
and kibana UI can be accessed.

2. prepare the inventory file

[oo_first_master]
$master-public-dns ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="~/cfile/libra.pem" openshift_public_hostname=$master-public-dns

[oo_first_master:vars]
deployment_type=openshift-enterprise
openshift_release=v3.5.0
openshift_logging_install_logging=false
openshift_logging_upgrade_logging=true

openshift_logging_kibana_hostname=kibana.$sub-domain
public_master_url=https://$master-public-dns:8443
openshift_logging_fluentd_hosts=$node

openshift_logging_image_prefix=registry.ops.openshift.com/openshift3/
openshift_logging_image_version=3.5.0

openshift_logging_namespace=juzhao
openshift_logging_use_ops=true
openshift_logging_es_cluster_size=1

3. Running the playbook from a control machine (my laptop) which is not oo_master:
git clone https://github.com/openshift/openshift-ansible
ansible-playbook -vvv -i ~/inventory   playbooks/common/openshift-cluster/openshift_logging.yml

Actual results:
TASK [openshift_logging : start elasticsearch] *********************************
task path: /home/fedora/openshift-ansible/roles/openshift_logging/tasks/upgrade_logging.yaml:16
Using module file /home/fedora/openshift-ansible/roles/lib_openshift/library/oc_scale.py
<ec2-54-86-165-237.compute-1.amazonaws.com> ESTABLISH SSH CONNECTION FOR USER: root
<ec2-54-86-165-237.compute-1.amazonaws.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/fedora/Downloads/libra-new.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r ec2-54-86-165-237.compute-1.amazonaws.com '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
failed: [ec2-54-86-165-237.compute-1.amazonaws.com] (item=deploymentconfig/logging-es-56xhpu2b) => {
    "failed": true, 
    "invocation": {
        "module_args": {
            "debug": false, 
            "kind": "dc", 
            "kubeconfig": "/etc/origin/master/admin.kubeconfig", 
            "name": "logging-es-56xhpu2b", 
            "namespace": "/tmp/openshift-logging-ansible-x00dCB/admin.kubeconfig", 
            "replicas": 1, 
            "state": "present"
        }, 
        "module_name": "oc_scale"
    }, 
    "object": "deploymentconfig/logging-es-56xhpu2b"
}

MSG:

{u'cmd': u'oc -n /tmp/openshift-logging-ansible-x00dCB/admin.kubeconfig get dc logging-es-56xhpu2b -o json', u'returncode': 1, u'results': [{}], u'stderr': u'error: invalid namespace "/tmp/openshift-logging-ansible-x00dCB/admin.kubeconfig": [may not contain \'/\']\n', u'stdout': u''}


PLAY RECAP *********************************************************************
ec2-54-86-165-237.compute-1.amazonaws.com : ok=253  changed=35   unreachable=0    failed=1  


Expected results:
Upgrade can be successful.

Additional info:

Comment 1 Jeff Cantrill 2017-02-09 15:47:51 UTC
fixed in: https://github.com/openshift/openshift-ansible/pull/3310

Comment 2 openshift-github-bot 2017-02-09 19:20:55 UTC
Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/fa1b4494a40218e173a54a10f3269fb0c7c39baa
bug 1420684. On logging upgrade use the correct value for namespace

https://github.com/openshift/openshift-ansible/commit/8ee9a7bfe9901db69e5d2ee1d118ed0f7ce81005
Merge pull request #3310 from jcantrill/bz_1420684_fix_namespace_on_logging_upgrade

bug 1420684. On logging upgrade use the correct value for namespace

Comment 3 Junqi Zhao 2017-02-10 08:17:47 UTC
blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1418911

Comment 4 Junqi Zhao 2017-02-10 08:19:13 UTC
(In reply to Junqi Zhao from comment #3)
> blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1418911

correct:
blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1421011

Comment 5 Junqi Zhao 2017-02-13 09:10:33 UTC
although "503 Service Unavailable" shows in Kibana UI after upgrade, but ansible can use the right namespace now.

# openshift version
openshift v3.5.0.19+199197c
kubernetes v1.5.2+43a9be4
etcd 3.1.0


Image id:
logging-elasticsearch    d715f4d34ad4
logging-kibana    e0ab09c2cbeb
logging-fluentd    47057624ecab
logging-auth-proxy    139f7943475e
logging-curator    7f034fdf7702


set it to VERIFIED and close it.

Comment 8 errata-xmlrpc 2017-12-14 21:01:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438


Note You need to log in before you can comment on or make changes to this bug.