Bug 1383233

Summary: [ocp-on-osp] Stack failed at task "Deploy Openshift IP failover for router"
Product: OpenShift Container Platform Reporter: Gan Huang <ghuang>
Component: InstallerAssignee: Jan Provaznik <jprovazn>
Status: CLOSED CURRENTRELEASE QA Contact: Gan Huang <ghuang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.3.0CC: aos-bugs, jokerman, jprovazn, mlamouri, mmccomas, sbaubeau
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-20 08:39:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Gan Huang 2016-10-10 09:15:40 UTC
Description of problem:
When creating heat stack which selects IP failover mode for OpenShift router, ansible-playbook would failed at task "Deploy Openshift IP failover for router". This looks like to be caused by a existing bug in OCP 3.3: https://bugzilla.redhat.com/show_bug.cgi?id=1332432

Version-Release number of selected component (if applicable):
openshift-on-openstack v0.9.1

# oc version
oc v3.3.0.34
kubernetes v1.3.0+52492b4

openshift v3.3.0.34
kubernetes v1.3.0+52492b4


How reproducible:
100%

Steps to Reproduce:
1.Create a heat stack which is OCP3.3 + IP failover mode for router
2.
3.

Actual results:

Heat stack was failed.

Check the ansible log on bastion host:
TASK [Deploy Openshift IP failover for router] *********************************
task path: /var/lib/ansible/playbooks/ipfailover.yml:13
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<ghuang6-neutron-lb-openshift-master-0.example.com> ESTABLISH SSH CONNECTION FOR USER: cloud-user
<ghuang6-neutron-lb-openshift-master-0.example.com> SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=cloud-user -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/%h-%r ghuang6-neutron-lb-openshift-master-0.example.com '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-cbaufsurovkgobuxxsndnuxwaydehrez; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
fatal: [ghuang6-neutron-lb-openshift-master-0.example.com]: FAILED! => {
    "changed": true, 
    "cmd": [
        "oadm", 
        "ipfailover", 
        "--create", 
        "--service-account=ipfailover", 
        "--interface=eth0", 
        "--selector=region=infra", 
        "--replicas=2", 
        "--virtual-ips=192.168.0.4", 
        "--credentials=/etc/origin/master/openshift-router.kubeconfig"
    ], 
    "delta": "0:00:00.237775", 
    "end": "2016-10-10 02:20:02.918713", 
    "failed": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oadm ipfailover --create --service-account=ipfailover --interface=eth0 --selector='region=infra' --replicas=2 --virtual-ips=\"192.168.0.4\" --credentials=/etc/origin/master/openshift-router.kubeconfig", 
            "_uses_shell": false, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "warn": true
        }, 
        "module_name": "command"
    }, 
    "rc": 1, 
    "start": "2016-10-10 02:20:02.680938", 
    "stderr": "Flag --credentials has been deprecated, use --service-account to specify the service account the ipfailover pod will use to make API calls\n    error: serviceaccounts \"ipfailover\" already exists", 
    "stdout": "--> Creating IP failover ipfailover ...\n    deploymentconfig \"ipfailover\" created\n--> Failed", 
    "stdout_lines": [
        "--> Creating IP failover ipfailover ...", 
        "    deploymentconfig \"ipfailover\" created", 
        "--> Failed"
    ], 
    "warnings": []
}
	to retry, use: --limit @/var/lib/ansible/playbooks/main.retry



Expected results:
Stack created successfully.

Additional info:

Comment 1 Sylvain Baubeau 2016-10-10 14:13:34 UTC
Should be fixed by https://github.com/redhat-openstack/openshift-on-openstack/pull/266

Comment 2 Jan Provaznik 2016-10-11 07:05:05 UTC
This is caused by OCP3.3 bug https://bugzilla.redhat.com/show_bug.cgi?id=1332432

A workaround for openshift-on-openstack is merged now:
https://github.com/redhat-openstack/openshift-on-openstack/commit/2cc2e9fb8e176fff96b6075df2635a31ff2f4684

Comment 3 Jan Provaznik 2016-10-12 12:21:38 UTC
fixed in 0.9.2

Comment 4 Gan Huang 2016-10-13 07:23:03 UTC
Verified with openshift-on-openstack v0.9.2 + OSP8 + OCP3.3

openshift v3.3.0.35
kubernetes v1.3.0+52492b4
etcd 2.3.0+git

IPfailover works well although the command returns "Failed".
    "rc": 1,
    "start": "2016-10-13 01:58:07.250624",
    "stderr": "Flag --credentials has been deprecated, use --service-account to specify the service account the ipfailover pod will use to make API calls\n    error: serviceaccounts \"ipfailover\" already exists",
    "stdout": "--> Creating IP failover ipfailover ...\n    deploymentconfig \"ipfailover\" created\n--> Failed",
    "stdout_lines": [
        "--> Creating IP failover ipfailover ...",
        "    deploymentconfig \"ipfailover\" created",
        "--> Failed"
    ],
    "warnings": []
}
...ignoring

Current workaround works, we need remove the workaround once BZ#1332432 is fixed.

So move it to verified currently.