Bug 1391837

Summary: incorrect lbversion check
Product: OpenShift Container Platform Reporter: Jan Provaznik <jprovazn>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED ERRATA QA Contact: Gan Huang <ghuang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: agoldste, aos-bugs, ghuang, jeder, jialiu, jokerman, jprovazn, mmccomas, tdawson
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: aos-scalability-34
Fixed In Version: Doc Type: Bug Fix
Doc Text:
This bug fixes an issue on the OpenShift master when using the Openstack cloud provider. The LBaaS version check was done improperly leading to failures when using v2 of the LBaaS plugin. This fix corrects the check so that v2 is detected properly.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-18 12:49:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Provaznik 2016-11-04 08:14:39 UTC
Description of problem:
On OpenStack instances, configure master to enable cloud provider, master can not be started because of error 'Config error: unrecognised lb-version "v2"'


Version-Release number of selected component (if applicable):
atomic-openshift-3.4.0.16-1.git.0.cc70b72.el7.x86_64
cockpit-kubernetes-0.114-2.el7.x86_64

Error logs:
/usr/bin/openshift start master controllers --config=/etc/origin/master/master-config.yaml --loglevel=10 --listen=https://0.0.0.0:8444 fails with:

I1104 03:46:11.044004   92992 round_trippers.go:318] GET https://test-ocp34-master-0.example.com:8443/api/v1/watch/pods?fieldSelector=status.phase%21%3DPending%2Cstatus.phase%21%3DRunning%2Cstatus.phase%21%3DUnknown&resourceVersion=7&timeoutSeconds=551 200 OK in 34 milliseconds
I1104 03:46:11.044019   92992 round_trippers.go:324] Response Headers:
I1104 03:46:11.044024   92992 round_trippers.go:327]     Cache-Control: no-store
I1104 03:46:11.044028   92992 round_trippers.go:327]     Content-Type: application/vnd.kubernetes.protobuf;stream=watch
I1104 03:46:11.044038   92992 round_trippers.go:327]     Date: Fri, 04 Nov 2016 07:46:11 GMT
I1104 03:46:11.118536   92992 openstack.go:410] Using LBaaS extension v2
I1104 03:46:11.118547   92992 openstack.go:413] Claiming to support LoadBalancer
W1104 03:46:11.118552   92992 openstack.go:420] Config error: unrecognised lb-version "v2"
F1104 03:46:11.118558   92992 master.go:453] Unable to start service controller: the cloud provider does not support external load balancers.


Additional info:
In this block in pkg/cloudprovider/providers/openstack/openstack.go:

	if os.lbOpts.LBVersion == "v2" {
		return &LbaasV2{LoadBalancer{network, compute, os.lbOpts}}, true
	} else if lbversion == "v1" {
		return &LbaasV1{LoadBalancer{network, compute, os.lbOpts}}, true
	} else {
		glog.Warningf("Config error: unrecognised lb-version \"%v\"", lbversion)
		return nil, false
	}

the check:
if os.lbOpts.LBVersion == "v2"

should be probably replaced with:
if lbversion == "v2"

Comment 1 Andy Goldstein 2016-11-07 15:03:40 UTC
https://github.com/openshift/origin/pull/11789

Comment 2 Seth Jennings 2016-11-07 15:28:46 UTC
Worth noting the severity of this bug (i.e. it is fatal to the master) is mitigated by https://bugzilla.redhat.com/show_bug.cgi?id=1389205

Kube PR:
https://github.com/kubernetes/kubernetes/pull/36249

Origin cherry pick PR:
https://github.com/openshift/origin/pull/11789

Comment 3 Jan Provaznik 2016-11-07 16:25:07 UTC
Thanks for the quick fix Seth.

BZ 1389205 is fatal for openstack envs where LBaaS is not available, in opposite this BZ is fatal for openstack envs where LBaaS v2 is available (supposing LBaaSv1 is deprecated, it would be majority), so AIUI fixing of 1389205 has no impact on severity of this bug.

Comment 4 Seth Jennings 2016-11-07 16:40:09 UTC
The call chain is

MasterConfig RunServiceLoadBalancerController()
ServiceController.New()
ServiceController.init()
cloudprovider.LoadBalancer()

PR:
https://github.com/openshift/origin/pull/11648/files

Before the PR, if LoadBalancer() returned an error, it would result in a glog.Fatalf() in RunServiceLoadBalancerController().

After the PR, LoadBalancer() failure only does glog.Errorf() and prevents the start of the service controller, rather than killing the whole master.

In other words it prevents this from being fatal:

F1104 03:46:11.118558   92992 master.go:453] Unable to start service controller: the cloud provider does not support external load balancers.

Comment 5 Jan Provaznik 2016-11-07 16:52:10 UTC
Ah, right, thanks.

Comment 6 Troy Dawson 2016-11-09 19:50:03 UTC
This has been merged into ose and is in OSE v3.4.0.24 or newer.

Comment 8 Johnny Liu 2016-11-10 02:55:22 UTC
@Jan, because QE do not have a openstack with LbaasV2 installed, we only have a OSP9 + LbaasV1, could you help verify this bug?

Comment 9 Jan Provaznik 2016-11-10 08:09:44 UTC
Unfortunately I don't have an env with LBaasV2 at the moment too, we hit this when we manually enabled LBaaSv2 in the scale lab as a workaround for BZ 1389205 but disabled it again when BZ 1389205 was fixed.

Comment 10 Johnny Liu 2016-11-10 09:05:49 UTC
Is it possible to enable LBaasV2 on OSP9, if yes, could you tell me how to enable it?

Comment 11 Jan Provaznik 2016-11-14 07:55:31 UTC
Supposing neutron-lbaas packages are available in OSP9 then yes (I think these are avail, but I can not confirm - I used OSP10). You can follow http://docs.openstack.org/mitaka/networking-guide/config-lbaas.html

Comment 12 Gan Huang 2016-11-16 08:06:03 UTC
Testing in OSP9 which enables LBaaSv2.(cloudprovider is enabled in OpenShift)

Verified with openshift v3.4.0.26

1.Reproduced with 

openshift v3.4.0.16+cc70b72
kubernetes v1.4.0+776c994

Master can't start due to "getsockopt: connection refused".

Start the atomic-openshift-master service manually, there's a similar error threw out like Jan's report:

#/usr/bin/openshift start master --config=/etc/origin/master/master-config.yaml --loglevel=10
I1116 02:56:19.056420    3650 openstack.go:410] Using LBaaS extension v2
I1116 02:56:19.056440    3650 openstack.go:413] Claiming to support LoadBalancer
W1116 02:56:19.056448    3650 openstack.go:420] Config error: unrecognised lb-version "v2"
F1116 02:56:19.056457    3650 master.go:453] Unable to start service controller: the cloud provider does not support external load balancers.

2. Trigger a fresh installation with 

openshift v3.4.0.26+f7e109e
kubernetes v1.4.0+776c994

Installation succeed, Master and Node services both are working well, S2I build succeed.

Move to verified.

Comment 14 errata-xmlrpc 2017-01-18 12:49:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066