Description of problem: When including a new configuration option recently supported by the cloud provider openstack[1], e.g. "enabled = true" in the user managed config-map, the kube controller manager Pods will be restarting since they are not able to recognize the new config and fail with the following traceback: I0422 15:39:28.989335 1 leaderelection.go:258] successfully acquired lease kube-system/kube-controller-manager I0422 15:39:28.989780 1 event.go:294] "Event occurred" object="kube-system/kube-controller-manager" kind="ConfigMap" apiVersion="v1" type="Normal" reason="LeaderElection" message="ostest-28rzc-master-0_1b6b3575-ceca-452c-84cb-bccfaa bde936 became leader" I0422 15:39:28.989881 1 event.go:294] "Event occurred" object="kube-system/kube-controller-manager" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="ostest-28rzc-master-0_1b6b3575-ceca- 452c-84cb-bccfaabde936 became leader" F0422 15:39:34.692089 1 controllermanager.go:258] error building controller context: cloud provider could not be initialized: could not init cloud provider "openstack": warning: can't store data at section "LoadBalancer", variable "enabled" goroutine 295 [running]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0x1) /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x7de36e0, 0x3, 0x0, 0xc000259500, 0x0, {0x63defce, 0x1}, 0xc0013ce090, 0x0) /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printf(0x0, 0x0, 0x0, {0x0, 0x0}, {0x4ad7b94, 0x25}, {0xc0013ce090, 0x1, 0x1}) /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:753 +0x1c5 k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatalf(...) /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1532 k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2({0x520dbb0, 0xc000ec89c0}, 0xc00037eab0, 0x4c57fd0) /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:258 +0x1c8 k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func4({0x520dbb0, 0xc000ec89c0}) /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:320 +0xe3 created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:211 +0x154 [1] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#load-balancer Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Install a 4.11 cluster 2. Include a new option in the user facing config map with "oc edit cm cloud-provider-config -n openshift-config" 3. Actual results: Expected results: Additional info:
*** Bug 2078567 has been marked as a duplicate of this bug. ***
Adding comment from https://bugzilla.redhat.com/show_bug.cgi?id=2078567 here for additional context: As the openstack platform is migrating to the external cloud provider, the user-managed cloud.conf in `openshift-config/cloud-provider-config` is now the source for both the in-tree cloud.conf (in `openshift-config-managed/kube-cloud-config`) and the CCM cloud.conf (in `openshift-cloud-controller-manager/cloud-conf`). However, even after switching to CCM we need to keep initializing the in-tree cloud provider in order to be able to detach volumes created with legacy cloud provider. Moreover, the legacy cloud provider errors out when it finds an option it does not understand, causing KCM to enters a crashloopback. We can't legitimately ask customer to limit themselves to the intersection of in-tree and external cloud provider options. We should make the legacy cloud provider option parsing more tolerant and ignore unknown options.
Verified on 4.12.0-0.nightly-2022-10-05-053337 on top of RHOS-17.0-RHEL-9-20220909.n.0 $ oc get cm/cloud-provider-config -n openshift-config -o yaml apiVersion: v1 data: ca-bundle.pem: <hidden> config: | [Global] secret-name = openstack-credentials secret-namespace = kube-system region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem [LoadBalancer] floating-network-id=2e03985d-1eb4-4478-83b5-fdba6bda8ee0 lb-provider=ovn lb-method=SOURCE_IP_PORT [LoadBalancerClass "class1"] floating-network-id=17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet-id=3535d323-bb67-4e9a-92fe-fe85e8489245 [LoadBalancerClass "class2"] floating-network-id=17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet-tags="class2tag" [LoadBalancerClass "class3"] floating-network-id=17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet="external_subnet3" kind: ConfigMap metadata: creationTimestamp: "2022-10-06T14:16:55Z" name: cloud-provider-config namespace: openshift-config resourceVersion: "10815728" uid: 21a208be-0925-4816-88ff-e013f3a99c9d $ oc get cm/kube-cloud-config -n openshift-config-managed -o yaml apiVersion: v1 data: ca-bundle.pem: <hidden> cloud.conf: | [Global] secret-name = openstack-credentials secret-namespace = kube-system region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem [LoadBalancer] floating-network-id=2e03985d-1eb4-4478-83b5-fdba6bda8ee0 lb-provider=ovn lb-method=SOURCE_IP_PORT [LoadBalancerClass "class1"] floating-network-id=17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet-id=3535d323-bb67-4e9a-92fe-fe85e8489245 [LoadBalancerClass "class2"] floating-network-id=17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet-tags="class2tag" [LoadBalancerClass "class3"] floating-network-id=17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet="external_subnet3" kind: ConfigMap metadata: creationTimestamp: "2022-10-06T14:22:22Z" name: kube-cloud-config namespace: openshift-config-managed resourceVersion: "10815730" uid: 7a691240-c910-442d-9829-8875b1a0a01d $ oc get cm/cloud-conf -n openshift-cloud-controller-manager -o yaml apiVersion: v1 data: ca-bundle.pem: <hidden> cloud.conf: | [Global] region = regionOne ca-file = /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem use-clouds = true clouds-file = /etc/openstack/secret/clouds.yaml cloud = openstack [LoadBalancer] floating-network-id = 2e03985d-1eb4-4478-83b5-fdba6bda8ee0 lb-provider = ovn lb-method = SOURCE_IP_PORT use-octavia = true [LoadBalancerClass "class1"] floating-network-id = 17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet-id = 3535d323-bb67-4e9a-92fe-fe85e8489245 [LoadBalancerClass "class2"] floating-network-id = 17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet-tags = class2tag [LoadBalancerClass "class3"] floating-network-id = 17521769-5c47-4869-8296-4c6668d4e2c8 floating-subnet = external_subnet3 kind: ConfigMap metadata: creationTimestamp: "2022-10-06T14:20:24Z" name: cloud-conf namespace: openshift-cloud-controller-manager resourceVersion: "10815729" uid: efe71473-5362-4e61-bf4e-554929be1e60 $ oc get pods -n openshift-kube-controller-manager -l app=kube-controller-manager NAME READY STATUS RESTARTS AGE kube-controller-manager-ostest-qbr7x-master-0 4/4 Running 4 34m kube-controller-manager-ostest-qbr7x-master-1 4/4 Running 4 36m kube-controller-manager-ostest-qbr7x-master-2 4/4 Running 1 (23m ago) 24m $ oc logs kube-controller-manager-ostest-qbr7x-master-2 -n openshift-kube-controller-manager [...] W1025 15:14:30.440858 1 openstack.go:364] Non-fatal error parsing OpenStack cloud config. This may happen when passing config directives exclusive to OpenStack CCM to the legacy cloud provider. Legacy cloud provider has correctly parsed all directives it knows about : warnings: can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass" can't store data at section "LoadBalancerClass"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399