Description of problem: When changing the vsphere credentials in the cluster, it looks like the operator doesn't watch the secret, only the ConfigMap. Version-Release number of selected component (if applicable): OCP 4.3.8 How reproducible: Always Steps to Reproduce: 1. Have a OpenShift with vSphere provider enabled 2. Change vSphere credentials (oc edit secret vsphere-creds -n kube-system) 3. Nothing happens after the secret is changed 4. If you edit the ConfigMap (oc edit cm cloud-provider-config -n openshift-config), the changes are made and it uses the new credentials Reference: https://access.redhat.com/solutions/4618011 Expected results: After editing the secret, operator needs to reload settings instead of watching only the ConfigMap.
Setting target release to current development version (4.5) for investigation. Where fixes (if any) are required/requested for prior versions, cloned BZs will be created when appropriate.
The secret containing the those credentials is not under cluster management. The cluster config operator watches the configMap referenced by the infrastructureSpec.CloudConfig. It does any transformations if needed, and it creates a new configmap in the openshift-config-managed namespace that is suitable for consumers, i.e kube controller manager. https://github.com/openshift/installer/blob/fc052ec7ec6d3c3d330c7e5c4ac81e085d035fb3/pkg/asset/manifests/cloudproviderconfig.go#L69 https://github.com/openshift/installer/blob/70447fecc997c590109ee63b4d9bd70b9d532c7d/pkg/asset/manifests/infrastructure.go#L179 https://github.com/openshift/api/blob/0f159fee64dbf711d40dac3fa2ec8b563a2aaca8/config/v1/types_infrastructure.go#L40 https://github.com/openshift/cluster-config-operator/blob/d8cc70c679ef227d1d3152f110f9056c7094ae96/pkg/operator/kube_cloud_config/controller.go the After changing the secret credential values, I'd expect the vsphere cloud controller to refresh its cache and pick those changes https://github.com/kubernetes/kubernetes/blob/f0be447922224c322e756930e5bda6c7e735fd04/staging/src/k8s.io/legacy-cloud-providers/vsphere/nodemanager.go#L385-L390 are you seeing it not picking them? If so can you share kube controller manager logs?
Hi Alberto, After changing the secret credentials values, the cache isn't refreshed. I just tested with the newly released OCP 4.4.3. Only when I change the ConfigMap (i.e. putting a newline in it), the machine-config-operator reconfigures the nodes to the new ConfigMap and secret credentials. It's very easy to reproduce. I couldn't find anything relevant in the kube controller manager logs. I'm attaching it. Thanks!
The logs here suggest that the PVCs are being created correctly. Though I'm suspecting that we are looking in the wrong place. I think we need to see the logs for kubelets where the PVCs are meant to be being attached. I wonder if the kubelets are not being configured correctly to use the secret and instead are using static credentials.
>After changing the secret credentials values, the cache isn't refreshed. ok cool, so (regardless of the internal technical details about cache or anything) what's the issue you are facing as a user after editing the secret. Is it that the provider is not functional any more? is it that the provider keeps being functional but you think it shouldn't because you set bad creds in the secret (if so this is a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1824481)?
> The logs here suggest that the PVCs are being created correctly. Yes, I changed the secret credentials to another valid user - but logs from VMWare says that those two PVCs were created by the old user. Only when I changed the ConfigMap, the MCO did its work and the new credentials were used. > what's the issue you are facing as a user after editing the secret. Is it that the provider is not functional any more? It's that when we change the secret credentials, the cluster won't use the new credentials. IMHO this is bad user experience. In one case, a customer had to change the credentials to another one that was working. In another case, a customer forgot some characters in the username at installation time and the provisioner wasn't working - they needed to change the credentials. In both cases, just modifying the secret credentials didn't work. The workaround was to put an empty newline in the ConfigMap to force re-reading the secret credentials.
This sounds to me like a duplicate for https://bugzilla.redhat.com/show_bug.cgi?id=1824481. There is a PR to fix that: https://github.com/kubernetes/kubernetes/pull/90836, but I'll close this in favour of the ref. as that one is more detailed and closer to be stated as a bug in kubernetes, then being a feature request. Feel free to re-open if you have any additional details, and comment on the upstream issue scrapped from the description: https://github.com/kubernetes/kubernetes/issues/90844 *** This bug has been marked as a duplicate of bug 1824481 ***