When the redeploy openshift ca playbook is executed, one of the tasks is to update the kubeconfigs with the new CA. That works normally as it should. However, when redeploy certificates playbook finishes - the SDN pod and also router pod as well shows that connecting to masterURL to API is signed by unknown certificate. I tried to reproduce that, however, for me it works as exepected. However, I see the difference - on my test the router and SDN pods - the applicaiton in the container is restarted ( monitored the restarts in the oc get pods). On customer however, the SDN pods are not restarted - and a workaround is to restart the pods. The only difference I saw is that customer uses crio and I was using the docker. Currently, checking it on CRIO lab. There is nothing much, no logs to provide really as the root cause is know. The error shown by the SDN pod is "x509 certificate signed by unknown issuer". Checked on customer side that the kubeconfig used by SDN were in fact updated.
there is workaround which fixes the issue, I just got confirmation from customer. Between rerunning the redeploy-openshift-ca and the redeploy-certificates - the SDN pods were deleted and new ones were started.
Given there's a workaround I'd like to leave it up to the team to set relative priority, Severity is the more appropriate field to represent the impact to the customer.
What specific version of openshift-ansible is being used? Were any certificates expired at the time the redeploy was being run? This could result in node services not being restarted. Even though playbooks complete successfully, logs can still be helpful in determining a root cause. Please attach playbook logs from when pods were left in a failed state.
In testing I have run into some issues with cert redeploy when using crio which I'm still investigating. Could you also attach a complete inventory?
Hello, yeah, customer also is using CRIO. >> What specific version of openshift-ansible is being used? latest version, we tested it also directly from the github repo and it fails >> Were any certificates expired at the time the redeploy was being run? This could result in node services not being restarted. No, the redeployment was done when the certificates are still valid. No certificates were expired. >> Even though playbooks complete successfully, logs can still be helpful in determining a root cause. Please attach playbook logs from when pods were left in a failed state. I am afraid I no longer have logs. However, if we can find the task which is responsible for restarting the SDN pods (restarting the process in the container, b/c on my lab my test didn't schedule new pods, just I see restarts of the process) it would add more context to the logic.
Verify this bug with openshift-ansible-3.11.318-1.git.0.bccee5b.el7.noarch.rpm. Setup a 3.11 cluster with using cri-o as container runtime, run redeploy openshift ca playbook, then redeploy certificates playbook. 11-16 18:08:25 TASK [openshift_node : Delete OpenShift SDN/OVS pods prior to upgrade] ********* 11-16 18:08:27 changed: [ec2-54-174-151-131.compute-1.amazonaws.com -> ec2-54-174-151-131.compute-1.amazonaws.com] => {"changed": true, "cmd": "oc get pods --config=/etc/origin/master/admin.kubeconfig --field-selector=spec.nodeName=ip-172-18-8-123.ec2.internal -o json -n openshift-sdn | oc delete --config=/etc/origin/master/admin.kubeconfig --force --grace-period=0 -f -\n", "delta": "0:00:01.655693", "end": "2020-11-16 05:08:27.346656", "rc": 0, "start": "2020-11-16 05:08:25.690963", "stderr": "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.", "stderr_lines": ["warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely."], "stdout": "pod \"ovs-v4hv7\" force deleted\npod \"sdn-frnd6\" force deleted", "stdout_lines": ["pod \"ovs-v4hv7\" force deleted", "pod \"sdn-frnd6\" force deleted"]} ... SDN pod on all the nodes were deleted. After redeploy-certificates playbook finished, checked the new router and SDN pods, no cert issue in the log.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 3.11.318 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5107