Bug 1940898

--- Additional comment from Michael Gugino on 2021-03-19 13:47:55 UTC ---

--snip--

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-vsphere/1372795001506893824

vsphere test has one worker node fail to issue a CSR.  The machine was created, 3 minutes later the machine was 'provisioned' indicating the cloud was running the instance.  However, this cluster has 5 nodes (2 workers, 3 masters) but 15 approved CSRs.  There should only be 10 approved CSRs with 5 nodes (1 client, 1 server).  This indicates there's some problem with the hosts or the CSR approver.


---- New info:

One worker failed to join the cluster.  Looking at another kubelet's logs, it's completely spammed with the following entries.  This looks significant.


Mar 19 07:42:23.567191 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 hyperkube[1728]: E0319 07:42:23.566944    1728 kubelet_volumes.go:154] orphaned pod "043e804b-7102-4832-8dcd-362d5d10c9fe" found, but volume paths are still present on disk : There were a total of 39 errors similar to this. Turn up verbosity to see them.
Mar 19 07:42:23.959907 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:23.970798 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:25.373491 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:25.386088 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:25.568172 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 hyperkube[1728]: E0319 07:42:25.568120    1728 kubelet_volumes.go:154] orphaned pod "043e804b-7102-4832-8dcd-362d5d10c9fe" found, but volume paths are still present on disk : There were a total of 39 errors similar to this. Turn up verbosity to see them.
Mar 19 07:42:26.296502 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:26.307259 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:26.622206 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:26.633790 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:27.027446 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:27.038595 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:27.257966 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:27.274212 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument
Mar 19 07:42:27.361240 ci-op-5dk64ln5-36a8b-9zsjg-worker-7knv5 systemd[1]: Failed to set up mount unit: Invalid argument