Description of problem: ----------------------- Installation failed: ... time="2021-02-08T12:50:32Z" level=debug msg="module.masters.ironic_deployment.openshift-master-deployment[2]: Creation complete after 14m34s [id=ce21dd2e-e6bf-4225-b45e-c76c6dffc478]" time="2021-02-08T12:50:32Z" level=debug time="2021-02-08T12:50:32Z" level=debug msg="Apply complete! Resources: 14 added, 0 changed, 0 destroyed." time="2021-02-08T12:50:32Z" level=debug msg="OpenShift Installer 4.7.0-0.nightly-2021-02-06-084550" time="2021-02-08T12:50:32Z" level=debug msg="Built from commit c0489117068cb00c5222bb0762a87605f41ebe04" time="2021-02-08T12:50:32Z" level=info msg="Waiting up to 20m0s for the Kubernetes API at https://api.ocp-edge-cluster-0.qe.lab.redhat.com:6443..." time="2021-02-08T12:50:33Z" level=info msg="API v1.20.0+ba45583 up" time="2021-02-08T12:50:33Z" level=info msg="Waiting up to 30m0s for bootstrapping to complete..." time="2021-02-08T13:20:33Z" level=info msg="Cluster operator network ManagementStateDegraded is False with : " time="2021-02-08T13:20:33Z" level=info msg="Cluster operator network Progressing is True with Deploying: DaemonSet \"openshift-multus/network-metrics-daemon\" is not available (awaiting 3 nodes)\nDaemon Set \"openshift-multus/multus-admission-controller\" is not available (awaiting 3 nodes)\nDeployment \"openshift-network-diagnostics/network-check-source\" is not available (awaiting 1 nodes)" ... Bunch of in a CrashLoopBackOff state ------------------------------------ oc get po -A | grep Crash openshift-apiserver-operator openshift-apiserver-operator-5fbd9ddd8d-2d2sd 0/1 CrashLoopBackOff 20 101m openshift-authentication-operator authentication-operator-7c48f84d4f-967wj 0/1 CrashLoopBackOff 31 101m openshift-cluster-storage-operator cluster-storage-operator-5874449d4-42c7d 0/1 CrashLoopBackOff 20 101m openshift-cluster-storage-operator csi-snapshot-controller-operator-94cfd9d87-f69rl 0/1 CrashLoopBackOff 20 101m openshift-config-operator openshift-config-operator-7d755ccb5f-w7gx6 0/1 CrashLoopBackOff 35 101m openshift-controller-manager-operator openshift-controller-manager-operator-6f7bd7cb7-9hbr2 0/1 CrashLoopBackOff 20 101m openshift-etcd-operator etcd-operator-7fd67c57f6-pzntc 0/1 CrashLoopBackOff 20 101m openshift-insights insights-operator-7d898f8b9c-xm5d6 0/1 CrashLoopBackOff 18 88m openshift-kube-apiserver-operator kube-apiserver-operator-75cb8f94f7-ghntc 0/1 CrashLoopBackOff 20 101m openshift-kube-controller-manager-operator kube-controller-manager-operator-5bdb799f57-blxx6 0/1 CrashLoopBackOff 76 7h51m openshift-kube-scheduler-operator openshift-kube-scheduler-operator-9f9f98945-j8hpd 0/1 CrashLoopBackOff 76 7h51m openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-79b5fcf84-z66wf 0/1 CrashLoopBackOff 38 3h43m openshift-machine-api cluster-baremetal-operator-8d6b5c5cd-mmsx4 0/1 CrashLoopBackOff 22 101m openshift-monitoring cluster-monitoring-operator-85c55fdf7b-k2gxq 0/2 CrashLoopBackOff 46 101m openshift-service-ca-operator service-ca-operator-bbb9b5cf7-9qv4m 0/1 CrashLoopBackOff 76 7h51m oc logs authentication-operator-7c48f84d4f-967wj -n openshift-authentication-operator ------------------------------------------------------------------------------ Copying system trust bundle I0208 20:13:11.549899 1 observer_polling.go:159] Starting file observer W0208 20:13:11.550190 1 cmd.go:204] Using insecure, self-signed certificates I0208 20:13:11.550508 1 crypto.go:601] Generating new CA for cluster-authentication-operator-signer@1612815191 cert, and key in /tmp/serving-cert-835690657/serving-signer.crt, /tmp/serving-cert-83 5690657/serving-signer.key I0208 20:13:11.881507 1 observer_polling.go:159] Starting file observer W0208 20:13:36.883395 1 builder.go:207] unable to get owner reference (falling back to namespace): Get "https://[fd02::1]:443/api/v1/namespaces/openshift-authentication-operator/pods/authenticatio n-operator-7c48f84d4f-967wj": dial tcp [fd02::1]:443: i/o timeout I0208 20:13:36.884139 1 builder.go:238] cluster-authentication-operator version v4.7.0-202102060108.p0-0-g771b1b3- I0208 20:13:36.885730 1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/tmp/serving-cert-835690657/tls.crt::/tmp/serving-cert-835690657/tls.key" I0208 20:14:06.670259 1 cmd.go:88] Received SIGTERM or SIGINT signal, shutting down controller. I0208 20:14:06.882044 1 observer_polling.go:162] Shutting down file observer F0208 20:14:09.302061 1 cmd.go:129] unable to load configmap based request-header-client-ca-file: Get "https://[fd02::1]:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentica tion?timeout=10s": context deadline exceeded goroutine 1 [running]: k8s.io/klog/v2.stacks(0xc000012001, 0xc000316c80, 0xf7, 0x260) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9 k8s.io/klog/v2.(*loggingT).output(0x3754940, 0xc000000003, 0x0, 0x0, 0xc000576380, 0x358e9b0, 0x6, 0x81, 0x414300) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:975 +0x19b k8s.io/klog/v2.(*loggingT).printDepth(0x3754940, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc0001228b0, 0x1, 0x1) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:732 +0x16f k8s.io/klog/v2.(*loggingT).print(...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:714 k8s.io/klog/v2.Fatal(...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:1482 github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommandWithContext.func1(0xc0006b2580, 0xc00042dd10, 0x0, 0x3) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:129 +0x785 github.com/spf13/cobra.(*Command).execute(0xc0006b2580, 0xc00042dce0, 0x3, 0x3, 0xc0006b2580, 0xc00042dce0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/spf13/cobra/command.go:854 +0x2c2 github.com/spf13/cobra.(*Command).ExecuteC(0xc0006b22c0, 0xc0000b8180, 0x3754420, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/spf13/cobra/command.go:958 +0x375 github.com/spf13/cobra.(*Command).Execute(...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/spf13/cobra/command.go:895 main.main() /go/src/github.com/openshift/cluster-authentication-operator/cmd/authentication-operator/main.go:29 +0x155 goroutine 6 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x3754940) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:1169 +0x8b created by k8s.io/klog/v2.init.0 /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:417 +0xdf goroutine 53 [select]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2514120, 0x26bb0c0, 0xc00083f3b0, 0x1, 0xc0000c20c0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2514120, 0x12a05f200, 0x0, 0xc0004ca601, 0xc0000c20c0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 k8s.io/apimachinery/pkg/util/wait.Forever(0x2514120, 0x12a05f200) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f created by k8s.io/component-base/logs.InitLogs /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a goroutine 44 [select]: k8s.io/apimachinery/pkg/util/wait.contextForChannel.func1(0xc0000c2240, 0xc0005961f0, 0x271d1e0, 0xc000392000) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:279 +0xbd created by k8s.io/apimachinery/pkg/util/wait.contextForChannel /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:278 +0x8c goroutine 32 [syscall]: os/signal.signal_recv(0x26e04e0) /usr/lib/golang/src/runtime/sigqueue.go:147 +0x9d os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x25 created by os/signal.Notify.func1.1 /usr/lib/golang/src/os/signal/signal.go:150 +0x45 goroutine 52 [select]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2514120, 0x26bb0c0, 0xc00079a120, 0x1, 0xc0000c20c0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2514120, 0x12a05f200, 0x0, 0xc000486f01, 0xc0000c20c0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 k8s.io/apimachinery/pkg/util/wait.Forever(0x2514120, 0x12a05f200) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f created by k8s.io/component-base/logs.InitLogs /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a goroutine 132 [chan receive]: k8s.io/apiserver/pkg/server.SetupSignalContext.func1(0xc000a809e0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/signal.go:50 +0x5b created by k8s.io/apiserver/pkg/server.SetupSignalContext /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/signal.go:47 +0xf3 goroutine 134 [select]: k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc00042a180, 0xc000c8ff10, 0xc0000c24e0, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:539 +0x11d k8s.io/apimachinery/pkg/util/wait.PollUntil(0x2540be400, 0xc000a1ff10, 0xc0000c2240, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:492 +0xc5 k8s.io/apimachinery/pkg/util/wait.PollInfinite(0x2540be400, 0xc000a1ff10, 0x0, 0x0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:464 +0x87 k8s.io/apimachinery/pkg/util/wait.PollImmediateInfinite(0x2540be400, 0xc000a1ff10, 0x70b64f, 0x3754940) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:481 +0x73 github.com/openshift/library-go/pkg/controller/fileobserver.(*pollingObserver).processReactors(0xc0007bc9b0, 0xc00041e660) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/fileobserver/observer_polling.go:86 +0x7c github.com/openshift/library-go/pkg/controller/fileobserver.(*pollingObserver).Run(0xc0007bc9b0, 0xc00041e660) /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/fileobserver/observer_polling.go:161 +0x12d created by github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommandWithContext.func1 /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:125 +0x689 goroutine 45 [select]: k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc0000c2780, 0x2540be400, 0x0, 0xc0000c26c0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:588 +0x17b created by k8s.io/apimachinery/pkg/util/wait.poller.func1 /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:571 +0x8c goroutine 46 [chan receive]: k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00089e000) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:219 +0x66 created by k8s.io/apimachinery/pkg/watch.NewBroadcaster /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:73 +0xf7 goroutine 47 [chan receive]: k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x26d86a0, 0xc0008744e0, 0xc000122350) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:301 +0xaa created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:299 +0x6e goroutine 48 [chan receive]: k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x26d86a0, 0xc0007969f0, 0xc000875260) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:301 +0xaa created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:299 +0x6e goroutine 162 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0004877a0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 163 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000487a40) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 139 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0005e7b60) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 140 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0005e7ce0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 141 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0005e7d40) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 142 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0005e7e60) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 143 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0005e7f20) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 144 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000b083c0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 194 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873200) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 195 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000873320) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 196 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873380) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 197 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0008734a0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 198 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873500) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 199 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000873620) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 204 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873c20) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 180 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000b09c20) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 181 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000b09d40) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 182 [chan receive]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000b09da0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 183 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000b09ec0) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 goroutine 205 [select]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000873d40) /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405 created by k8s.io/client-go/util/workqueue.newDelayingQueue /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185 The problem seems to be with `Kubernetes` service in `default` namespace missing IPv6 endpoints ----------------------------------------------- [kni@provisionhost-0-0 ~]$ oc get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP fd02::1 <none> 443/TCP 7h55m [kni@provisionhost-0-0 ~]$ oc get endpoint [kni@provisionhost-0-0 ~]$ oc get endpoints NAME ENDPOINTS AGE kubernetes 192.168.123.125:6443 7h55m [kni@provisionhost-0-0 ~]$ oc get endpointslices NAME ADDRESSTYPE PORTS ENDPOINTS AGE kubernetes IPv4 6443 192.168.123.125 7h55m oc get service kubernetes -o yaml ---------------------------------- apiVersion: v1 kind: Service metadata: creationTimestamp: "2021-02-08T12:26:30Z" labels: component: apiserver provider: kubernetes managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: .: {} f:component: {} f:provider: {} f:spec: f:clusterIP: {} f:ipFamilyPolicy: {} f:ports: .: {} k:{"port":443,"protocol":"TCP"}: .: {} f:name: {} f:port: {} f:protocol: {} f:targetPort: {} f:sessionAffinity: {} f:type: {} manager: kube-apiserver operation: Update time: "2021-02-08T12:26:30Z" name: kubernetes namespace: default resourceVersion: "267" selfLink: /api/v1/namespaces/default/services/kubernetes uid: 64dd0f23-6c03-4cc2-a80a-058ec247d455 spec: clusterIP: fd02::1 clusterIPs: - fd02::1 ipFamilies: - IPv6 ipFamilyPolicy: SingleStack ports: - name: https port: 443 protocol: TCP targetPort: 6443 sessionAffinity: None type: ClusterIP status: loadBalancer: {} oc get endpoints kubernetes -o yaml ------------------------------------ apiVersion: v1 kind: Endpoints metadata: creationTimestamp: "2021-02-08T12:26:30Z" labels: endpointslice.kubernetes.io/skip-mirror: "true" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: .: {} f:endpointslice.kubernetes.io/skip-mirror: {} f:subsets: {} manager: kube-apiserver operation: Update time: "2021-02-08T12:26:30Z" name: kubernetes namespace: default resourceVersion: "269" selfLink: /api/v1/namespaces/default/endpoints/kubernetes uid: 23b730ed-caf0-4176-b3a4-ae7818e9668f subsets: - addresses: - ip: 192.168.123.125 ports: - name: https port: 6443 protocol: TCP oc get endpointslices kubernetes -o yaml ---------------------------------------- addressType: IPv4 apiVersion: discovery.k8s.io/v1beta1 endpoints: - addresses: - 192.168.123.125 conditions: ready: true kind: EndpointSlice metadata: creationTimestamp: "2021-02-08T12:26:30Z" generation: 1 labels: kubernetes.io/service-name: kubernetes managedFields: - apiVersion: discovery.k8s.io/v1beta1 fieldsType: FieldsV1 fieldsV1: f:addressType: {} f:endpoints: {} f:metadata: f:labels: .: {} f:kubernetes.io/service-name: {} f:ports: {} manager: kube-apiserver operation: Update time: "2021-02-08T12:26:30Z" name: kubernetes namespace: default resourceVersion: "270" selfLink: /apis/discovery.k8s.io/v1beta1/namespaces/default/endpointslices/kubernetes uid: 76a150ba-2963-4f8c-8944-2082d0faf39a ports: - name: https port: 6443 protocol: TCP Version-Release number of selected component (if applicable): ------------------------------------------------------------- 4.7.0-0.nightly-2021-02-06-084550 Steps to Reproduce: ------------------- 1. Install dual stack OCP cluster on baremetal using IPI method and main network being IPv6(1st network in machineNetwork, clusterNetwork, serviceNetwork) for .e.g: machineNetwork: - cidr: fd2e:6f44:5dd8::/64 - cidr: 192.168.123.0/24 clusterNetwork: - cidr: fd01::/48 hostPrefix: 64 - cidr: 10.128.0.0/14 hostPrefix: 23 serviceNetwork: - fd02::/112 - 172.30.0.0/16 Actual results: --------------- Installation fails Expected results: ----------------- Installation succeeds
oc get po openshift-apiserver-operator-5fbd9ddd8d-2d2sd -n openshift-apiserver-operator -o jsonpath='{ .status.hostIP }' fd2e:6f44:5dd8::117
I think that this is a configuration problem, the apiserver log in the bootstrap show > I0208 13:09:03.980230 1 flags.go:59] FLAG: --advertise-address="<nil>" then it defaults to the IPv4 address using in the interface with the default network > I0208 13:09:03.981891 1 server.go:697] external host was not specified, using 192.168.123.125 if we want to use the IPv6 address of the interface in the default network we need to pass "::" to the --advertise-address flag
advertise-address is set by config observer owned by networking team. Moving component.
Actually, it is not due to an observer, but the advertise address is the node IP, compare https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/bindata/v4.1.0/kube-apiserver/pod.yaml#L57.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438