Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1926465

Summary: bootstrap kube-apiserver does not have --advertise-address set – was: [BM][IPI][DualStack] Installation fails cause Kubernetes service doesn't have IPv6 endpoints
Product: OpenShift Container Platform Reporter: Yurii Prokulevych <yprokule>
Component: kube-apiserverAssignee: Stefan Schimanski <sttts>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: high Docs Contact:
Priority: high    
Version: 4.7CC: achernet, aojeagar, aos-bugs, deads, kewang, mcornea, mfojtik, sttts, xxia
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:42:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1927644    

Description Yurii Prokulevych 2021-02-08 20:29:29 UTC 
Description of problem:
-----------------------
Installation failed:
...
time="2021-02-08T12:50:32Z" level=debug msg="module.masters.ironic_deployment.openshift-master-deployment[2]: Creation complete after 14m34s [id=ce21dd2e-e6bf-4225-b45e-c76c6dffc478]"
time="2021-02-08T12:50:32Z" level=debug
time="2021-02-08T12:50:32Z" level=debug msg="Apply complete! Resources: 14 added, 0 changed, 0 destroyed."
time="2021-02-08T12:50:32Z" level=debug msg="OpenShift Installer 4.7.0-0.nightly-2021-02-06-084550"
time="2021-02-08T12:50:32Z" level=debug msg="Built from commit c0489117068cb00c5222bb0762a87605f41ebe04"
time="2021-02-08T12:50:32Z" level=info msg="Waiting up to 20m0s for the Kubernetes API at https://api.ocp-edge-cluster-0.qe.lab.redhat.com:6443..."
time="2021-02-08T12:50:33Z" level=info msg="API v1.20.0+ba45583 up"
time="2021-02-08T12:50:33Z" level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
time="2021-02-08T13:20:33Z" level=info msg="Cluster operator network ManagementStateDegraded is False with : "
time="2021-02-08T13:20:33Z" level=info msg="Cluster operator network Progressing is True with Deploying: DaemonSet \"openshift-multus/network-metrics-daemon\" is not available (awaiting 3 nodes)\nDaemon
Set \"openshift-multus/multus-admission-controller\" is not available (awaiting 3 nodes)\nDeployment \"openshift-network-diagnostics/network-check-source\" is not available (awaiting 1 nodes)"
...

Bunch of in a CrashLoopBackOff state
------------------------------------
oc get po -A | grep Crash
openshift-apiserver-operator                       openshift-apiserver-operator-5fbd9ddd8d-2d2sd                    0/1     CrashLoopBackOff    20         101m
openshift-authentication-operator                  authentication-operator-7c48f84d4f-967wj                         0/1     CrashLoopBackOff    31         101m
openshift-cluster-storage-operator                 cluster-storage-operator-5874449d4-42c7d                         0/1     CrashLoopBackOff    20         101m
openshift-cluster-storage-operator                 csi-snapshot-controller-operator-94cfd9d87-f69rl                 0/1     CrashLoopBackOff    20         101m
openshift-config-operator                          openshift-config-operator-7d755ccb5f-w7gx6                       0/1     CrashLoopBackOff    35         101m
openshift-controller-manager-operator              openshift-controller-manager-operator-6f7bd7cb7-9hbr2            0/1     CrashLoopBackOff    20         101m
openshift-etcd-operator                            etcd-operator-7fd67c57f6-pzntc                                   0/1     CrashLoopBackOff    20         101m
openshift-insights                                 insights-operator-7d898f8b9c-xm5d6                               0/1     CrashLoopBackOff    18         88m
openshift-kube-apiserver-operator                  kube-apiserver-operator-75cb8f94f7-ghntc                         0/1     CrashLoopBackOff    20         101m
openshift-kube-controller-manager-operator         kube-controller-manager-operator-5bdb799f57-blxx6                0/1     CrashLoopBackOff    76         7h51m
openshift-kube-scheduler-operator                  openshift-kube-scheduler-operator-9f9f98945-j8hpd                0/1     CrashLoopBackOff    76         7h51m
openshift-kube-storage-version-migrator-operator   kube-storage-version-migrator-operator-79b5fcf84-z66wf           0/1     CrashLoopBackOff    38         3h43m
openshift-machine-api                              cluster-baremetal-operator-8d6b5c5cd-mmsx4                       0/1     CrashLoopBackOff    22         101m
openshift-monitoring                               cluster-monitoring-operator-85c55fdf7b-k2gxq                     0/2     CrashLoopBackOff    46         101m
openshift-service-ca-operator                      service-ca-operator-bbb9b5cf7-9qv4m                              0/1     CrashLoopBackOff    76         7h51m

 oc logs authentication-operator-7c48f84d4f-967wj -n openshift-authentication-operator
------------------------------------------------------------------------------
Copying system trust bundle
I0208 20:13:11.549899       1 observer_polling.go:159] Starting file observer
W0208 20:13:11.550190       1 cmd.go:204] Using insecure, self-signed certificates
I0208 20:13:11.550508       1 crypto.go:601] Generating new CA for cluster-authentication-operator-signer@1612815191 cert, and key in /tmp/serving-cert-835690657/serving-signer.crt, /tmp/serving-cert-83
5690657/serving-signer.key
I0208 20:13:11.881507       1 observer_polling.go:159] Starting file observer
W0208 20:13:36.883395       1 builder.go:207] unable to get owner reference (falling back to namespace): Get "https://[fd02::1]:443/api/v1/namespaces/openshift-authentication-operator/pods/authenticatio
n-operator-7c48f84d4f-967wj": dial tcp [fd02::1]:443: i/o timeout
I0208 20:13:36.884139       1 builder.go:238] cluster-authentication-operator version v4.7.0-202102060108.p0-0-g771b1b3-
I0208 20:13:36.885730       1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/tmp/serving-cert-835690657/tls.crt::/tmp/serving-cert-835690657/tls.key"
I0208 20:14:06.670259       1 cmd.go:88] Received SIGTERM or SIGINT signal, shutting down controller.
I0208 20:14:06.882044       1 observer_polling.go:162] Shutting down file observer
F0208 20:14:09.302061       1 cmd.go:129] unable to load configmap based request-header-client-ca-file: Get "https://[fd02::1]:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentica
tion?timeout=10s": context deadline exceeded
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000012001, 0xc000316c80, 0xf7, 0x260)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x3754940, 0xc000000003, 0x0, 0x0, 0xc000576380, 0x358e9b0, 0x6, 0x81, 0x414300)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:975 +0x19b
k8s.io/klog/v2.(*loggingT).printDepth(0x3754940, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc0001228b0, 0x1, 0x1)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:732 +0x16f
k8s.io/klog/v2.(*loggingT).print(...)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:714
k8s.io/klog/v2.Fatal(...)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:1482
github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommandWithContext.func1(0xc0006b2580, 0xc00042dd10, 0x0, 0x3)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:129 +0x785
github.com/spf13/cobra.(*Command).execute(0xc0006b2580, 0xc00042dce0, 0x3, 0x3, 0xc0006b2580, 0xc00042dce0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc0006b22c0, 0xc0000b8180, 0x3754420, 0x0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/spf13/cobra/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/spf13/cobra/command.go:895
main.main()
        /go/src/github.com/openshift/cluster-authentication-operator/cmd/authentication-operator/main.go:29 +0x155

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x3754940)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/klog/v2/klog.go:417 +0xdf

goroutine 53 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2514120, 0x26bb0c0, 0xc00083f3b0, 0x1, 0xc0000c20c0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2514120, 0x12a05f200, 0x0, 0xc0004ca601, 0xc0000c20c0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x2514120, 0x12a05f200)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a
goroutine 44 [select]:
k8s.io/apimachinery/pkg/util/wait.contextForChannel.func1(0xc0000c2240, 0xc0005961f0, 0x271d1e0, 0xc000392000)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:279 +0xbd
created by k8s.io/apimachinery/pkg/util/wait.contextForChannel
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:278 +0x8c

goroutine 32 [syscall]:
os/signal.signal_recv(0x26e04e0)
        /usr/lib/golang/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
        /usr/lib/golang/src/os/signal/signal.go:150 +0x45

goroutine 52 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x2514120, 0x26bb0c0, 0xc00079a120, 0x1, 0xc0000c20c0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2514120, 0x12a05f200, 0x0, 0xc000486f01, 0xc0000c20c0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x2514120, 0x12a05f200)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a

goroutine 132 [chan receive]:
k8s.io/apiserver/pkg/server.SetupSignalContext.func1(0xc000a809e0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/signal.go:50 +0x5b
created by k8s.io/apiserver/pkg/server.SetupSignalContext
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apiserver/pkg/server/signal.go:47 +0xf3
goroutine 134 [select]:
k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc00042a180, 0xc000c8ff10, 0xc0000c24e0, 0x0, 0x0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:539 +0x11d
k8s.io/apimachinery/pkg/util/wait.PollUntil(0x2540be400, 0xc000a1ff10, 0xc0000c2240, 0x0, 0x0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:492 +0xc5
k8s.io/apimachinery/pkg/util/wait.PollInfinite(0x2540be400, 0xc000a1ff10, 0x0, 0x0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:464 +0x87
k8s.io/apimachinery/pkg/util/wait.PollImmediateInfinite(0x2540be400, 0xc000a1ff10, 0x70b64f, 0x3754940)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:481 +0x73
github.com/openshift/library-go/pkg/controller/fileobserver.(*pollingObserver).processReactors(0xc0007bc9b0, 0xc00041e660)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/fileobserver/observer_polling.go:86 +0x7c
github.com/openshift/library-go/pkg/controller/fileobserver.(*pollingObserver).Run(0xc0007bc9b0, 0xc00041e660)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/fileobserver/observer_polling.go:161 +0x12d
created by github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommandWithContext.func1
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/github.com/openshift/library-go/pkg/controller/controllercmd/cmd.go:125 +0x689

goroutine 45 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1(0xc0000c2780, 0x2540be400, 0x0, 0xc0000c26c0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:588 +0x17b
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:571 +0x8c

goroutine 46 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00089e000)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:219 +0x66
created by k8s.io/apimachinery/pkg/watch.NewBroadcaster
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/apimachinery/pkg/watch/mux.go:73 +0xf7

goroutine 47 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x26d86a0, 0xc0008744e0, 0xc000122350)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:301 +0xaa
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:299 +0x6e

goroutine 48 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x26d86a0, 0xc0007969f0, 0xc000875260)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:301 +0xaa
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/tools/record/event.go:299 +0x6e
goroutine 162 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0004877a0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 163 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000487a40)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185

goroutine 139 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0005e7b60)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 140 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0005e7ce0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185

goroutine 141 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0005e7d40)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 142 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0005e7e60)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185
goroutine 143 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0005e7f20)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 144 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000b083c0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185

goroutine 194 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873200)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 195 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000873320)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185

goroutine 196 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873380)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 197 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0008734a0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185
goroutine 198 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873500)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 199 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000873620)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185

goroutine 204 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000873c20)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 180 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000b09c20)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 181 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000b09d40)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185

goroutine 182 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000b09da0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135

goroutine 183 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000b09ec0)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185
goroutine 205 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000873d40)
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /go/src/github.com/openshift/cluster-authentication-operator/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:68 +0x185


The problem seems to be with `Kubernetes` service in `default` namespace missing IPv6 endpoints
-----------------------------------------------

[kni@provisionhost-0-0 ~]$ oc get service
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   fd02::1      <none>        443/TCP   7h55m
[kni@provisionhost-0-0 ~]$ oc get endpoint

[kni@provisionhost-0-0 ~]$ oc get endpoints
NAME         ENDPOINTS              AGE
kubernetes   192.168.123.125:6443   7h55m

[kni@provisionhost-0-0 ~]$ oc get endpointslices
NAME         ADDRESSTYPE   PORTS   ENDPOINTS         AGE
kubernetes   IPv4          6443    192.168.123.125   7h55m



oc get service kubernetes -o yaml
----------------------------------
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2021-02-08T12:26:30Z"
  labels:
    component: apiserver
    provider: kubernetes
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:component: {}
          f:provider: {}
      f:spec:
        f:clusterIP: {}
        f:ipFamilyPolicy: {}
        f:ports:
          .: {}
          k:{"port":443,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kube-apiserver
    operation: Update
    time: "2021-02-08T12:26:30Z"
  name: kubernetes
  namespace: default
  resourceVersion: "267"
  selfLink: /api/v1/namespaces/default/services/kubernetes
  uid: 64dd0f23-6c03-4cc2-a80a-058ec247d455
spec:
  clusterIP: fd02::1
  clusterIPs:
  - fd02::1
  ipFamilies:
  - IPv6
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 6443
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

oc get endpoints kubernetes -o yaml
------------------------------------
apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: "2021-02-08T12:26:30Z"
  labels:
    endpointslice.kubernetes.io/skip-mirror: "true"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:endpointslice.kubernetes.io/skip-mirror: {}
      f:subsets: {}
    manager: kube-apiserver
    operation: Update
    time: "2021-02-08T12:26:30Z"
  name: kubernetes
  namespace: default
  resourceVersion: "269"
  selfLink: /api/v1/namespaces/default/endpoints/kubernetes
  uid: 23b730ed-caf0-4176-b3a4-ae7818e9668f
subsets:
- addresses:
  - ip: 192.168.123.125
  ports:
  - name: https
    port: 6443
    protocol: TCP

oc get endpointslices kubernetes -o yaml
----------------------------------------
addressType: IPv4
apiVersion: discovery.k8s.io/v1beta1
endpoints:
- addresses:
  - 192.168.123.125
  conditions:
    ready: true
kind: EndpointSlice
metadata:
  creationTimestamp: "2021-02-08T12:26:30Z"
  generation: 1
  labels:
    kubernetes.io/service-name: kubernetes
  managedFields:
  - apiVersion: discovery.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:addressType: {}
      f:endpoints: {}
      f:metadata:
        f:labels:
          .: {}
          f:kubernetes.io/service-name: {}
      f:ports: {}
    manager: kube-apiserver
    operation: Update
    time: "2021-02-08T12:26:30Z"
  name: kubernetes
  namespace: default
  resourceVersion: "270"
  selfLink: /apis/discovery.k8s.io/v1beta1/namespaces/default/endpointslices/kubernetes
  uid: 76a150ba-2963-4f8c-8944-2082d0faf39a
ports:
- name: https
  port: 6443
  protocol: TCP

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
4.7.0-0.nightly-2021-02-06-084550



Steps to Reproduce:
-------------------
1. Install dual stack OCP cluster on baremetal using IPI method and main network being IPv6(1st network in machineNetwork, clusterNetwork, serviceNetwork) for .e.g:

  machineNetwork:
  - cidr: fd2e:6f44:5dd8::/64
  - cidr: 192.168.123.0/24
  clusterNetwork:
  - cidr: fd01::/48
    hostPrefix: 64
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - fd02::/112
  - 172.30.0.0/16


Actual results:
---------------
Installation fails


Expected results:
-----------------
Installation succeeds
Comment 2 Yurii Prokulevych 2021-02-08 20:35:14 UTC 
oc get po openshift-apiserver-operator-5fbd9ddd8d-2d2sd  -n openshift-apiserver-operator -o jsonpath='{ .status.hostIP }'
fd2e:6f44:5dd8::117
Comment 3 Antonio Ojea 2021-02-09 09:15:43 UTC 
I think that this is a configuration problem, the apiserver log in the bootstrap show

> I0208 13:09:03.980230       1 flags.go:59] FLAG: --advertise-address="<nil>"

then it defaults to the IPv4 address using in the interface with the default network

> I0208 13:09:03.981891       1 server.go:697] external host was not specified, using 192.168.123.125

if we want to use the IPv6 address of the interface in the default network we need to pass "::" to the --advertise-address flag
Comment 4 Stefan Schimanski 2021-02-09 11:38:13 UTC 
advertise-address is set by config observer owned by networking team. Moving component.
Comment 5 Stefan Schimanski 2021-02-09 12:26:34 UTC 
Actually, it is not due to an observer, but the advertise address is the node IP, compare https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/bindata/v4.1.0/kube-apiserver/pod.yaml#L57.
Comment 12 errata-xmlrpc 2021-07-27 22:42:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438