The cluster-network-operator seems to be crashing with ExitCode: 2 which suggests that they might be starting a binary with missing required argument. That seems to be the only clue here so far, moving to network.
Daneyon, can you take a look?
I was able to get the full stack trace out of the network-operator E0813 14:48:11.485042 1 runtime.go:66] Observed a panic: "index out of range" (runtime error: index out of range) /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /usr/local/go/src/runtime/asm_amd64.s:573 /usr/local/go/src/runtime/panic.go:502 /usr/local/go/src/runtime/panic.go:28 /go/src/github.com/openshift/cluster-network-operator/pkg/util/proxyconfig/merge.go:36 /go/src/github.com/openshift/cluster-network-operator/pkg/controller/proxyconfig/status.go:24 /go/src/github.com/openshift/cluster-network-operator/pkg/controller/proxyconfig/controller.go:128 /go/src/github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213 /go/src/github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /usr/local/go/src/runtime/asm_amd64.s:2361 panic: runtime error: index out of range [recovered] panic: runtime error: index out of range goroutine 285 [running]: github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107 panic(0x12946a0, 0x1e9a340) /usr/local/go/src/runtime/panic.go:502 +0x229 github.com/openshift/cluster-network-operator/pkg/util/proxyconfig.MergeUserSystemNoProxy(0xc42080c340, 0xc420099e00, 0xc42080c680, 0xc420967200, 0x16, 0xc4205fdca0, 0x4, 0x0) /go/src/github.com/openshift/cluster-network-operator/pkg/util/proxyconfig/merge.go:36 +0xa7c github.com/openshift/cluster-network-operator/pkg/controller/proxyconfig.(*ReconcileProxyConfig).syncProxyStatus(0xc42044d600, 0xc42080c340, 0xc420099e00, 0xc42080c680, 0xc420967200, 0x143e8fc, 0x11) /go/src/github.com/openshift/cluster-network-operator/pkg/controller/proxyconfig/status.go:24 +0x3d1 github.com/openshift/cluster-network-operator/pkg/controller/proxyconfig.(*ReconcileProxyConfig).Reconcile(0xc42044d600, 0x0, 0x0, 0xc420608520, 0x7, 0x1eb1600, 0xd2, 0xc420455560, 0x101c994) /go/src/github.com/openshift/cluster-network-operator/pkg/controller/proxyconfig/controller.go:128 +0xbb3 github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc4201b4f00, 0x0) /go/src/github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213 +0x1c6 github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1() /go/src/github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36 github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420874d00) /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54 github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420874d00, 0x3b9aca00, 0x0, 0x1, 0xc4201341e0) /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420874d00, 0x3b9aca00, 0xc4201341e0) /go/src/github.com/openshift/cluster-network-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start /go/src/github.com/openshift/cluster-network-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x35b Seems like network.Status.ServiceNetwork[0] is out of range i.e. ServiceNetwork is an empty slice
My network resource looks like this $ oc get network cluster -oyaml apiVersion: config.openshift.io/v1 kind: Network metadata: creationTimestamp: "2019-08-13T14:45:52Z" generation: 1 name: cluster resourceVersion: "422" selfLink: /apis/config.openshift.io/v1/networks/cluster uid: 0c96e1f6-bdd9-11e9-be58-fa163e7dcb6f spec: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 externalIP: policy: {} networkType: OpenShiftSDN serviceNetwork: - 172.30.0.0/16 status: {} Maybe should be Spec.ServiceNetwork?
https://github.com/openshift/cluster-network-operator/pull/286 submitted to fix bug.
Verify this bug with payload 4.2.0-0.nightly-2019-08-14-050713, bootstrap completed successfully, network-operator is running well on master node.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922