Earlier the logs had TLS handshake error https://bugzilla.redhat.com/show_bug.cgi?id=1762145 but this time i got this Logs from admission controller while validating bad config I1210 23:29:55.740963 1 webhook.go:111] validating network config spec: asdf 2019/12/10 23:29:55 http: panic serving 10.129.0.1:53696: assignment to entry in nil map goroutine 31041 [running]: net/http.(*conn).serve.func1(0xc000672000) /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/net/http/server.go:1769 +0x139 panic(0x11bd820, 0x14aac10) /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/panic.go:522 +0x1b5 github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.preprocessCNIConfig(0xc000608cb0, 0xe, 0xc000608d80, 0x4, 0x8, 0x8, 0x1, 0xc0006c1950, 0x710299, 0xc000672140) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:90 +0x197 github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.validateNetworkAttachmentDefinition(0xc0007e78c0, 0x1b, 0xc0007e78a0, 0x12, 0xc000608cb0, 0xe, 0x0, 0x0, 0xc000608c98, 0x7, ...) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:118 +0x27e github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.ValidateHandler(0x14e5020, 0xc0004520e0, 0xc000254a00) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:403 +0x146 net/http.HandlerFunc.ServeHTTP(0x139e438, 0x14e5020, 0xc0004520e0, 0xc000254a00) /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/net/http/server.go:1995 +0x44 net/http.(*ServeMux).ServeHTTP(0x2137ae0, 0x14e5020, 0xc0004520e0, 0xc000254a00) /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/net/http/server.go:2375 +0x1d6 net/http.serverHandler.ServeHTTP(0xc0003ad5f0, 0x14e5020, 0xc0004520e0, 0xc000254a00) /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/net/http/server.go:2774 +0xa8 net/http.(*conn).serve(0xc000672000, 0x14edc60, 0xc0004e8400) /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/net/http/server.go:1878 +0x851 created by net/http.(*Server).Serve /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/net/http/ser
I'm currently able to replicate the issue. $ cat def.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: macvlan-bridge spec: config: 'asdf' $ oc create -f def.yaml networkattachmentdefinition.k8s.cni.cncf.io/macvlan-bridge created $ oc get pods --all-namespaces | grep -i multus [...snip...] $ oc logs multus-admission-controller-5gkdd --namespace=openshift-multus I1212 17:32:27.482683 1 main.go:53] starting net-attach-def-admission-controller webhook server I1212 17:32:27.493962 1 localmetrics.go:50] UPdating net-attach-def metrics for any with value 0 I1212 17:32:27.494006 1 localmetrics.go:50] UPdating net-attach-def metrics for sriov with value 0 I1212 17:32:27.494854 1 controller.go:179] Starting net-attach-def-admission-controller I1212 17:32:27.595213 1 controller.go:188] net-attach-def-admission-controller synced and ready I1212 18:48:17.380940 1 webhook.go:111] validating network config spec: asdf 2019/12/12 18:48:17 http: panic serving 10.129.0.1:57938: assignment to entry in nil map goroutine 1118 [running]: net/http.(*conn).serve.func1(0xc4201e4960) /usr/local/go/src/net/http/server.go:1726 +0xd0 panic(0x10ea600, 0x1337eb0) /usr/local/go/src/runtime/panic.go:502 +0x229 github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.preprocessCNIConfig(0xc4206c2c50, 0xe, 0xc4206c2e38, 0x4, 0x8, 0x8, 0x0, 0x69ec80, 0xc420887888, 0x121cb00) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:90 +0x1be github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.validateNetworkAttachmentDefinition(0xc420723ba0, 0x1b, 0xc420723b80, 0x12, 0xc4206c2c50, 0xe, 0x0, 0x0, 0xc4206c2c09, 0x7, ...) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:118 +0x24c github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.ValidateHandler(0x1352020, 0xc4201e22a0, 0xc420778b00) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:403 +0x112 net/http.HandlerFunc.ServeHTTP(0x12cb9d0, 0x1352020, 0xc4201e22a0, 0xc420778b00) /usr/local/go/src/net/http/server.go:1947 +0x44 net/http.(*ServeMux).ServeHTTP(0x1becdc0, 0x1352020, 0xc4201e22a0, 0xc420778b00) /usr/local/go/src/net/http/server.go:2340 +0x130 net/http.serverHandler.ServeHTTP(0xc42045c000, 0x1352020, 0xc4201e22a0, 0xc420778b00) /usr/local/go/src/net/http/server.go:2697 +0xbc net/http.(*conn).serve(0xc4201e4960, 0x1353060, 0xc420688e80) /usr/local/go/src/net/http/server.go:1830 +0x651 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2798 +0x27b
Also, using release 4.3.0-0.ci-2019-12-05-183852
I'm still trying to isolate the cause, however, interestingly enough -- some errors are appropriately trapped, such as: ``` $ cat def2.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: macvlan-bridge-2 spec: config: '{}' $ oc create -f def2.yaml Error from server: error when creating "def2.yaml": admission webhook "multus-validating-config.k8s.io" denied the request: invalid config: missing 'type' in cni config $ oc logs multus-admission-controller-5gkdd --namespace=openshift-multus I1212 17:32:27.482683 1 main.go:53] starting net-attach-def-admission-controller webhook server I1212 17:32:27.493962 1 localmetrics.go:50] UPdating net-attach-def metrics for any with value 0 I1212 17:32:27.494006 1 localmetrics.go:50] UPdating net-attach-def metrics for sriov with value 0 I1212 17:32:27.494854 1 controller.go:179] Starting net-attach-def-admission-controller I1212 17:32:27.595213 1 controller.go:188] net-attach-def-admission-controller synced and ready I1212 18:48:17.380940 1 webhook.go:111] validating network config spec: asdf 2019/12/12 18:48:17 http: panic serving 10.129.0.1:57938: assignment to entry in nil map goroutine 1118 [running]: net/http.(*conn).serve.func1(0xc4201e4960) /usr/local/go/src/net/http/server.go:1726 +0xd0 panic(0x10ea600, 0x1337eb0) /usr/local/go/src/runtime/panic.go:502 +0x229 github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.preprocessCNIConfig(0xc4206c2c50, 0xe, 0xc4206c2e38, 0x4, 0x8, 0x8, 0x0, 0x69ec80, 0xc420887888, 0x121cb00) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:90 +0x1be github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.validateNetworkAttachmentDefinition(0xc420723ba0, 0x1b, 0xc420723b80, 0x12, 0xc4206c2c50, 0xe, 0x0, 0x0, 0xc4206c2c09, 0x7, ...) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:118 +0x24c github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook.ValidateHandler(0x1352020, 0xc4201e22a0, 0xc420778b00) /go/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/gopath/src/github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pkg/webhook/webhook.go:403 +0x112 net/http.HandlerFunc.ServeHTTP(0x12cb9d0, 0x1352020, 0xc4201e22a0, 0xc420778b00) /usr/local/go/src/net/http/server.go:1947 +0x44 net/http.(*ServeMux).ServeHTTP(0x1becdc0, 0x1352020, 0xc4201e22a0, 0xc420778b00) /usr/local/go/src/net/http/server.go:2340 +0x130 net/http.serverHandler.ServeHTTP(0xc42045c000, 0x1352020, 0xc4201e22a0, 0xc420778b00) /usr/local/go/src/net/http/server.go:2697 +0xbc net/http.(*conn).serve(0xc4201e4960, 0x1353060, 0xc420688e80) /usr/local/go/src/net/http/server.go:1830 +0x651 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2798 +0x27b I1212 19:00:00.162334 1 webhook.go:111] validating network config spec: {} ``` (note that the log for processing the value of '{}' comes after the previous error)
Looks like we needed to validate that the configuration string was in JSON format before continuing with the rest of the validation. Upstream pull request @ https://github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/pull/37
Merged upstream, downstream PRs: Into master @ https://github.com/openshift/multus-admission-controller/pull/21 For backport to 4.3 @ https://github.com/openshift/multus-admission-controller/pull/22
Hi Has this been merged? In such a case: can it be closed? -Alex
This has a tangle of BZ dependencies for the BZ bot. The PR is still pending (https://github.com/openshift/multus-admission-controller/pull/22), and I need to try to detangle the BZ deps