Created attachment 1673161 [details] kubelet-crio.log Description of problem: I have specified an additionalNetwork containing "ipam": {"type": "static"}. When I try to add this network to a pod specifying a specific ip, the additional interface is silently ignore. The pod comes up with no additional network interfaces. The full pod definition is: === apiVersion: v1 kind: Pod metadata: name: busybox1 labels: app: busybox1 annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "osp-internalapi-static", "ips": "192.168.222.1/24" } ]' spec: containers: - image: busybox command: - sleep - "3600" imagePullPolicy: IfNotPresent name: busybox restartPolicy: Always === Note that I am not able to specify ips as an array in the above because it is rejected as invalid input. The full additionalNetworks stanza is: === - name: osp-internalapi-static namespace: default rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "bridge", "bridge": "br-ospinfra", "vlan": 100, "capabilities": { "ips": true }, "ipam": { "type": "static" } }' type: Raw === Note that if I add the IP address to the cni definition (in ipam.addresses) and remove it from the pod definition then the network is created as expected. In debugging, Tomofumi Hayashi asked me to try with the multus admission controller disabled, which I did with: === oc -n openshift-cluster-version scale --replicas=0 deploy/cluster-version-operator oc -n openshift-multus delete daemonset/multus-admission-controller === Unfortunately this didn't help. Reproduced, with logs: === oc get pod busybox1 -o yaml apiVersion: v1 kind: Pod metadata: annotations: k8s.ovn.org/pod-networks: '{"default":{"ip_address":"10.129.2.6/23","mac_address":"fa:30:ef:81:02:07","gateway_ip":"10.129.2.1"}}' k8s.v1.cni.cncf.io/networks: '[{"name":"osp-internalapi-static","namespace":"default","ips":"192.168.222.1/24","mac":"02:8c:0c:00:00:0d"}]' k8s.v1.cni.cncf.io/networks-status: |- [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.129.2.6" ], "mac": "fa:30:ef:81:02:07", "dns": {} }] creationTimestamp: "2020-03-24T15:58:32Z" labels: app: busybox1 name: busybox1 namespace: default resourceVersion: "4137705" selfLink: /api/v1/namespaces/default/pods/busybox1 uid: bf61b98f-d979-4d61-8780-424f7a6dcb0c spec: containers: - command: - sleep - "3600" image: busybox imagePullPolicy: IfNotPresent name: busybox resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-mw227 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: - name: default-dockercfg-rxcjv nodeName: worker-2 priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: default-token-mw227 secret: defaultMode: 420 secretName: default-token-mw227 status: conditions: - lastProbeTime: null lastTransitionTime: "2020-03-24T15:58:32Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2020-03-24T15:58:35Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2020-03-24T15:58:35Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2020-03-24T15:58:32Z" status: "True" type: PodScheduled containerStatuses: - containerID: cri-o://b7bc1fc4408acee83e31e1a7b2fc5493704606e053c3f4b3a7e45b45f195af94 image: docker.io/library/busybox:latest imageID: docker.io/library/busybox@sha256:afe605d272837ce1732f390966166c2afff5391208ddd57de10942748694049d lastState: {} name: busybox ready: true restartCount: 0 started: true state: running: startedAt: "2020-03-24T15:58:34Z" hostIP: 192.168.111.25 phase: Running podIP: 10.129.2.6 podIPs: - ip: 10.129.2.6 qosClass: BestEffort startTime: "2020-03-24T15:58:32Z" === === $ oc get net-attach-def/osp-internalapi-static -o yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: creationTimestamp: "2020-03-23T16:53:58Z" generation: 9 name: osp-internalapi-static namespace: default ownerReferences: - apiVersion: operator.openshift.io/v1 blockOwnerDeletion: true controller: true kind: Network name: cluster uid: a1738e2d-ddd8-43a2-bae9-0bfa68c635ac resourceVersion: "3997440" selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/osp-internalapi-static uid: fc764ba7-d6f4-4a14-8700-17a5b8a3983f spec: config: '{ "cniVersion": "0.3.1", "type": "bridge", "bridge": "br-ospinfra", "vlan": 100, "capabilities": { "ips": true }, "ipam": { "type": "static" } }' === The pod does not have a multus interface: === $ kubectl exec busybox1 ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 3: eth0@if79: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1400 qdisc noqueue link/ether fa:30:ef:81:02:07 brd ff:ff:ff:ff:ff:ff === The logs from worker-2 during the above are attached to this BZ as kubelet-crio.log. Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-03-17-091701 How reproducible: Always
It appears that the error is in the definition of the pod, but the valid syntax was being rejected by 2 separate admission controllers: multus-admission-controller and kubemacpool-mac-controller-manager. The rejected input is: === apiVersion: v1 kind: Pod metadata: name: busybox1 labels: app: busybox1 annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "osp-internalapi-static", "ips": [ "192.168.222.1/24" ] } ]' spec: containers: - image: busybox command: - sleep - "3600" imagePullPolicy: IfNotPresent name: busybox restartPolicy: Always === The failure is: === $ oc create -f busybox.yaml Error from server: error when creating "busybox.yaml": admission webhook "mutatepods.example.com" denied the request: parsePodNetworkAnnotation: failed to parse pod Network Attachment Selection Annotation JSON format: json: cannot unmarshal array into Go struct field NetworkSelectionElement.ips of type string === The workaround is to disable both admission controllers: oc -n openshift-cluster-version scale --replicas=0 deploy/cluster-version-operator oc -n openshift-network-operator scale --replicas=0 deploy/network-operator oc -n openshift-cnv scale --replicas=0 deploy/kubemacpool-mac-controller-manager oc -n openshift-multus delete daemonset/multus-admission-controller With this in place the above pod is created successfully, and the additional multus interface is added with the correct static IP.
Upstream PR: https://github.com/k8snetworkplumbingwg/net-attach-def-admission-controller/
This issue is actually comes from kubemacpool in the following bz. https://bugzilla.redhat.com/show_bug.cgi?id=1816971 Now CNV team removes kubemacpool in CNV2.3, so it is resolved.