Description of problem: VMs' connectivity is available even the two VMs are in different vlan Version-Release number of selected component (if applicable): Client Version: 4.4.0-rc.6 Server Version: 4.4.0-rc.8 Kubernetes Version: v1.17.1 CNV2.3 How reproducible: Always Steps to Reproduce: 1. Create bridge on nodes --- apiVersion: nmstate.io/v1alpha1 kind: NodeNetworkConfigurationPolicy metadata: name: linux-bridge-test-connectivity spec: desiredState: interfaces: - name: br1test type: linux-bridge state: up bridge: options: stp: enabled: false port: [] 2. Create two net-attach-def with different vlan id apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br1test creationTimestamp: "2020-04-23T13:07:51Z" generation: 1 name: br1vlan200-nad namespace: connectivity-test-ovs-linux-bridge resourceVersion: "8011536" selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/connectivity-test-ovs-linux-bridge/network-attachment-definitions/br1vlan200-nad uid: e99df370-a304-4dd1-8634-15a61c0fec66 spec: config: '{"cniVersion": "0.3.1", "name": "br1test", "plugins": [{"type": "cnv-bridge", "bridge": "br1test"}], "vlan": 200}' apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br1test creationTimestamp: "2020-04-23T13:07:50Z" generation: 1 name: br1vlan300-nad namespace: connectivity-test-ovs-linux-bridge resourceVersion: "8011532" selfLink: /apis/k8s.cni.cncf.io/v1/namespaces/connectivity-test-ovs-linux-bridge/network-attachment-definitions/br1vlan300-nad uid: 8f8ab504-432f-4b22-b636-c733f3393b09 spec: config: '{"cniVersion": "0.3.1", "name": "br1test", "plugins": [{"type": "cnv-bridge", "bridge": "br1test"}], "vlan": 300}' 3. Create two VMs and attaced the multus network with different nad created in step2 4. Ping between the VMs Actual results: VMs' connectivity is available even the two VMs are in different vlan Expected results: The connectivity should be failed since the two VMs are in different vlan Additional info: The bridge vlan info is attached.
Created attachment 1681156 [details] bridge vlan
Based on the `bridge vlan` output, it seems that that those ports were indeed kept untagged and thus are connected. I suspect this is a bug in the CNI plugin. Yan, could you please try the same setup, but with "type": "bridge". To see if the CNI shipped by OCP gives better results?
Hi, Petr I tried with "type": "bridge", connectivity is still available, same result as "type": "cnv-bridge".
Created attachment 1681361 [details] bridge vlan_type:bridge
If I understand correctly the VMs are started at the same node, so we need the linux-bridge to have all the vlans range set at vlan-filtering on the bridge, but looks like we don't do that at release-0.15 bridge [1] [1] https://github.com/nmstate/kubernetes-nmstate/blob/release-0.15/build/bin/vlan-filtering Was this working before ?
Created attachment 1694430 [details] vlan info
Created attachment 1694431 [details] vm spec
So looks like linuxbridge-cni is not setting vlan filteright right at veth [core@host-172-16-0-36 ~]$ bridge vlan show port vlan ids br1test 1 PVID untagged veth2f041a7a 1 PVID untagged vethfd2980de 1 PVID untagged
Looks like we have an issue with NetworkAttachmentDefinition configuration plugins sections has to be removed they should look like this apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br1test name: br1vlan200-nad namespace: test1 spec: config: '{"cniVersion": "0.3.1", "name": "br1test", "type": "cnv-bridge", "bridge": "br1test", "vlan": 200}'
We need to keep the chain of plugins with "tuning" to support MTU/MAC configuration. If the absence of plugins makes a difference there must be another issue in the original definition. Ha, I think I see it. The "vlan" has to be in the bridge plugin configuration, not outside of it. config: '{"cniVersion": "0.3.1", "name": "br1test", "plugins": [{"type": "cnv-bridge", "bridge": "br1test"}], "vlan": 300}' should be config: '{"cniVersion": "0.3.1", "name": "br1test", "plugins": [{"type": "cnv-bridge", "bridge": "br1test", "vlan": 300}]}' (and it misses tuning)
After fixing the NAD as #comment9, connection is unreachable for the vms with different vlan. [fedora@vmb ~]$ ping 192.168.0.11 PING 192.168.0.11 (192.168.0.11) 56(84) bytes of data. From 192.168.0.12 icmp_seq=1 Destination Host Unreachable From 192.168.0.12 icmp_seq=2 Destination Host Unreachable From 192.168.0.12 icmp_seq=3 Destination Host Unreachable
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3194