Description of problem: On OSP16.1 installed with default MTU settings sending UDP/ICMP packets from internal network to external with sizes higher than MTU results with 100% packet loss regardless of setting ovn_emit_need_to_frag=True in ml2. [heat-admin@controller-0 ~]$ sudo podman exec -it neutron_api crudini --get /etc/neutron/plugins/ml2/ml2_conf.ini ovn ovn_emit_need_to_frag True [heat-admin@controller-0 ~]$ sudo ovs-appctl -t ovs-vswitchd dpif/show-dp-features br-int | grep "Check pkt length action" Check pkt length action: Yes External network- flat, MTU 1500 Internal network geneve, MTU 1442 Made the following attempts: NETCAT (from VM on internal network to external network) $ cat 1400_bytes_file | nc -u 10.0.0.178 65000 File reached the external network successfully - OK $ cat 1600_bytes_file | nc -u 10.0.0.178 65000 File did not reach the external network - NOT OK PING - Size smaller than MTU, works - OK $ ping 10.0.0.178 -s 1400 PING 10.0.0.178 (10.0.0.178) 1400(1428) bytes of data. 1408 bytes from 10.0.0.178: icmp_seq=1 ttl=63 time=4.60 ms ^C --- 10.0.0.178 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 4.602/4.602/4.602/0.000 ms PING - Size bigger than MTU, does not work, regardless of -M dont, -M want settings - NOT OK $ ping 10.0.0.178 -s 1600 -M dont PING 10.0.0.178 (10.0.0.178) 1600(1628) bytes of data. From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1482) From 192.168.1.1 icmp_seq=2 Frag needed and DF set (mtu = 1482) ^C --- 10.0.0.178 ping statistics --- 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms $ ping 10.0.0.178 -s 1600 -M want PING 10.0.0.178 (10.0.0.178) 1600(1628) bytes of data. From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1482) From 192.168.1.1 icmp_seq=2 Frag needed and DF set (mtu = 1482) ^C --- 10.0.0.178 ping statistics --- 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1002ms maximal working size with ping 1414 The same scenarios were tested with OSP16.1 using ML2/OVS and same type of traffic was able to reach external network. Version-Release number of selected component (if applicable): 16.1-RHEL-8/RHOS-16.1-RHEL-8-20200505.n.0 How reproducible: 100% Steps to Reproduce: 1. Create external network, internal network, router, connect both networks to the router, keypair, security group with rules allowing icmp, ssh, udp. 2. Launch instance on the internal network using created keypair and security group 3. Using netcat try to send a UDP packet from the instance to the external network addreess which is smaller (e.g. 1400 bytes) and then bigger (e.g. 1600 bytes) than MTU size 4 Try to ping with same sizes Actual results: UDP/ICMP packets with size bigger than MTU do not reach external network Expected results: UDP/ICMP packets with size bigger than MTU reach external network successfully after fragmenting and reassembling Additional info:
puppet-ovn-15.4.1-0.20200311045730.192ac4e.el8ost.noarch python3-networking-ovn-7.1.1-0.20200505113427.071bd83.el8ost.noarch ovn2.13-2.13.0-18.el8fdp.x86_64 openvswitch2.13-2.13.0-18.el8fdp.x86_64
[heat-admin@controller-0 ~]$ ovn-nbctl show switch fef954a0-dfca-4a68-8b87-1c993a1ec2ce (neutron-3d88807d-1415-45e9-a9d5-9849c545d44b) (aka external) port fe4ea8f7-1163-44db-b1e4-60222155b430 addresses: ["fa:16:3e:f1:28:ed 10.0.0.178"] port provnet-3d88807d-1415-45e9-a9d5-9849c545d44b type: localnet addresses: ["unknown"] port 2e0ba96d-a448-41a9-892d-ae2d3a8d21e8 type: localport addresses: ["fa:16:3e:1d:44:e2 10.0.0.151"] port 05fbf01e-bb99-4129-8ddd-e2262fac3a7b type: router router-port: lrp-05fbf01e-bb99-4129-8ddd-e2262fac3a7b switch f366bc5f-166f-4e8c-a29e-ead50b5dfdc1 (neutron-040e120f-2743-4ab7-ae24-7864d21def9a) (aka internal_A) port 8537fd46-0eff-4d19-91b0-58031f18cfed type: router router-port: lrp-8537fd46-0eff-4d19-91b0-58031f18cfed port 3b68cc85-9fef-4485-8448-3fee5f4d83f4 addresses: ["fa:16:3e:f0:b7:a7 192.168.1.188"] port 877d4203-dff9-4968-bb58-990c3183a468 addresses: ["fa:16:3e:33:20:b9 192.168.1.84"] port 50086416-8719-4e7c-816a-d424b4f42d89 type: localport addresses: ["fa:16:3e:74:62:ec 192.168.1.2"] router 9586a768-8b0f-4c00-815e-ae05e94929f5 (neutron-ce6c4dbf-3698-4303-bd33-82f37f8975fd) (aka routerA) port lrp-05fbf01e-bb99-4129-8ddd-e2262fac3a7b mac: "fa:16:3e:92:da:29" networks: ["10.0.0.177/24"] gateway chassis: [9cb5b6d6-b468-47bc-a18c-0a9e4544ab72 0d2347bf-ae10-43c7-8a6c-7dad0e31205d 264ffaed-01d3-450b-8a7d-34b8211c5555] port lrp-8537fd46-0eff-4d19-91b0-58031f18cfed mac: "fa:16:3e:e2:8b:4c" networks: ["192.168.1.1/24"] nat 63687bcd-2304-4d13-9a07-ba9a41bf123f external ip: "10.0.0.177" logical ip: "192.168.1.0/24" type: "snat" nat efc57895-a5fa-4b2c-a110-9b706c10d918 external ip: "10.0.0.172" logical ip: "192.168.1.188" type: "dnat_and_snat" [heat-admin@controller-0 ~]$ ovn-nbctl list logical_router_port _uuid : 533d9460-88c5-494d-814a-67612688600e enabled : [] external_ids : {"neutron:network_name"=neutron-3d88807d-1415-45e9-a9d5-9849c545d44b, "neutron:revision_number"="8", "neutron:router_name"="ce6c4dbf-3698-4303-bd33-82f37f8975fd", "neutron:subnet_ids"="0b88748a-3c03-405d-a901-4e9ed3e4948d"} gateway_chassis : [0105d693-e120-41ee-adf0-1b2849736022, 080486c2-9e2b-45ee-959d-5c3bebe477b6, 6f6fcc99-258c-4d5d-921f-5400331d3ceb] ha_chassis_group : [] ipv6_prefix : [] ipv6_ra_configs : {} mac : "fa:16:3e:92:da:29" name : lrp-05fbf01e-bb99-4129-8ddd-e2262fac3a7b networks : ["10.0.0.177/24"] options : {gateway_mtu="1500"} peer : [] _uuid : a2cd2303-fa8a-405c-b3ce-70cd074f5595 enabled : [] external_ids : {"neutron:network_name"=neutron-040e120f-2743-4ab7-ae24-7864d21def9a, "neutron:revision_number"="3", "neutron:router_name"="ce6c4dbf-3698-4303-bd33-82f37f8975fd", "neutron:subnet_ids"="20ea6c57-d7d7-48bf-b373-6fef6180c8dc"} gateway_chassis : [] ha_chassis_group : [] ipv6_prefix : [] ipv6_ra_configs : {} mac : "fa:16:3e:e2:8b:4c" name : lrp-8537fd46-0eff-4d19-91b0-58031f18cfed networks : ["192.168.1.1/24"] options : {} peer : [] [heat-admin@controller-0 ~]$ ovn-nbctl list logical_switch _uuid : fef954a0-dfca-4a68-8b87-1c993a1ec2ce acls : [] dns_records : [57f7eda5-fc8c-47fb-921a-909b59b6e624] external_ids : {"neutron:mtu"="1500", "neutron:network_name"=external, "neutron:qos_policy_id"=null, "neutron:revision_number"="2"} forwarding_groups : [] load_balancer : [] name : neutron-3d88807d-1415-45e9-a9d5-9849c545d44b other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} ports : [15e9e7b4-b919-4fe8-a618-9211ea060f2e, 45f0234a-1319-4dad-bbd3-390f63a35e40, 8201a23b-ef36-4c67-927d-6dfafadd1fd7, cfb35313-f87c-4673-81aa-ff7ff31b7610] qos_rules : [] _uuid : f366bc5f-166f-4e8c-a29e-ead50b5dfdc1 acls : [] dns_records : [765572f9-eeb1-4b2d-91d6-c121bd78f696] external_ids : {"neutron:mtu"="1442", "neutron:network_name"=internal_A, "neutron:qos_policy_id"=null, "neutron:revision_number"="2"} forwarding_groups : [] load_balancer : [] name : neutron-040e120f-2743-4ab7-ae24-7864d21def9a other_config : {mcast_flood_unregistered="false", mcast_snoop="false"} ports : [14f7769c-6bab-4a58-b0fc-c11537382aad, 5b57f30f-2822-492d-b178-2bc758311725, 86cbfff9-0c6b-4c15-a3cd-501ffbe5c683, d17bcb48-6db3-4ee6-9586-559b7c459cbf] qos_rules : []
If the VMs are configured with 1442 MTU (geneve), then how can the packets greater than 1442 go out of the VM ? I don't know if it works with ML2OVS or not, but surely it will not work with OVN. We added the fragmentation support to address use case where the internal geneve networks are of jumbu mtus (i.e 8442), and the external network's MTU is lesser than that. @Daniel - When we worked on this in core OVN, this usecase is what we discussed right ? @can you please test out the scenario I mentioned above ? i.e internal networ MTU > external network MTU. If there is a requirement to support otherway round, I don't think it will work at the moment. Request to please raise RFE bugs for the OVN component. Thanks
Tested on RHOS-16.1-RHEL-8-20200625.n.0 with OVN from next FDP 20.E (not released yet) ovn2.13-2.13.0-37.el8fdp.x86_64 using a scratch kernel from https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=29756081 Tested with big ICMP and UDP packets (bigger than MTU of the external network) All works as expected according the following guidelines: DF=0 --> the device will learn the real MTU and after first pkt, the traffic flows DF=1 --> the router discard the traffic but the sender does not perform fragmentation so traffic does not flow Verified also that TCP traffic passes. Some details: [cloud-user@rhel-8-vm ~]$ ip route get 10.0.0.31 10.0.0.31 via 192.168.1.1 dev eth0 src 192.168.1.34 uid 1000 cache [cloud-user@rhel-8-vm ~]$ ping -s 8900 10.0.0.31 -c 5 PING 10.0.0.31 (10.0.0.31) 8900(8928) bytes of data. From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1482) 8908 bytes from 10.0.0.31: icmp_seq=2 ttl=63 time=2.22 ms 8908 bytes from 10.0.0.31: icmp_seq=3 ttl=63 time=1.49 ms 8908 bytes from 10.0.0.31: icmp_seq=4 ttl=63 time=1.31 ms 8908 bytes from 10.0.0.31: icmp_seq=5 ttl=63 time=1.21 ms --- 10.0.0.31 ping statistics --- 5 packets transmitted, 4 received, +1 errors, 20% packet loss, time 12ms rtt min/avg/max/mdev = 1.208/1.557/2.222/0.398 ms [cloud-user@rhel-8-vm ~]$ [cloud-user@rhel-8-vm ~]$ ip route get 10.0.0.31 10.0.0.31 via 192.168.1.1 dev eth0 src 192.168.1.34 uid 1000 cache expires 591sec mtu 1482