Bug 2079891
| Summary: | [RFE] Add user MTU config option to increase mempool sharing | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Kevin Traynor <ktraynor> |
| Component: | openvswitch2.17 | Assignee: | Kevin Traynor <ktraynor> |
| Status: | CLOSED ERRATA | QA Contact: | liting <tli> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | FDP 22.C | CC: | ctrautma, dmarchan, hakhande, jhsiao, ralongi, tredaelli |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openvswitch3.0-3.0.0-0.3.el9fdp | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-11-21 18:19:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2004286 | ||
v1 patches submitted to upstream here: https://mail.openvswitch.org/pipermail/ovs-dev/2022-April/393300.html https://mail.openvswitch.org/pipermail/ovs-dev/2022-April/393301.html https://mail.openvswitch.org/pipermail/ovs-dev/2022-April/393302.html Upstream status: - Patches have been Reviewed-by David Marchand - Ian (OVS maintainer) has indicated he will review/test them - v5 patches: https://mail.openvswitch.org/pipermail/ovs-dev/2022-June/395140.html http://patchwork.ozlabs.org/project/openvswitch/list/?series=306367 This patchset has merged [1] in OVS upstream and is in the OVS 3.0 release. Will update Bz when this is available in downstream OVS. [1] https://github.com/openvswitch/ovs/commit/3757e9f8e9c3190cf981740fec662e84b12d07ef Adding some notes to help with validation and debug by showing a basic use case, where ports of different MTUs on the same NUMA share a single mempool. Also added what would be seen in debug logs in case anyone is debugging in the future, but this is optional. They key part is the ports with different MTUs share the same mempool. # Add an MTU that shared mempools will be based on $ ovs-vsctl --no-wait set Open_vSwitch . other_config:shared-mempool-config=9000 # Enable dpdk $ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true # Check that MTU size is stored |netdev_dpdk|INFO|User configured shared mempool set for: MTU 9000, NUMA ALL. # optional: enable debug for netdev_dpdk $ ovs-appctl vlog/set netdev_dpdk:dbg # Add a port (with default 1500 MTU) $ ovs-vsctl add-port br0 myport -- set Interface myport type=dpdk options:dpdk-devargs=0000:01:00.0 # Debug Logs: ensure that user configured shared mempool MTU is used |netdev_dpdk|DBG|Found user configured shared mempool with mbufs of size 9344, suitable for port with MTU 1500, NUMA 0. |netdev_dpdk|DBG|Allocated "ovs530eb21900093260262144" mempool with 262144 mbufs # Add another port (with default 1500 MTU) $ ovs-vsctl add-port br0 urport -- set Interface urport type=dpdk options:dpdk-devargs=0000:01:00.1 # Debug Logs: ensure that same mempool is used |netdev_dpdk|DBG|Found user configured shared mempool with mbufs of size 9344, suitable for port with MTU 1500, NUMA 0. |netdev_dpdk|DBG|Reusing mempool "ovs530eb21900093260262144" # Change MTU of *one* of these devices to a much higher MTU. Previously this would have seen the need for an additional mempool $ ovs-vsctl -- set interface myport mtu_request=9000 # Debug Logs: check same mempool used |netdev_dpdk|DBG|Found user configured shared mempool with mbufs of size 9344, suitable for port with MTU 9000, NUMA 0. |netdev_dpdk|DBG|Reusing mempool "ovs530eb21900093260262144" # At this point myport has MTU 9000 and urport has MTU 1500 # Ensure that these ports with significantly different MTUs are sharing the same mempool $ ovs-appctl netdev-dpdk/get-mempool-info myport | grep 'mempool <ovs' mempool <ovs530eb21900093260262144>@0x17f155a00 $ ovs-appctl netdev-dpdk/get-mempool-info urport | grep 'mempool <ovs' mempool <ovs530eb21900093260262144>@0x17f155a00 Hi Kevin, I run the testing on openvswitch3.0-3.0.0-3.el9fdp, and the following is the steps and result. The debug log "No user configured shared mempool mbuf sizes found suitable" is not same as your debug log on #comment6. But the mempool ovs62a2ca2f00093260262144 are same. I'm not sure if it verify pass. Can you help me have a look? Thanks. [root@dell-per730-56 ~]# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-shared-memory-config=9000 [root@dell-per730-56 ~]# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true [root@dell-per730-56 ~]# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024,1024 [root@dell-per730-56 ~]# ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x400000400000 [root@dell-per730-56 ~]# ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev [root@dell-per730-56 ~]# ovs-appctl vlog/set netdev_dpdk:dbg [root@dell-per730-56 ~]# ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk ofport_request=10 options:dpdk-devargs=0000:07:00.0 Debug log: 2022-11-01T03:34:34.384Z|00090|netdev_dpdk|DBG|No user configured shared mempool mbuf sizes found suitable for port with MTU 1500, NUMA 0. 2022-11-01T03:34:34.384Z|00091|netdev_dpdk|DBG|Port dpdk0: Requesting a mempool of 262144 mbufs of size 2176 on socket 0 for 1 Rx and 3 Tx queues, cache line size of 64 2022-11-01T03:34:34.435Z|00001|dpdk(pmd-c46/id:7)|INFO|PMD thread uses DPDK lcore 1. 2022-11-01T03:34:34.445Z|00001|dpdk(pmd-c22/id:8)|INFO|PMD thread uses DPDK lcore 2. 2022-11-01T03:34:34.461Z|00092|netdev_dpdk|DBG|Allocated "ovs62a2ca2f00021580262144" mempool with 262144 mbufs 2022-11-01T03:34:35.915Z|00093|netdev_dpdk|INFO|Port 2: 04:3f:72:ad:27:8a [root@dell-per730-56 ~]# ovs-appctl netdev-dpdk/get-mempool-info dpdk0 | grep 'mempool <ovs' mempool <ovs62a2ca2f00021580262144>@0x17e7337c0 Then set dpdk0 mtu to 9000 [root@dell-per730-56 ~]# ovs-vsctl -- set interface dpdk0 mtu_request=9000 Debug log: 2022-11-01T03:35:43.233Z|00141|netdev_dpdk|DBG|No user configured shared mempool mbuf sizes found suitable for port with MTU 9000, NUMA 0. 2022-11-01T03:35:43.233Z|00142|netdev_dpdk|DBG|Port dpdk0: Requesting a mempool of 262144 mbufs of size 9344 on socket 0 for 1 Rx and 3 Tx queues, cache line size of 64 2022-11-01T03:35:43.987Z|00143|netdev_dpdk|DBG|Allocated "ovs62a2ca2f00093260262144" mempool with 262144 mbufs 2022-11-01T03:35:45.036Z|00144|netdev_dpdk|INFO|Port 2: 04:3f:72:ad:27:8a [root@dell-per730-56 ~]# ovs-appctl netdev-dpdk/get-mempool-info dpdk0 | grep 'mempool <ovs' mempool <ovs62a2ca2f00093260262144>@0x14eec9e80 thanks, Li Ting (In reply to liting from comment #7) > Hi Kevin, > Hi Li Ting, > I run the testing on openvswitch3.0-3.0.0-3.el9fdp, and the following is the > steps and result. The debug log "No user configured shared mempool mbuf > sizes found suitable" is not same as your debug log on #comment6. But the > mempool ovs62a2ca2f00093260262144 are same. I'm not sure if it verify pass. > Can you help me have a look? Thanks. > > [root@dell-per730-56 ~]# ovs-vsctl --no-wait set Open_vSwitch . > other_config:dpdk-shared-memory-config=9000 The command is: ovs-vsctl --no-wait set Open_vSwitch . other_config:shared-mempool-config=9000 The other form was used in initial patches but changed on upstream review. I changed to ovs-vsctl --no-wait set Open_vSwitch . other_config:shared-mempool-config=9000, and verified it pass. #mtu 1500 debug log 2022-11-02T09:32:51.435Z|00096|netdev_dpdk|DBG|Found user configured shared mempool with mbufs of size 9344, suitable for port with MTU 1500, NUMA 0. 2022-11-02T09:32:51.435Z|00097|netdev_dpdk|DBG|Port dpdk0: Requesting a mempool of 262144 mbufs of size 9344 on socket 0 for 1 Rx and 3 Tx queues, cache line size of 64 2022-11-02T09:32:51.435Z|00001|dpdk(pmd-c46/id:32)|INFO|PMD thread uses DPDK lcore 2. 2022-11-02T09:32:52.503Z|00098|netdev_dpdk|DBG|Allocated "ovs62a2ca2f00093260262144" mempool with 262144 mbufs [root@dell-per730-56 ovs-dpdk-rss]# ovs-appctl netdev-dpdk/get-mempool-info dpdk0 | grep 'mempool <ovs' mempool <ovs62a2ca2f00093260262144>@0x17e7337c0 #mtu 9000 debug log: 2022-11-02T09:34:02.189Z|00147|netdev_dpdk|DBG|Found user configured shared mempool with mbufs of size 9344, suitable for port with MTU 9000, NUMA 0. 2022-11-02T09:34:02.189Z|00148|netdev_dpdk|DBG|Reusing mempool "ovs62a2ca2f00093260262144" [root@dell-per730-56 ovs-dpdk-rss]# ovs-appctl netdev-dpdk/get-mempool-info dpdk0 | grep 'mempool <ovs' mempool <ovs62a2ca2f00093260262144>@0x17e7337c0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (openvswitch2.17 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8567 |
This RFE is to allow the user give an MTU hint which can then be used to increase sharing of mempools between port based on different MTUs. From patches cover-letter: This patchset optimizes for two cases when using shared mempools. If there are ports with different MTUs, that usually leads to multiple shared mempools being created because mempool mbuf size and hence creation is based from MTU. In fact, a port with a smaller MTU could share a mempool with mbuf sizes that can accomodate larger MTUs (assuming same NUMA). So instead of multiple shared mempools being created based on MTU, the ports can use a single shared mempool. Another issue optimized for is that if there is an intended MTU for ports of say 9000, but the port is initially added without MTU specified, then it will fall back to a default MTU of 1500. As it is not mandatory for the user to set MTU, it cannot be assumed that a new MTU will be set after a port is added, so mempools based on an MTU of 1500 are used. When the MTU is subsequently set to 9000, the 1500 mempool will not be needed and may be freed, but during the in-between time both mempools are required. Both these cases can be optimized for. However, to just switch to increased mbuf size based mempools automatically could lead to increased memory consumption and break upgrades depending on config. So the user should give a hint about the MTUs they want the mempool mbufs size to be based on. While it is flexible for multiple sizes and numa, it would most likely be used with a single value. e.g. $ ovs-vsctl --no-wait set Open_vSwitch . \ other_config:dpdk-shared-memory-config=9000 With this all dpdk ports will share mempools on the relevant NUMA with an mbuf size based on MTU 9000.