Bug 1552465
| Summary: | High TRex packets loss during live migration over ovs+dpdk+vhost-user | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Pei Zhang <pezhang> | ||||||
| Component: | openvswitch | Assignee: | Maxime Coquelin <maxime.coquelin> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Pei Zhang <pezhang> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 7.5 | CC: | ailan, atragler, chayang, echaudro, juzhang, lmiksik, maxime.coquelin, michen, pezhang, siliu, tredaelli | ||||||
| Target Milestone: | rc | Keywords: | Regression | ||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | openvswitch-2.9.0-38.el7fdp | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 1568678 (view as bug list) | Environment: | |||||||
| Last Closed: | 2018-06-21 13:36:35 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1405177 [details]
XML of VM
Additional info (continued): 2. This should not be qemu-kvm-rhev bug. As with below versions, the TRex packets loss looks expected: Versions: openvswitch-2.7.1-1.git20170710.el7fdp.x86_64 qemu-kvm-rhev-2.10.0-21.el7.x86_64 Results: With below 20 migration runs, the TRex packets loss is around 0.4 million. ===========Stream Rate: 1Mpps=========== No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 1Mpps 123 12686 15 293542.0 1 1Mpps 123 13641 14 461975.0 2 1Mpps 120 13835 14 461714.0 3 1Mpps 121 13355 15 282249.0 4 1Mpps 118 13845 14 459771.0 5 1Mpps 124 14509 15 468748.0 6 1Mpps 123 13485 15 295984.0 7 1Mpps 118 12848 14 280997.0 8 1Mpps 119 14236 14 458436.0 9 1Mpps 123 13981 15 292635.0 ===========Stream Rate: 1Mpps=========== No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 1Mpps 118 13183 13 467103.0 1 1Mpps 127 13688 15 469440.0 2 1Mpps 125 13677 18 530158.0 3 1Mpps 124 13279 15 293528.0 4 1Mpps 121 13703 15 469770.0 5 1Mpps 110 14318 14 470910.0 6 1Mpps 119 14223 13 460204.0 7 1Mpps 114 13799 15 473836.0 8 1Mpps 123 12392 16 309198.0 9 1Mpps 122 14100 14 459757.0 Hi Pei, Just to confirm, I guess this is without IOMMU support enabled for the virtio devices? Thanks, Maxime (In reply to Maxime Coquelin from comment #5) > Hi Pei, > > Just to confirm, I guess this is without IOMMU support enabled for the > virtio devices? Hi Maxime, you are right, this is testing without IOMMU support enabled. (1)ovs doesn't enable IOMMU support. (2)VM XML doesn't enable IOMMU support. (3)In VM, load vfio with noiommu mode: # modprobe vfio enable_unsafe_noiommu_mode=Y Best Regards, Pei > Thanks, > Maxime Hi Pei, I took this BZ, as it looks familiar to BZ1512463, which I was about to request if you can replicate this with the latest 2.9.0-8. However looking at the details above this is already the same setup, however, you now get different results. Am I correct? If so, can you close the other BZ, as it was for 2.8, which we will not ship, and let me access your replication and figure out where this issue got introduced? Maxime if you have any other ideas, please let me know. (In reply to Eelco Chaudron from comment #7) > Hi Pei, I took this BZ, as it looks familiar to BZ1512463, which I was about > to request if you can replicate this with the latest 2.9.0-8. Hi Eelco, With openvswitch-2.9.0-8.el7fdn.x86_64, still hit this issue. > However looking at the details above this is already the same setup, > however, you now get different results. Am I correct? These 2 bugs are using same setup. Actually I don't quite sure if they are same issues, here are my concerns: (1) For Bug 1512463, the ping network can totally not recover for 1 ~ 3 seconds in some migration runs, not always. (2) Regarding to this bug, above ping issue still exists. The difference is when the ping works well, however still hit high TRex packets loss in some migration run. For example, in below No.13 and No.14 migration runs, when the ping loss is unexpected 112, the trex loss is high; but when ping loss is normal as 14, the trex loss is still high. ===========Stream Rate: 1Mpps=========== No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss ... 13 1Mpps 120 17604 112 2627539.0 14 1Mpps 128 17833 14 12018787.0 > If so, can you close the other BZ, as it was for 2.8, which we will not > ship, and let me access your replication and figure out where this issue got > introduced? I'll close Bug 1512463 as duplicate of this one. After this one is fixed, if ping issue still exists, then let's re-open it. Is this OK? My hosts are available now and I'll update them in next comment. Please just wait a sec. Thanks, Pei > > > Maxime if you have any other ideas, please let me know. *** Bug 1512463 has been marked as a duplicate of this bug. *** Did some experimenting and came to the conclusion that this additional delay got introduced due to the following patch (on dpdk 16.11.3):
commit 6bf02ab821fb02033b07e4d194811648ce8c30f0
Author: Tiwei Bie <tiwei.bie>
Date: Tue Aug 1 17:01:21 2017 +0800
vhost: make page logging atomic
[ backported from upstream commit 897f13a1f726cefdc68762da83f9d2225a85c27e ]
Each dirty page logging operation should be atomic. But it's not
atomic in current implementation. So it's possible that some dirty
pages can't be logged successfully when different threads try to
log different pages into the same byte of the log buffer concurrently.
This patch fixes this issue.
Fixes: b171fad1ffa5 ("vhost: log used vring changes")
Reported-by: Xiao Wang <xiao.w.wang>
Signed-off-by: Tiwei Bie <tiwei.bie>
Reviewed-by: Maxime Coquelin <maxime.coquelin>
When I undo this patch, the numbers are ok:
No Stream_Rate Downtime Totaltime Ping_Loss moongen_Loss
0 1Mpps 141 13360 16 502976
1 1Mpps 134 12820 16 489831
2 1Mpps 143 14325 17 509406
3 1Mpps 136 12830 15 489608
4 1Mpps 142 14442 16 502628
5 1Mpps 126 13286 15 470381
6 1Mpps 139 14251 16 495002
7 1Mpps 134 12933 15 479034
8 1Mpps 134 12607 16 485175
9 1Mpps 131 13340 15 479923
10 1Mpps 133 14514 15 482964
11 1Mpps 133 13312 18 548132
12 1Mpps 141 13771 16 499937
13 1Mpps 145 12562 17 509381
14 1Mpps 137 13595 15 487223
15 1Mpps 147 14436 17 510727
Also removed the patch from 2.9, however there the high traffic loss with the low TTL timeout is gone. However, I still see the high TTL issue reported in BZ1512463. I'll try to pinpoint where this got introduced, as this was first seen in an OVS 2.8 beta. ===========Stream Rate: 1Mpps=========== No Stream_Rate Downtime Totaltime Ping_Loss moongen_Loss 0 1Mpps 144 12656 16 506396 1 1Mpps 122 14309 114 2393523 2 1Mpps 140 14375 16 500156 3 1Mpps 136 12959 16 487447 4 1Mpps 152 14027 17 518767 5 1Mpps 130 12663 16 479817 6 1Mpps 125 14340 114 2398783 7 1Mpps 131 14174 15 440981 8 1Mpps 137 14274 15 492664 9 1Mpps 140 13849 16 492308 10 1Mpps 556 14029 58 1292156 11 1Mpps 137 14642 115 2483477 12 1Mpps 127 14226 15 478721 13 1Mpps 132 12717 15 415200 14 1Mpps 132 13583 14 478556 15 1Mpps 146 12933 16 506331 Patch accepted upstream:
commit 394313fff39d0f994325c47f7eab39daf5dc9e11
Author: Maxime Coquelin <maxime.coquelin>
Date: Wed Mar 21 16:44:13 2018 +0100
vhost: avoid concurrency when logging dirty pages
This patch aims at fixing a migration performance regression
faced since atomic operation is used to log pages as dirty when
doing live migration.
Instead of setting a single bit by doing an atomic read-modify-write
operation to log a page as dirty, this patch write 0xFF to the
corresponding byte, and so logs 8 page as dirty.
The advantage is that it avoids concurrent atomic operations by
multiple PMD threads, the drawback is that some clean pages are
marked as dirty and so are transferred twice.
Fixes: 897f13a1f726 ("vhost: make page logging atomic")
Cc: stable
Signed-off-by: Maxime Coquelin <maxime.coquelin>
Reviewed-by: Jianfeng Tan <jianfeng.tan>
Hi Timothy, Patch 394313fff39d0f994325c47f7eab39daf5dc9e11 is in upstream master branch now. It backports without conflict on top of DPDK v17.11 LTS. Can you pick it directly? Thanks in advance, Maxime Hi Timothy, Please wait before backporting. Pei did some more testing this morning, and face issues when more than one queue pair. It needs more investigation, but I think this is because this patch marks 8 pages at a time as dirty to avoid contention, but the problem seems to be that it never converges with multiple queue pairs. Commit now available upstream (v18.04-rc5):
commit c16915b8710911a75f0fbdb1aa5243f4cdfaf26a
Author: Maxime Coquelin <maxime.coquelin>
Date: Thu May 17 13:44:47 2018 +0200
vhost: improve dirty pages logging performance
This patch caches all dirty pages logging until the used ring index
is updated.
The goal of this optimization is to fix a performance regression
introduced when the vhost library started to use atomic operations
to set bits in the shared dirty log map. While the fix was valid
as previous implementation wasn't safe against concurrent accesses,
contention was induced.
With this patch, during migration, we have:
1. Less atomic operations as only a single atomic OR operation
per 32 or 64 (depending on CPU) pages.
2. Less atomic operations as during a burst, the same page will
be marked dirty only once.
3. Less write memory barriers.
Fixes: 897f13a1f726 ("vhost: make page logging atomic")
Cc: stable
Suggested-by: Michael S. Tsirkin <mst>
Signed-off-by: Maxime Coquelin <maxime.coquelin>
Reviewed-by: Tiwei Bie <tiwei.bie>
Summary: This bug has been fixed very well. Live migration with vhost-user single queue and 2 queues both work well as expected. (Bug 1512463 is tracking the TTL issue). Thank you all, Maxime, Eelco. Versions: kernel-3.10.0-887.el7.x86_64 qemu-kvm-rhev-2.12.0-2.el7.x86_64 libvirt-3.9.0-14.el7.x86_64 openvswitch-2.9.0-38.el7fdn.x86_64 dpdk-17.11-10.el7fdb.x86_64 Testing Results: (1) vhost-user single queues (ovs is vhost-user client): =======================Stream Rate: 1Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 1Mpps 146 14061 15 473762.0 1 1Mpps 137 16205 213 4446881.0 2 1Mpps 144 14370 15 460195.0 3 1Mpps 159 14257 15 489399.0 4 1Mpps 148 14502 15 469131.0 5 1Mpps 149 13551 15 471911.0 6 1Mpps 151 13901 15 308662.0 7 1Mpps 144 13853 15 467519.0 8 1Mpps 148 14065 15 476304.0 9 1Mpps 157 14241 16 485533.0 =======================Stream Rate: 2Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 2Mpps 145 13993 15 819557.0 1 2Mpps 145 14092 15 805606.0 2 2Mpps 149 14269 16 962528.0 3 2Mpps 155 14078 17 964581.0 4 2Mpps 144 14070 15 922834.0 5 2Mpps 151 14096 15 950100.0 6 2Mpps 142 13429 14 914852.0 7 2Mpps 153 13463 15 962724.0 8 2Mpps 148 14414 16 950910.0 9 2Mpps 148 14215 15 817256.0 =======================Stream Rate: 3Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 3Mpps 159 14421 15 1478081.0 1 3Mpps 137 14440 112 7177375.0 2 3Mpps 149 13880 15 954313.0 3 3Mpps 139 14261 15 1222838.0 4 3Mpps 154 13637 15 1448481.0 5 3Mpps 151 14140 14 1309262.0 6 3Mpps 140 14849 114 7332040.0 7 3Mpps 134 15950 212 13318512.0 8 3Mpps 130 14866 112 7288058.0 9 3Mpps 137 15312 114 7270321.0 =======================Stream Rate: 4Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 4Mpps 156 14977 16 2040846.0 1 4Mpps 152 14641 16 1581355.0 2 4Mpps 140 15550 113 9823552.0 3 4Mpps 154 14624 16 1964888.0 4 4Mpps 136 14475 15 1876555.0 5 4Mpps 127 15926 113 9823355.0 6 4Mpps 144 14802 15 1911330.0 7 4Mpps 132 15589 113 9792719.0 8 4Mpps 153 14807 15 1949005.0 9 4Mpps 144 14678 15 1911219.0 (2)vhost-user 2 queues(ovs is vhost-user client): =======================Stream Rate: 1Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 1Mpps 158 26346 20 397298.0 1 1Mpps 144 23397 117 2471128.0 2 1Mpps 147 23234 117 2467102.0 3 1Mpps 153 24428 18 493740.0 4 1Mpps 157 21885 18 484966.0 5 1Mpps 149 18535 18 480116.0 6 1Mpps 155 22244 19 511991.0 7 1Mpps 163 15530 20 510145.0 8 1Mpps 144 23914 117 2459891.0 9 1Mpps 137 18231 118 2471084.0 =======================Stream Rate: 2Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 2Mpps 162 15549 20 1111763.0 1 2Mpps 149 15924 19 1034097.0 2 2Mpps 150 15702 19 994752.0 3 2Mpps 148 15402 19 1024076.0 4 2Mpps 149 15838 19 969778.0 5 2Mpps 152 15390 19 974624.0 6 2Mpps 139 16500 117 4977939.0 7 2Mpps 151 15490 19 1034991.0 8 2Mpps 153 15425 18 978547.0 9 2Mpps 152 15465 18 1029640.0 =======================Stream Rate: 3Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 3Mpps 166 15766 20 1586279.0 1 3Mpps 158 15005 20 1513520.0 2 3Mpps 161 15509 20 1541615.0 3 3Mpps 151 14843 18 1640205.0 4 3Mpps 155 15365 19 1654094.0 5 3Mpps 164 15198 20 1547613.0 6 3Mpps 151 15267 18 1501002.0 7 3Mpps 160 14803 20 1533560.0 8 3Mpps 138 16774 118 7463825.0 9 3Mpps 156 14887 19 1647701.0 =======================Stream Rate: 4Mpps========================= No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 4Mpps 156 15863 20 2038635.0 1 4Mpps 156 14483 18 2014078.0 2 4Mpps 153 15757 19 2024704.0 3 4Mpps 155 14867 19 2008493.0 4 4Mpps 152 15581 18 1992717.0 5 4Mpps 154 15066 20 2060573.0 6 4Mpps 143 16586 117 10077407.0 7 4Mpps 145 14632 18 1933209.0 8 4Mpps 158 15764 19 2031230.0 9 4Mpps 151 15856 118 9960544.0 Move this bug to 'VERIFIED' based on Comment 25. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1962 |
Created attachment 1405175 [details] TRex server log Description of problem: There are very high TRex packets loss during nfv live migration, mostly it's 10~20 times loss then expected loss. Version-Release number of selected component (if applicable): kernel-3.10.0-855.el7.x86_64 libvirt-3.9.0-13.el7.x86_64 dpdk-17.11-7.el7.x86_64 microcode-20180108.tgz qemu-kvm-rhev-2.10.0-21.el7.x86_64 openvswitch-2.9.0-1.el7fdb.x86_64 How reproducible: 13/16 Steps to Reproduce: 1. Boot ovs in src and des host, see reference[1] 2. Boot VM 3. Load vfio and start testpmd in VM 4. Start TRex as packets generator in the third host, see reference [4] Default paramaters: Traffic Generator: TRex Frame Size: 64Byte Bidirectional: Yes Stream; 1Mpps CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz NIC: 10-Gigabit X540-AT2 5. Do live migration from src to des host # /bin/virsh migrate --verbose --persistent --live rhel7.5_nonrt qemu+ssh://192.168.1.2/system 6. Check migration results during live migration, some runs get high TRex loss which are not expected, it's even up to 12 million packets. ===========Stream Rate: 1Mpps=========== No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 1Mpps 128 16567 15 6937850.0 1 1Mpps 134 17254 16 5421912.0 2 1Mpps 115 15879 14 5167824.0 3 1Mpps 121 16385 15 1075464.0 4 1Mpps 121 16320 14 1129379.0 5 1Mpps 137 17547 16 622630.0 6 1Mpps 130 15560 15 7938009.0 7 1Mpps 129 18414 14 6715591.0 8 1Mpps 118 16478 15 7467388.0 9 1Mpps 132 17640 16 3896654.0 10 1Mpps 129 16090 15 4293640.0 11 1Mpps 128 16116 14 597561.0 12 1Mpps 129 14150 15 477036.0 13 1Mpps 120 17604 112 2627539.0 14 1Mpps 128 17833 14 12018787.0 15 1Mpps 129 17099 15 1964834.0 After checking the TRex server statistics of 1 migration run, the TRex terminal shows that there are high dorp rate(about 220Mbps drop) which lasting 13 seconds during live migration. current_time drop_rate ... 14.1 sec 0.00 bps 14.6 sec 0.00 bps 15.1 sec 0.00 bps 15.6 sec 168.30 Mbps 16.1 sec 168.30 Mbps 16.6 sec 195.18 Mbps 17.1 sec 195.18 Mbps 17.6 sec 205.48 Mbps 18.1 sec 205.48 Mbps 18.6 sec 210.59 Mbps 19.1 sec 210.59 Mbps 19.6 sec 218.94 Mbps 20.1 sec 218.94 Mbps 20.6 sec 211.80 Mbps 21.1 sec 211.80 Mbps 21.6 sec 214.75 Mbps 22.1 sec 214.75 Mbps 22.6 sec 217.30 Mbps 23.1 sec 217.30 Mbps 23.7 sec 215.12 Mbps 24.2 sec 215.12 Mbps 24.7 sec 209.03 Mbps 25.2 sec 209.03 Mbps 25.7 sec 205.19 Mbps 26.2 sec 205.19 Mbps 26.7 sec 207.24 Mbps 27.2 sec 207.24 Mbps 27.7 sec 181.60 Mbps 28.2 sec 181.60 Mbps 28.7 sec 0.00 bps 29.2 sec 0.00 bps 29.7 sec 0.00 bps 30.2 sec 0.00 bps ... The full TRex server log is attached to this Description. Actual results: High TRex packets loss during live migration. Expected results: The TRex packets loss should looks like, should be around 0.4 million packets loss. This was testing with below versions: qemu-kvm-rhev-2.9.0-16.el7.x86_64 openvswitch-2.7.1-1.git20170710.el7fdp.x86_64 No Stream_Rate Downtime Totaltime Ping_Loss trex_Loss 0 1Mpps 151 12898 16 352527.0 1 1Mpps 152 13028 16 350399.0 3 1Mpps 147 12759 17 339891.0 4 1Mpps 141 14180 16 465830.0 5 1Mpps 168 12824 17 407926.0 6 1Mpps 165 13483 20 483641.0 7 1Mpps 167 12574 20 410246.0 8 1Mpps 173 12865 18 413503.0 9 1Mpps 168 12574 16 405473.0 10 1Mpps 176 13175 18 417233.0 11 1Mpps 157 13308 15 381677.0 12 1Mpps 165 13058 26 562075.0 13 1Mpps 168 13150 15 402580.0 14 1Mpps 169 13535 17 475747.0 Additional info: 1. This should be regression bugs as above "Expected results" showed. Reference: [1] # ovs-vsctl show 8e578a3b-ba64-4dab-a9e1-80c37b9811dd Bridge "ovsbr1" Port "dpdk2" Interface "dpdk2" type: dpdk options: {dpdk-devargs="0000:06:00.0", n_rxq="1", n_txq="1"} Port "ovsbr1" Interface "ovsbr1" type: internal Port "vhost-user2" Interface "vhost-user2" type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser2.sock"} Bridge "ovsbr0" Port "vhost-user0" Interface "vhost-user0" type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser0.sock"} Port "dpdk1" Interface "dpdk1" type: dpdk options: {dpdk-devargs="0000:04:00.1", n_rxq="1", n_txq="1"} Port "dpdk0" Interface "dpdk0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="1", n_txq="1"} Port "vhost-user1" Interface "vhost-user1" type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser1.sock"} Port "ovsbr0" Interface "ovsbr0" type: internal [4] # cat start_packets_flow.sh DIRECTORY=~/src/lua-trafficgen cd $DIRECTORY ./binary-search.py \ --traffic-generator=trex-txrx \ --validation-runtime=60 \ --rate-unit=mpps \ --rate=1 \ --run-bidirec=1 \ --run-revunidirec=0 \ --frame-size=64 \ --num-flows=1024 \ --one-shot=1 \