Bug 625307 - Big drop in UDP throughput from external or host to guest & from guest to guest .
Big drop in UDP throughput from external or host to guest & from guest to ...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.1
All Linux
low Severity medium
: rc
: ---
Assigned To: Alex Williamson
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-08-18 22:50 EDT by Quan Wenli
Modified: 2010-08-29 23:38 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-27 09:28:14 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Quan Wenli 2010-08-18 22:50:39 EDT
Description of problem:

Big drop in UDP throughput from external or host to guest  & from guest to guest
.There are no drop from guest to external.


Here is the result form virtio-net driver,e1000 ,vhost driver also met this problem.

external -> guest
---------------------

UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.13 (192.168.0.13) port 0 AF_INET
Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

124928      32   10.00     4066640      0      104.1     12.55    146.153
124928           10.00     2198349              56.3     36.14    105.209

124928      64   10.00     3758439      0      192.4     11.28    81.587
124928           10.00     1769237              90.6     32.76    59.254

124928     128   10.00     3396788      0      347.8     11.06    43.569
124928           10.00     1625423             166.4     32.63    32.124

124928     256   10.00     2649223      0      542.5     11.08    13.632
124928           10.00     2601442             532.7     44.31    13.628

124928     512   10.00     1930513      0      790.7     8.92     7.412 
124928           10.00     1925589             788.7     23.14    4.808 

124928    1024   10.00     1115533      0      913.8     5.41     3.885 
124928           10.00     1114605             913.0     10.40    1.866 

124928    2048   10.00      567280      0      929.4     4.06     2.867 
124928           10.00      566677             928.4     11.12    1.963 

124928    4096   10.00      290456      0      951.7     2.51     1.732 
124928           10.00      290220             950.9     7.79     1.343 

124928    8192   10.00      145448      0      953.2     2.11     1.455 
124928           10.00      145302             952.2     8.75     1.506 

124928   16834   10.00       71146      0      958.1     2.06     1.410 
124928           10.00       71127             957.8     7.84     1.341 


Host -> guest
-----------------

UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.13 (192.168.0.13) port 0 AF_INET
Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

124928      32   10.00     2344341      0       60.0     33.39    392.376
124928           10.00     2178873              55.8     40.82    119.910

124928      64   10.00     2708441      0      138.7     37.38    270.417
124928           10.00     1769303              90.6     33.61    60.794

124928     128   10.00     2864850      0      293.3     41.61    109.641
124928           10.00     2428843             248.7     57.91    38.153

124928     256   10.00     1733841      0      355.1     33.32    61.870
124928           10.00     1723375             352.9     25.04    11.626

124928     512   10.00     2193392      0      898.4     32.63    29.886
124928           10.00     1746863             715.5     36.40    8.336 

124928    1024   10.00     2894321      0     2370.9     37.02    20.069
124928           10.00     1475779            1208.9     44.54    6.037 

124928    2048   10.00     3259130      0     5339.6     39.50    12.278
124928           10.00     1286732            2108.1     43.09    3.349 

124928    4096   10.00     1162544      0     3809.2     26.27    4.544 
124928           10.00     1156170            3788.3     3.74     0.162 

124928    8192   10.00      985174      0     6455.9     30.39    3.092 
124928           10.00      982920            6441.2     21.49    0.547 

124928   16834   10.00      642219      0     8648.5     32.48    2.510 
124928           10.00      629777            8481.0     21.62    0.418 


guestA1 - guestA2 (A1,A2 on the same host)

-------------
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.14 (192.168.0.14) port 0 AF_INET
Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

124928      32   10.00     1386075      0       35.5     50.00    231.367
124928           10.00     1383109              35.4     25.84    119.561

124928      64   10.00     1691710      0       86.6     49.95    104.089
124928           10.00     1535628              78.6     26.22    54.642

124928     128   10.00     2102085      0      215.3     50.00    46.298
124928           10.00     1727961             176.9     45.81    42.415

124928     256   10.00     1479318      0      303.0     50.10    28.056
124928           10.00     1428607             292.6     26.46    14.820

124928     512   10.00     1566892      0      641.8     50.10    12.871
124928           10.00     1557013             637.7     30.29    7.782 

124928    1024   10.00     1418685      0     1162.2     50.00    7.266 
124928           10.00     1376221            1127.4     25.35    3.684 

124928    2048   10.00     1044395      0     1711.1     50.00    5.051 
124928           10.00      989970            1621.9     22.51    2.274 

124928    4096   10.00      998491      0     3271.7     46.46    2.847 
124928           10.00      815880            2673.3     27.75    1.701 

124928    8192   10.00      778581      0     5102.1     44.66    1.983 
124928           10.00      563265            3691.2     28.30    1.256 

124928   16834   10.00      396312      0     5337.0     49.50    1.618 
124928           10.00      372103            5011.0     29.04    0.949 

guestA1 - guestB1 (A1 B1 on the respective host)

UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.15 (192.168.0.15) port 0 AF_INET
Socket  Message  Elapsed      Messages                   CPU      Service
Size    Size     Time         Okay Errors   Throughput   Util     Demand
bytes   bytes    secs            #      #   10^6bits/sec % SS     us/KB

124928      32   10.00     1380907      0       35.4     49.95    232.378
124928           10.00     1375729              35.2     23.62    109.865

124928      64   10.00     1465999      0       75.1     50.00    109.443
124928           10.00     1462024              74.9     23.80    52.101

124928     128   10.00     1516719      0      155.3     50.00    52.943
124928           10.00     1511099             154.7     26.58    28.144

124928     256   10.00     1428856      0      292.6     50.00    28.046
124928           10.00     1426238             292.1     26.20    14.694

124928     512   10.00     1308377      0      535.9     50.00    15.321
124928           10.00     1305437             534.7     20.39    6.249 

124928    1024   10.00     1136815      0      931.2     49.90    8.898 
124928           10.00     1121584             918.8     11.32    2.018 

124928    2048   10.00      575896      0      943.4     14.48    2.522 
124928           10.00      574299             940.8     10.31    1.796 

124928    4096   10.00      291152      0      954.0     6.34     1.090 
124928           10.00      290918             953.3     8.37     1.439 

124928    8192   10.00      144074      0      944.2     4.02     0.697 
124928           10.00      143992             943.6     7.42     1.287 

124928   16834   10.00       69343      0      933.8     3.17     0.558 
124928           10.00       69157             931.3     7.45     1.311 



Version-Release number of selected component (if applicable):

kernel-2.6.32-63.el6.x86_64
qemu-kvm-0.12.1.2-2.109.el6.x86_64

How reproducible:
always 

Steps to Reproduce:
1.running netperf from external or  host to guest  & from guest to guest .

for b in 32 64 128 256 512 1024 2048 4096 8192 16834; do ./netperf -t UDP_STREAM -f m -H 192.168.0.12 -P 0 -l 10 -- -m $b; done

  
Actual results:

Big drop  in UDP throughput from external or  host to guest  & from guest to guest .

Expected results:


Additional info:
Comment 2 Alex Williamson 2010-08-25 10:06:59 EDT
Please point out exactly what the problem is and where the throughput is not what it should be.  It's pretty clear to me from the results that the system is on a 1Gbps network and we're hitting line rate when going across the network with sufficient packet size.  8Gbps host->guest and 5Gbps guest->guest also don't appear to be substantially terrible.  Is this bug valid?
Comment 3 Quan Wenli 2010-08-26 01:50:10 EDT
List some big drop throughput.

1 > external -> guest
Message Size bytes  Throughput 10^6bits/sec     drop packages % (TX-RX/TX)
------------------+-------------------------+-----------------------------
     
32                      104.1                    45.8%
	                 56.3
64                      192.4                    52.9%
                         90.6
128                     347.8                    52.1%
                        166.4 
2 > Host -> guest

64                      138.7                    34.7%
                        90.6 
1024                    2370.9                   49%
                        1208.9
2048                    5339.6                   60.5%
                        2108.1
3 > guestA1 - guestA2 (A1,A2 on the same host)

4096                    3271.7                   18.3%
		        2673.3 
8192                    5102.1                   27.7%
                        3691.2 

there are no any drop packages from guest to external.
not sure if it's a problem or what's the range of drops can be accept.
Comment 4 Alex Williamson 2010-08-26 15:05:27 EDT
(In reply to comment #3)
> List some big drop throughput.
> 
> 1 > external -> guest
> Message Size bytes  Throughput 10^6bits/sec     drop packages % (TX-RX/TX)
> ------------------+-------------------------+-----------------------------
> 
> 32                      104.1                    45.8%
>                   56.3

Ok, I see the drop you're talking about now, but this is how UDP works, if queues get full, packets get dropped.

> there are no any drop packages from guest to external.

But that's not true, every example posted in comment #0 shows some packet loss.

> not sure if it's a problem or what's the range of drops can be accept.

I think this is just a symptom of differences in rate of packet production versus packet consumption.  Testing over the physical network provides another packet limiter, so we see few dropped packets.  Testing guest-host removes this, so we see more dropped packets.  Testing guest-guest shows we have similar packet generation vs consumption rates, so loss is lower.  This can be seen on physical hardware without virtualization as well.  Doing the same test from a physical system on 1Gbps ethernet to a another physical system on 802.11g wifi on the same network results in:

124928      32   10.00     1226248      0      31.39
124928           10.00       15212              0.39

124928      64   10.00      933493      0      47.79
124928           10.00       15841              0.81

124928     128   10.00      631869      0      64.70
124928           10.00       35376              3.62

124928     256   10.00      383813      0      78.60
124928           10.00       31709              6.49

124928     512   10.00      215031      0      88.08
124928           10.00       28075             11.50

124928    1024   10.00      114454      0      93.76
124928           10.00       21740             17.81

124928    2048   10.00       57416      0      94.07
124928           10.00         669              1.10

124928    4096   10.00       29182      0      95.62
124928           10.00         439              1.44

124928    8192   10.00       14608      0      95.73
124928           10.00         209              1.37

124928   16834   10.00        7122      0      95.91
124928           10.00         444              5.98

This is far worse than the results you're seeing.  I think this should be closed as NOTABUG.
Comment 6 Quan Wenli 2010-08-29 23:38:45 EDT
(In reply to comment #4)

> I think this is just a symptom of differences in rate of packet production
> versus packet consumption.  Testing over the physical network provides another
> packet limiter, so we see few dropped packets.  Testing guest-host removes
> this, so we see more dropped packets.  Testing guest-guest shows we have
> similar packet generation vs consumption rates, so loss is lower.  This can be
> seen on physical hardware without virtualization as well.  Doing the same test
> from a physical system on 1Gbps ethernet to a another physical system on
> 802.11g wifi on the same network results in:
> 

Hi alex ,thanks for your analysis. 

I still have one question:
From checking " result for rhel6 qemu 113 vs rhel6 qemu 111  vs rhel 5.5 GA in virtio-userspace in the https://bugzilla.redhat.com/show_bug.cgi?id=624769#c15.(picture 4)"

We can see rhel6 virtualization was worse than rhel5.5 virtualization in test of small package size about packet consumption.is it a problem?

Note You need to log in before you can comment on or make changes to this bug.