RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 955540 - Host network is not available when Guest act as a netperf client testing 'UDP_STREAM' and another host act as a netserver
Summary: Host network is not available when Guest act as a netperf client testing 'UDP...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Michael S. Tsirkin
QA Contact: xiywang
URL:
Whiteboard:
Depends On:
Blocks: 965975 1190573
TreeView+ depends on / blocked
 
Reported: 2013-04-23 09:17 UTC by Qian Guo
Modified: 2018-02-07 23:12 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 965975 1190573 (view as bug list)
Environment:
Last Closed: 2018-02-07 23:12:29 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Qian Guo 2013-04-23 09:17:12 UTC
Description of problem:
Boot a rhel7 guest in rhel7 host(hostA), and run netserver in another rhel7 host(hostB) and run 'netperf -H $ip_hostB -t UDP_STREAM' in this rhel7 guest. During the netperf test, network of hostA is not reachable, ping will lose pkgs.

Version-Release number of selected component (if applicable):
host and guest kernel version:
# uname -r
3.9.0-0.rc7.53.el7.x86_64

host:
# rpm -q qemu-kvm
qemu-kvm-1.4.0-2.1.el7.x86_64
# rpm -q bridge-utils
bridge-utils-1.5-5.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot RHEL7 guest in hostA
# /usr/libexec/qemu-kvm -cpu host -m 4G -smp 4,sockets=1,cores=4,threads=1 -M pc -enable-kvm  -name win2012  -drive file=/home/rhel7u64cp1.qcow2,if=none,format=qcow2,werror=stop,rerror=stop,media=disk,id=drive-scsi0-disk0 -device virtio-scsi-pci,id=scsi0,addr=0x4 -device scsi-hd,scsi-id=0,lun=0,bus=scsi0.0,drive=drive-scsi0-disk0,id=virtio-disk0 -nodefaults -nodefconfig -monitor stdio   -netdev tap,id=bd,script=/etc/qemu-ifup,ifname=qiguo1 -device virtio-net-pci,netdev=bd,mac=54:52:1a:46:0b:02,id=vnic1  -vnc :10 -vga std  -boot menu=on -device virtio-balloon-pci,id=balloon1

2.In hostB, run netserver, and inside guest run netperf and protocol UDP_STREAM
# netperf -H 10.66.7.126 -t UDP_STREAM

3.In the meantime, ping hostA   

Actual results:
During the netperf test, could not ping hostA.


Expected results:
During the netperf test w/ UDP_DTREAM, can access hostA

Additional info:
1. test e1000, not hit this issue
2. test virtio-net-pci w/ vhost=on or not, both have this bug.

3.Test virtio-net-pci, the 'ethtool -k ' output:
# ethtool -k eth0
Features for eth0:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]

4.Test e1000, the 'ethtool -k ' output:

# ethtool -k eth0
Features for eth0:
rx-checksumming: off
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off
rx-all: off

Comment 1 Qian Guo 2013-04-23 09:23:56 UTC
The netperf  test result:

Test w/ virtio-net-pci:
# netperf -H 10.66.7.126 -t UDP_STREAM
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.7.126 () port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   65507   10.00      553431      0    29002.37
212992           10.00       19213           1006.85

Test w/ e1000:
# netperf -H 10.66.7.126 -t UDP_STREAM
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.7.126 () port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   65507   10.00       16211      0     849.53
212992           10.00        3863            202.44

When test the same netperf between guest and hostA, not hit this issue

Comment 3 Qian Guo 2013-04-25 02:58:19 UTC
Hi, Michael

I test this case w/ two hosts, that in the same broadcast domain, not hit this issue, both network of hosts works well during UDP_STREAM tests:

HostA:
# ethtool em1
Settings for em1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 2
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: on
	Supports Wake-on: pumbg
	Wake-on: g
	Current message level: 0x00000007 (7)
			       drv probe link
	Link detected: yes
# ethtool -i em1
driver: e1000e
version: 2.2.14-k
firmware-version: 0.13-4
bus-info: 0000:00:19.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no


HostB:
# ethtool em1
Settings for em1:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 2
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: on
	Supports Wake-on: pumbg
	Wake-on: g
	Current message level: 0x00000007 (7)
			       drv probe link
	Link detected: yes

# ethtool -i em1
driver: e1000e
version: 2.2.14-k
firmware-version: 0.13-4
bus-info: 0000:00:19.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no


And the test result of netperf:
# netperf -H 10.66.5.221 -t UDP_STREAM
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.5.221 () port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   65507   10.00       18213      0     954.45
212992           10.00       18213            954.45


From the results, if this bug is a virt bug? 
If it's not a virt bug, pls feel free to change the components.

Best Regards!
Qian Guo

Comment 4 Michael S. Tsirkin 2013-05-29 07:09:15 UTC
Just checking: this is not a regression right?
Same issue observed with 6.4?

Comment 5 Michael S. Tsirkin 2013-05-29 07:11:24 UTC
also I am guessing this is with multiqueue disabled?

Comment 6 jason wang 2013-05-30 03:10:45 UTC
Please answer Michael's question and also try:

- Could it happens when you are doing pktgen from guest
- Does the problem disappear when adding sndbuf=1048576 in cli

Thanks

Comment 7 Qian Guo 2013-05-30 03:20:52 UTC
(In reply to jason wang from comment #6)
> Please answer Michael's question and also try:
> 
> - Could it happens when you are doing pktgen from guest
> - Does the problem disappear when adding sndbuf=1048576 in cli
> 
> Thanks
Since there's some urgent things in my hand, I will test assp after that, and update here.

Thanks!

Comment 8 Qian Guo 2013-06-03 07:18:02 UTC
(In reply to Michael S. Tsirkin from comment #4)
> Just checking: this is not a regression right?
> Same issue observed with 6.4?
Hi, Michael

I test w/ RHEL6.4GA, same issue, GuestA in HostA, and HostB run netserver, then during test UDP_STREAM, could not ping HostA from anywhere.

guest and host kernel:
# uname -r
2.6.32-358.el6.x86_64

qemu-kvm version:
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.355.el6.x86_64


And I did not test w/ multi-queue, so shall I test it in rhel7 w/ multi-queue?

I will test by comment #6, and update here.

Thanks !
Qian Guo

Comment 9 Qian Guo 2013-06-04 06:17:14 UTC
(In reply to jason wang from comment #6)
> Please answer Michael's question and also try:
> 
> - Could it happens when you are doing pktgen from guest
Test pktgen inside guest, i just use single queue
# ./pktgen.sh eth0 1
cat pktgen.sh

#!/bin/sh
# usage sh pktgen.sh $device $queues

modprobe -r pktgen
modprobe  pktgen
echo reset > /proc/net/pktgen/pgctrl

ifconfig $1 up

function pgset() {
    local result

    echo $1 > $PGDEV

    result=`cat $PGDEV | fgrep "Result: OK:"`
    if [ "$result" = "" ]; then
         cat $PGDEV | fgrep Result:
    fi
}

function pg() {
    echo inject > $PGDEV
    cat $PGDEV
}

for i in 0 `seq $(($2-1))`
do
    echo "Adding queue 0 of $1"
    dev=$1@$i

    PGDEV=/proc/net/pktgen/kpktgend_$i
    pgset "rem_device_all"
    pgset "add_device $dev"
    pgset "max_before_softirq 100000"

    # Configure the individual devices
    echo "Configuring devices $dev"

    PGDEV=/proc/net/pktgen/$dev

    pgset "queue_map_min $i"
    pgset "queue_map_max $i"
      pgset "count 10000000"
    pgset "min_pkt_size 60"
    pgset "max_pkt_size 60"
    pgset "dst 10.66.10.90"
    pgset "dst_mac 9a:05:06:07:08:10"
    pgset "udp_src_min 0"
    pgset "udp_src_max 65535"
    pgset "udp_dst_min 0"
    pgset "udp_dst_max 65535"
done

# Time to run

PGDEV=/proc/net/pktgen/pgctrl

echo "Running... ctrl^C to stop"

pgset "start"

echo "Done"


It works well, during pktgen test in guest, pinging this host well, no pkg lost.

> - Does the problem disappear when adding sndbuf=1048576 in cli
And no pkg lost when set sndbuf=1048576 and test UDP_STREAM netperf test.

It is the result when test w/ sndbuf=104876:

# netperf -H 10.66.6.254 -t UDP_STREAM -l 120
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.6.254 () port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   65507   120.09     218778      0     954.71
229376           120.09     218778            954.71


> 
> Thanks


Note You need to log in before you can comment on or make changes to this bug.