Bug 1286492 - vhost-user interface cannot get ip if host is a numa machine
vhost-user interface cannot get ip if host is a numa machine
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
x86_64 Linux
high Severity medium
: rc
: ---
Assigned To: Victor Kaplansky
Pei Zhang
: ZStream
: 1290063 1290065 (view as bug list)
Depends On: 1370005
Blocks: 1305606 1313485 1288337 1290065 1315197
  Show dependency treegraph
 
Reported: 2015-11-29 22:02 EST by Luyao Huang
Modified: 2016-11-07 16:39 EST (History)
21 users (show)

See Also:
Fixed In Version: QEMU 2.6
Doc Type: Bug Fix
Doc Text:
Using a vhost-net device in a guest running on a NUMA host in some cases prevented the guest from obtaining a dynamic IP address. This update ensures that regions with different file handlers cannot be merged, which allows vhost-net devices to work properly.
Story Points: ---
Clone Of:
: 1315197 (view as bug list)
Environment:
Last Closed: 2016-11-07 16:39:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Luyao Huang 2015-11-29 22:02:38 EST
Description of problem:
vhost-user interface cannot get ip if host is a numa machine

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-31.el7_2.3.x86_64
3.10.0-327.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
0. prepare a numa machine :
# numactl --hard
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 32511 MB
node 0 free: 31053 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 32768 MB
node 1 free: 31643 MB
node distances:
node   0   1 
  0:  10  11 
  1:  11  10 


1. prepare the tests/vhost-user-bridge for test on host:
clone upstream qemu and make the test
and use https://bugzilla.redhat.com/attachment.cgi?id=1098548 to cover the vhost-user-bridge.c in the $qemu/tests/, then make it

# make tests/vhost-user-bridge
  CC    tests/vhost-user-bridge.o
  LINK  tests/vhost-user-bridge


2. mount hugepages:

# mount -t hugetlbfs hugetlbfs -opagesize=2M /dev/hugepages2M


3. start a slirp qemu on host

# cat run-slirp.sh
#!/bin/sh -x

/usr/libexec/qemu-kvm -enable-kvm \
     -net none \
     -net socket,vlan=0,udp=localhost:4444,localaddr=localhost:5555 \
     -net user,vlan=0


# sh run-slirp.sh
Warning: vlan 0 with no nics
VNC server running on `::1:5900'

4. start vhost-user-bridge:

./vhost-user-bridge -l localhost:4444 -r localhost:5555 -u /tmp/vubr.sock
ud socket: /tmp/vubr.sock
local:     localhost:4444
remote:    localhost:5555
Added sock 4 for watching. max_sock: 4
Waiting for connections on UNIX socket /tmp/vubr.sock ...
Added sock 5 for watching. max_sock: 5
Waiting for data from udp backend on localhost:4444...


5. start a qemu process which numa node > 1 on source host:

# /usr/libexec/qemu-kvm -enable-kvm -m 2048 -smp 2     -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/hugepages2M,share=on     -numa node,nodeid=0,memdev=mem -object memory-backend-file,id=mem1,size=1024M,mem-path=/dev/hugepages2M,share=on     -numa node,nodeid=1,memdev=mem1 -mem-prealloc     -chardev socket,id=char0,path=/tmp/vubr.sock     -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce     -device virtio-net-pci,netdev=mynet1,mac=54:52:00:1a:2c:01    /nfs/lhuang/r7_ext4.raw    -monitor stdio     -vga std -vnc :10

6. try to get ip via dhclient command in guest:

IN GUEST:
# dhclient -d

6. check the output in vhost-user-bridge:

==================   Vhost user message from QEMU   ==================
Request: VHOST_USER_SET_VRING_ENABLE (18)
Flags:   0x1
Size:    8
State.index: 1
State.enable:   1
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000002
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001
Got kick_data: 0000000000000001

Actual results:

guest cannot get ip if host is a numa machine and guest numa node > 1

and i have test if guest only have one node vhost-user interface can get ip address

Expected results:

can get ip address on a numa machine

Additional info:
Comment 1 Michael S. Tsirkin 2015-12-01 11:46:22 EST
does this help?
http://brewweb.devel.redhat.com/brew/taskinfo?taskID=10174897
Comment 2 Victor Kaplansky 2015-12-01 15:42:47 EST
I've applied Michael's patch (https://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg00164.html) to git://git.app.eng.bos.redhat.com/virt/rhel7/qemu-kvm.git (tag: qemu-kvm-rhev-2.3.0-31.el7_2.3).

And the issue seems to be fixed.
Here is the new Memory Table sent to vubr:

==================   Vhost user message from QEMU   ==================
Request: VHOST_USER_SET_MEM_TABLE (5)
Flags:   0x1
Size:    264
Fds: 10 11 12 13 14 15 16 17
Nregions: 8
Region 0
    guest_phys_addr: 0x0000000040000000
    memory_size:     0x0000000040000000
    userspace_addr   0x00002aab00000000
    mmap_offset      0x0000000000000000
    mmap_addr:       0x00002aaac0000000
Region 1
    guest_phys_addr: 0x0000000000000000
    memory_size:     0x00000000000a0000
    userspace_addr   0x00002aaac0000000
    mmap_offset      0x0000000000000000
    mmap_addr:       0x00002aab00000000
Region 2
    guest_phys_addr: 0x00000000000c0000
    memory_size:     0x000000000000b000
    userspace_addr   0x00002aaac00c0000
    mmap_offset      0x00000000000c0000
    mmap_addr:       0x00002aab40000000
Region 3
    guest_phys_addr: 0x00000000000cb000
    memory_size:     0x0000000000003000
    userspace_addr   0x00002aaac00cb000
    mmap_offset      0x00000000000cb000
    mmap_addr:       0x00002aab80000000
Region 4
    guest_phys_addr: 0x00000000000ce000
    memory_size:     0x000000000001e000
    userspace_addr   0x00002aaac00ce000
    mmap_offset      0x00000000000ce000
    mmap_addr:       0x00002aabc0000000
Region 5
    guest_phys_addr: 0x00000000000ec000
    memory_size:     0x0000000000004000
    userspace_addr   0x00002aaac00ec000
    mmap_offset      0x00000000000ec000
    mmap_addr:       0x00002aac00000000
Region 6
    guest_phys_addr: 0x00000000000f0000
    memory_size:     0x0000000000010000
    userspace_addr   0x00002aaac00f0000
    mmap_offset      0x00000000000f0000
    mmap_addr:       0x00002aac40000000
Region 7
    guest_phys_addr: 0x0000000000100000
    memory_size:     0x000000003ff00000
    userspace_addr   0x00002aaac0100000
    mmap_offset      0x0000000000100000
    mmap_addr:       0x00002aac80000000
==========================================================

qemu activation:
----------------

NUM=1

${WORK_DIR}/qemu-kvm/bin/x86_64-softmmu/qemu-system-x86_64 \
    -snapshot \
    -enable-kvm -m 2048 -smp 2 \
    -chardev socket,id=char0,path=/tmp/vubr${NUM}.sock \
    -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
    -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:0$NUM \
    -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/hugepages,share=on \
    -numa node,nodeid=0,memdev=mem -mem-prealloc \
    -object memory-backend-file,id=mem1,size=1024M,mem-path=/dev/hugepages,share=on \
    -numa node,nodeid=1,memdev=mem1 -mem-prealloc \
    -net user,hostfwd=tcp::1002$NUM-:22,net=10.2.2.0/24 -net nic \
    /root/fedora-infiniband.qcow2

-- Victor
Comment 18 juzhang 2016-03-16 05:11:12 EDT
Hi Xiangchun,

This is RHEL7.3 bz but you are using RHEL7.2.z build. Remove the needinfo.

Best Regards,
Junyi
Comment 20 Mike McCune 2016-03-28 19:44:20 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 22 Amnon Ilan 2016-06-19 04:39:49 EDT
*** Bug 1297748 has been marked as a duplicate of this bug. ***
Comment 23 Amnon Ilan 2016-06-19 04:42:42 EDT
*** Bug 1290063 has been marked as a duplicate of this bug. ***
Comment 24 Amnon Ilan 2016-06-19 04:43:59 EDT
*** Bug 1290065 has been marked as a duplicate of this bug. ***
Comment 25 Pei Zhang 2016-09-22 03:39:00 EDT
==Verification==
Versions:
Host:
3.10.0-510.rt56.415.el7.x86_64
qemu-kvm-rhev-2.6.0-26.el7.x86_64

Guest:
3.10.0-510.rt56.415.el7.x86_64

Steps:
1. Boot slirp as background
# /usr/libexec/qemu-kvm \
-net none \
-net socket,vlan=0,udp=localhost:4444,localaddr=localhost:5555 \
-net user,vlan=0

2. Start vubr as server
# ./vhost-user-bridge -l localhost:4444 -r localhost:5555 -u /tmp/vubr.sock

3. Boot guest with 2 NUMA nodes
/usr/libexec/qemu-kvm -m 4096 -smp 4 \
-object memory-backend-file,id=mem0,size=2048M,mem-path=/dev/hugepages,share=on \
-object memory-backend-file,id=mem1,size=2048M,mem-path=/dev/hugepages,share=on \
-numa node,nodeid=0,memdev=mem0 \
-numa node,nodeid=1,memdev=mem1 \
-mem-prealloc \
-vga std -vnc :10 \
/home/pezhang/rhel7.3.qcow2 \
-chardev socket,id=char0,path=/tmp/vubr.sock \
-device virtio-net-pci,netdev=mynet1,mac=54:52:00:1a:2c:01 \
-netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
-monitor stdio \
-serial unix:/tmp/monitor,server,nowait \

4. Get ip in guest, and #wget works well
# dhclient

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::5652:ff:fe1a:2c01  prefixlen 64  scopeid 0x20<link>
        inet6 fec0::5652:ff:fe1a:2c01  prefixlen 64  scopeid 0x40<site>
        ether 54:52:00:1a:2c:01  txqueuelen 1000  (Ethernet)
        RX packets 16  bytes 3998 (3.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 37  bytes 4904 (4.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# wget http://download.eng.bos.redhat.com/brewroot/packages/kernel-rt/3.10.0/510.rt56.415.el7/x86_64/kernel-rt-3.10.0-510.rt56.415.el7.x86_64.rpm
...
Saving to: ‘kernel-rt-3.10.0-510.rt56.415.el7.x86_64.rpm’

100%[======================================>] 34,313,752   556KB/s   in 76s    

2016-09-22 15:34:09 (441 KB/s) - ‘kernel-rt-3.10.0-510.rt56.415.el7.x86_64.rpm’ saved [34313752/34313752]


So this bug has been fixed well. Thank you.
Comment 26 Pei Zhang 2016-09-22 03:39:47 EDT
Set this bug as 'VERIFIED' as Comment 25.
Comment 28 errata-xmlrpc 2016-11-07 16:39:03 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html

Note You need to log in before you can comment on or make changes to this bug.