Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1321299

Summary: Migration is not workable with vhostuser interface
Product: Red Hat Enterprise Linux 7 Reporter: xiywang
Component: openvswitch-dpdkAssignee: Open vSwitch development team <ovs-team>
Status: CLOSED DUPLICATE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: ailan, aloughla, atragler, fleitner, hhuang, juzhang, knoel, mleitner, pliu, pmatilai, rkhan, xfu, xiywang, xuzhang, zshi
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-30 02:09:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1366498    

Description xiywang 2016-03-25 10:01:53 UTC
Description of problem:
If use vhostuser to boot a guest, then migration fails with error message displayed.

Version-Release number of selected component (if applicable):
guest and host kernel:
3.10.0-364.rt56.241.el7.x86_64
host qemu:
qemu-kvm-rhev-2.5.0-3.el7.x86_64
ovs-dpdk:
openvswitch-dpdk-2.5.0-3.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. setup vhostuser environment with script below:
#!/bin/sh

HUGETLB_PATH="/mnt/hugetlbfs/"
eth_model="X540-AT2"

# mount hugetlbfs
if [ ! -d "$HUGETLB_PATH" ]; then
	mkdir $HUGETLB_PATH
fi

PAGESIZE_1G=`cat /proc/cmdline | grep "default_hugepagesz=1G"`
if [ ! -z "$PAGESIZE_1G" ]; then
	mount -t hugetlbfs hugetlbfs $HUGETLB_PATH -o size=16G
else
	PAGESIZE_2M=`cat /proc/cmdline | grep "default_hugepagesz=2M"`
	if [ ! -z "$PAGESIZE_2M" ]; then
		mount -t  hugetlbfs -o pagesize=2048K none $HUGETLB_PATH
	else
		echo "please add hugepage setting in /boot/grub2/grub.cfg first."	
		exit
	fi
fi

# load/unload modules
for kmod in openvswitch vfio vfio-pci; do
	if lsmod | grep -q $kmod; then
	echo "not loading $kmod (already loaded)"
else
	if modprobe -v $kmod; then
		echo "loaded $kmod module"
	else
		echo "Failed to load $kmmod module, exiting"
		exit 1
	fi
fi
done

# bind the devices to dpdk module
dpdk_nics=`lspci -D  | grep $eth_model | awk '{print $1}'`
echo DPDK adapters: $dpdk_nics
dpdk_nic_kmod=vfio-pci

for nic in $dpdk_nics; do
	dpdk_nic_bind.py -b $dpdk_nic_kmod $nic
done
dpdk_nic_bind.py --status

# completely remove old ovs configuration
echo "remove old ovs configuration"
killall ovs-vswitchd
killall ovsdb-server
sleep 3
rm -rf /var/run/openvswitch/*
rm -rf /etc/openvswitch/*db*
rm -rf /var/log/openvswitch/*

# start database server
echo "start new ovs"
export DB_SOCK="/var/run/openvswitch/db.sock"
mkdir -p /var/run/openvswitch
mkdir -p /etc/openvswitch
/bin/ovsdb-tool create /etc/openvswitch/conf.db /usr/share/openvswitch/vswitch.ovsschema
rm -rf /dev/usvhost-1
/sbin/ovsdb-server -v --remote=punix:$DB_SOCK \
    --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
    --pidfile --detach || exit 1

# start ovs
echo "openvswitch init"
/bin/ovs-vsctl --no-wait init
screen -dmS ovs sudo su -g qemu -c "umask 002; /sbin/ovs-vswitchd --dpdk -c 0x1 -n 3 --socket-mem 1024,1024 -- unix:$DB_SOCK --pidfile --mlockall --log-file=/var/log/openvswitch/ovs-vswitchd.log"

# configure the bridge
echo "configure the bridge0"
/bin/ovs-vsctl --if-exists del-br ovsbr0
/bin/ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
/bin/ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk
/bin/ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser

echo "configure the bridge1"
/bin/ovs-vsctl --if-exists del-br ovsbr1
/bin/ovs-vsctl add-br ovsbr1 -- set bridge ovsbr1 datapath_type=netdev
/bin/ovs-vsctl add-port ovsbr1 dpdk1 -- set Interface dpdk1 type=dpdk
/bin/ovs-vsctl add-port ovsbr1 vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser

/bin/ovs-vsctl show

echo "configure flowtable for bridge0"
/bin/ovs-ofctl del-flows ovsbr0
/bin/ovs-ofctl add-flow ovsbr0 "in_port=1,idle_timeout=0 actions=output:2"
/bin/ovs-ofctl add-flow ovsbr0 "in_port=2,idle_timeout=0 actions=output:1"

echo "configure flowtable for bridge1"
/bin/ovs-ofctl del-flows ovsbr1
/bin/ovs-ofctl add-flow ovsbr1 "in_port=1,idle_timeout=0 actions=output:2"
/bin/ovs-ofctl add-flow ovsbr1 "in_port=2,idle_timeout=0 actions=output:1"


2. boot a guest in src host
/usr/libexec/qemu-kvm -name rhel7.2-rt-355 -machine pc-i440fx-rhel7.2.0 -cpu host,+tsc-deadline,pmu=off -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 \

-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugetlbfs,share=on,size=4096M -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \

 -drive file=/home/rhel7.2-rt-355.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,snapshot=off -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \

-netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a1:d0:5f \

-chardev socket,id=charnet1,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet1,chardev=charnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:b2:02:01 -chardev socket,id=charnet2,path=/var/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=hostnet2,chardev=charnet2 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:b2:02:02 \

-monitor stdio -device qxl-vga,id=video0 -serial unix:/tmp/console,server,nowait -vnc :1 -spice port=5900,disable-ticketing

3. boot a guest in the same host (as dst host), with 'incoming'

4. do local migration, failed
(qemu) migrate -d tcp:10.73.64.240:4444



Actual results:
Error message displayed after step 4 is executed:
migrate: Migration disabled: vhost-user backend lacks VHOST_USER_PROTOCOL_F_LOG_SHMFD feature.

Expected results:
migration works without error

Additional info:
I've tested migration with upstream dpdk and ovs, it works fine.
I'm wondering if downstream gonna support migration or not. If not, please feel free to close this bug.

Comment 2 Panu Matilainen 2016-03-29 13:53:12 UTC
(In reply to xiywang from comment #0)
[...]
> Actual results:
> Error message displayed after step 4 is executed:
> migrate: Migration disabled: vhost-user backend lacks
> VHOST_USER_PROTOCOL_F_LOG_SHMFD feature.
> 
> Expected results:
> migration works without error
> 
> Additional info:
> I've tested migration with upstream dpdk and ovs, it works fine.
> I'm wondering if downstream gonna support migration or not. If not, please
> feel free to close this bug.

I assume by "upstream dpdk and ovs" you mean "built from upstream git repo recently". In that case yes, vhost-user live migration support was just added in January and will be part of the next DPDK version (16.04) which should be released in the next few weeks. 

So the answer basically is: "when openvswitch-dpdk is updated to use dpdk >= 16.04", but as to when that is going to happen I dont have an answer at the moment.

Comment 3 xiywang 2016-03-29 14:19:23 UTC
(In reply to Panu Matilainen from comment #2)
> (In reply to xiywang from comment #0)
> [...]
> > Actual results:
> > Error message displayed after step 4 is executed:
> > migrate: Migration disabled: vhost-user backend lacks
> > VHOST_USER_PROTOCOL_F_LOG_SHMFD feature.
> > 
> > Expected results:
> > migration works without error
> > 
> > Additional info:
> > I've tested migration with upstream dpdk and ovs, it works fine.
> > I'm wondering if downstream gonna support migration or not. If not, please
> > feel free to close this bug.
> 
> I assume by "upstream dpdk and ovs" you mean "built from upstream git repo
> recently". In that case yes, vhost-user live migration support was just
> added in January and will be part of the next DPDK version (16.04) which
> should be released in the next few weeks. 
> 
> So the answer basically is: "when openvswitch-dpdk is updated to use dpdk >=
> 16.04", but as to when that is going to happen I dont have an answer at the
> moment.

OK, we can skip the "when" part of this question. But I get the information (correct me if I'm wrong), that we're going to use dpdk >= 16.04 in the future eventually and support migration, am I right? I just need to get the message that whether we will support live migration, no matter when.

Comment 4 Panu Matilainen 2016-03-31 12:10:04 UTC
(In reply to xiywang from comment #3)
> OK, we can skip the "when" part of this question. But I get the information
> (correct me if I'm wrong), that we're going to use dpdk >= 16.04 in the
> future eventually and support migration, am I right? I just need to get the
> message that whether we will support live migration, no matter when.

Some day in some product version, sure.

As for the current RHEL-7 based offerings, I dont think anybody knows what the precise plan is. OVS 2.5 is a LTS release so we'll probably be sticking with that for a good while, and DPDK 2.2 is the officially blessed DPDK version with it. DPDK 16.04 seems to work fine with OVS 2.5 but its not officially supported upstream, at least yet. Its of course not even released yet so...

Flavio (cc'd), thoughts/comments?

Comment 5 Flavio Leitner 2016-04-06 17:54:28 UTC
Well, it's WIP in upstream yet as already said and it is too soon to say anything else.

I don't know why this bug was opened since vhost-user migration is not supported in any official product.  There is no customer case attached as well and it doesn't look like a RFE either.

Am I missing something?

Thanks,
fbl