RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1262232 - self announcement and ctrl offloads does not work after migration
Summary: self announcement and ctrl offloads does not work after migration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: jason wang
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-11 08:32 UTC by jason wang
Modified: 2015-12-04 16:57 UTC (History)
13 users (show)

Fixed In Version: qemu-kvm-rhev-2.3.0-28.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-12-04 16:57:43 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
The gdbview log that before the migration (16.88 KB, text/plain)
2015-10-20 05:42 UTC, Qian Guo 		no flags Details
The gdbview log that after the migration (20.94 KB, text/plain)
2015-10-20 05:43 UTC, Qian Guo 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2546 0 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2015-12-04 21:11:56 UTC

Description jason wang 2015-09-11 08:32:12 UTC 
Description of problem:

    After commit 019a3edbb25f1571e876f8af1ce4c55412939e5d ("virtio: make
    features 64bit wide"). Device's guest_features was actually set after
    vdc->load(). This breaks the assumption that device specific load()
    function can check guest_features. For virtio-net, self announcement
    and guest offloads won't work after migration.
    
    Fixing this by defer them to virtio_net_load() where guest_features
    were guaranteed to be set. Other virtio devices looks fine

Notes: qemu-kvm-rhev commit is :

commit c9736b10ca3daefdb7284a9b39f8ec88afe7382e
Author: Xiao Wang <jasowang>
Date:   Tue Jul 7 09:18:24 2015 +0200

    virtio: make features 64bit wide
    
    Message-id: <1436260751-25015-22-git-send-email-jasowang>
    Patchwork-id: 66796
    O-Subject: [RHEL7.2 qemu-kvm-rhev PATCH V2 21/68] virtio: make features 64bit wide
    Bugzilla: 1227343
    RH-Acked-by: Michael S. Tsirkin <mst>
    RH-Acked-by: David Gibson <dgibson>
    RH-Acked-by: Laurent Vivier <lvivier>
    RH-Acked-by: Thomas Huth <thuth>


Version-Release number of selected component (if applicable):


How reproducible:

100%

Steps to Reproduce:
1. tcpdump tap after migration
2.
3.

Actual results:
No GARP were found by tcpdump.

Expected results:
Guest should send GARP.

Additional info:
Comment 1 jason wang 2015-09-11 08:33:03 UTC 
Upstream posted at http://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03059.html
Comment 5 Miroslav Rezanina 2015-10-05 05:23:39 UTC 
Fix included in qemu-kvm-rhev-2.3.0-28.el7
Comment 6 Qian Guo 2015-10-09 07:35:49 UTC 
Hi, Jason

Test with unfixed and fixed build, got the following results:

with unfixed build:
qemu-kvm-rhev-2.3.0-24.el7.x86_64

After migration, from tcpdump the tap:

14:44:59.248334 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
14:44:59.281400 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46
14:44:59.431372 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46
14:44:59.474553 IP 192.168.1.11 > 192.168.1.1: ICMP echo request, id 3748, seq 31, length 64
14:44:59.474585 IP 192.168.1.1 > 192.168.1.11: ICMP echo reply, id 3748, seq 31, length 64
14:44:59.681413 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46
14:45:00.031383 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46




With fixed build qemu-kvm-rhev-2.3.0-29.el7.x86_64

After migration, got the packets:

14:20:58.807154 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length 28
14:20:58.807173 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
14:20:58.833400 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46
14:20:58.856625 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length 28
14:20:58.856655 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
14:20:58.983374 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46
14:20:59.006655 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length 28
14:20:59.006674 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
14:20:59.233394 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46
14:20:59.256616 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length 28
14:20:59.256639 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
14:20:59.583391 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown) tell 54:52:01:2a:0b:01 (oui Unknown), length 46
14:20:59.606607 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length 28
14:20:59.606626 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32



From the result, with the fixed build, there're arp request 

"ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length 28"

Is this the fixed expected result.

Thanks,
Qian
Comment 7 jason wang 2015-10-15 01:42:45 UTC 
(In reply to Qian Guo from comment #6)
> Hi, Jason
> 
> Test with unfixed and fixed build, got the following results:
> 
> with unfixed build:
> qemu-kvm-rhev-2.3.0-24.el7.x86_64
> 
> After migration, from tcpdump the tap:
> 
> 14:44:59.248334 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor
> advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
> 14:44:59.281400 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 14:44:59.431372 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 14:44:59.474553 IP 192.168.1.11 > 192.168.1.1: ICMP echo request, id 3748,
> seq 31, length 64
> 14:44:59.474585 IP 192.168.1.1 > 192.168.1.11: ICMP echo reply, id 3748, seq
> 31, length 64
> 14:44:59.681413 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 14:45:00.031383 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 
> 
> 
> 
> With fixed build qemu-kvm-rhev-2.3.0-29.el7.x86_64
> 
> After migration, got the packets:
> 
> 14:20:58.807154 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length
> 28
> 14:20:58.807173 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor
> advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
> 14:20:58.833400 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 14:20:58.856625 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length
> 28
> 14:20:58.856655 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor
> advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
> 14:20:58.983374 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 14:20:59.006655 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length
> 28
> 14:20:59.006674 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor
> advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
> 14:20:59.233394 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 14:20:59.256616 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length
> 28
> 14:20:59.256639 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor
> advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
> 14:20:59.583391 ARP, Reverse Request who-is 54:52:01:2a:0b:01 (oui Unknown)
> tell 54:52:01:2a:0b:01 (oui Unknown), length 46
> 14:20:59.606607 ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length
> 28
> 14:20:59.606626 IP6 fe80::5652:1ff:fe2a:b01 > ff02::1: ICMP6, neighbor
> advertisement, tgt is fe80::5652:1ff:fe2a:b01, length 32
> 
> 
> 
> From the result, with the fixed build, there're arp request 
> 
> "ARP, Request who-has 192.168.1.11 tell 192.168.1.11, length 28"
> 
> Is this the fixed expected result.

Yes.

Thanks

> 
> Thanks,
> Qian
Comment 8 Qian Guo 2015-10-15 01:45:56 UTC 
According to comment 7, this bug is fixed.
Comment 9 Qian Guo 2015-10-15 02:13:55 UTC 
(In reply to Qian Guo from comment #8)
> According to comment 7, this bug is fixed.

Talked with Jason, I will test the offloads part for windows guest, will update here once I got the result.

Thanks,
Qian
Comment 10 Qian Guo 2015-10-20 05:40:32 UTC 
Hi, Yan & Jaosn

I followed Yan's suggestions, but I did not reproduce it with unfixed build(qemu-kvm-rhev-2.3.0-26.el7.x86_64).

I disabled the GUEST_TSO in src qemu:
-device virtio-net-pci,guest_tso4=off,guest_tso6=off,id=net0,netdev=hostnet0,mac=52:54:01:a2:0c:01

Then inside guest, disable/enable netkvm driver, then get followings from gdbview that GUST_TSOs are not loaded.

.......

00000079	5:18:42 AM	[ParaNdis_InitializeContext] Message interrupt assigned	
00000080	5:18:42 AM	[ParaNdis_ResetVirtIONetDevice] Done	
00000081	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_CSUM	
00000082	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_GUEST_CSUM	
00000083	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_MAC	
00000084	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_GSO	
00000085	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_GUEST_ECN	
00000086	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_GUEST_UFO	
00000087	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_TSO4	
00000088	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_TSO6	
00000089	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_ECN	
00000090	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_UFO	
00000091	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_MRG_RXBUF	
00000092	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_STATUS	
00000093	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_VQ	
00000094	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_RX	
00000095	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_VLAN	
00000096	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_RX_EXTRA	
00000097	5:18:42 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_MAC_ADDR	
00000098	5:18:42 AM	VirtIO Host Feature VIRTIO_F_INDIRECT	
00000099	5:18:42 AM	VirtIO Host Feature VIRTIO_F_ANY_LAYOUT	
00000100	5:18:42 AM	VirtIO Host Feature VIRTIO_RING_F_EVENT_IDX	
00000101	5:18:42 AM	[ParaNdis_InitializeContext] Link status on driver startup: 1	
00000102	5:18:42 AM	Permanent device MAC: 52-54-01-a2-0c-01	
00000103	5:18:42 AM	No valid MAC configured	
00000104	5:18:42 AM	Actual MAC: 52-54-01-a2-0c-01	


Then migration, from dst, disable/enable again, from the gdbview, the GUEST_TSOs are not loaded either.

00000079	5:20:15 AM	[ParaNdis_InitializeContext] Message interrupt assigned	
00000080	5:20:15 AM	[ParaNdis_ResetVirtIONetDevice] Done	
00000081	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_CSUM	
00000082	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_GUEST_CSUM	
00000083	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_MAC	
00000084	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_GSO	
00000085	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_GUEST_ECN	
00000086	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_GUEST_UFO	
00000087	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_TSO4	
00000088	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_TSO6	
00000089	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_ECN	
00000090	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_HOST_UFO	
00000091	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_MRG_RXBUF	
00000092	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_STATUS	
00000093	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_VQ	
00000094	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_RX	
00000095	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_VLAN	
00000096	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_RX_EXTRA	
00000097	5:20:15 AM	VirtIO Host Feature VIRTIO_NET_F_CTRL_MAC_ADDR	
00000098	5:20:15 AM	VirtIO Host Feature VIRTIO_F_INDIRECT	
00000099	5:20:15 AM	VirtIO Host Feature VIRTIO_F_ANY_LAYOUT	
00000100	5:20:15 AM	VirtIO Host Feature VIRTIO_RING_F_EVENT_IDX	
00000101	5:20:15 AM	[ParaNdis_InitializeContext] Link status on driver startup: 1	
00000102	5:20:15 AM	Permanent device MAC: 52-54-01-a2-0c-01	
00000103	5:20:15 AM	No valid MAC configured	
00000104	5:20:15 AM	Actual MAC: 52-54-01-a2-0c-01	



So, with unfixed build, the GUEST_TSOs works after before and after migration.

Maybe I missed something, could you help check the log, I will attach them.

Thanks,
Qian
Comment 11 Qian Guo 2015-10-20 05:42:05 UTC 
Created attachment 1084600 [details]
The gdbview log that before the migration
Comment 12 Qian Guo 2015-10-20 05:43:47 UTC 
Created attachment 1084601 [details]
The gdbview log that after the migration
Comment 13 Qian Guo 2015-10-23 03:30:52 UTC 
Additional info:

I also tested with qemu-kvm-rhev-2.3.0-31.el7.x86_64, same result as comment 10.
Feel free to contact me if any further test.

Thanks,
Qian
Comment 14 Qian Guo 2015-10-23 05:36:54 UTC 
Tested with unfixed and fixed builds with following scenario:

1.enable the tso(by default, and I also enabled it in the cli).
2.disable LSO4/LSO6 inside windows guest, there's no switch to control the tso feature.
3.Test netperf inside guest as client and host as netserver
*the throughput is 3032.04(10^6 bit/s)
4.migration, then test netperf again
*the throughput is 3177.41(10^6 bit/s)

Additionally, I also checked gdbview, no change before and after migration

Hi, Jason

Could you help have a check if we can set this bug as verified due to above tests and comments :)

Thanks,
Qian
Comment 15 jason wang 2015-10-23 05:59:22 UTC 
(In reply to Qian Guo from comment #14)
> Tested with unfixed and fixed builds with following scenario:
> 
> 1.enable the tso(by default, and I also enabled it in the cli).
> 2.disable LSO4/LSO6 inside windows guest, there's no switch to control the
> tso feature.
> 3.Test netperf inside guest as client and host as netserver
> *the throughput is 3032.04(10^6 bit/s)
> 4.migration, then test netperf again
> *the throughput is 3177.41(10^6 bit/s)
> 
> Additionally, I also checked gdbview, no change before and after migration
> 
> Hi, Jason
> 
> Could you help have a check if we can set this bug as verified due to above
> tests and comments :)
> 
> Thanks,
> Qian

I think the bug could be verified first. Leave the windows guest control offload for future investigation.
Comment 16 juzhang 2015-10-23 06:03:55 UTC 
(In reply to jason wang from comment #15)
> (In reply to Qian Guo from comment #14)
> > Tested with unfixed and fixed builds with following scenario:
> > 
> > 1.enable the tso(by default, and I also enabled it in the cli).
> > 2.disable LSO4/LSO6 inside windows guest, there's no switch to control the
> > tso feature.
> > 3.Test netperf inside guest as client and host as netserver
> > *the throughput is 3032.04(10^6 bit/s)
> > 4.migration, then test netperf again
> > *the throughput is 3177.41(10^6 bit/s)
> > 
> > Additionally, I also checked gdbview, no change before and after migration
> > 
> > Hi, Jason
> > 
> > Could you help have a check if we can set this bug as verified due to above
> > tests and comments :)
> > 
> > Thanks,
> > Qian
> 
> I think the bug could be verified first. Leave the windows guest control
> offload for future investigation.

Got it.

Hi Jason and Yan,

Any further testing need to be tested with window guest later, free to let us know.
Comment 17 juzhang 2015-10-23 06:04:56 UTC 
According to comment7 and comment15, set this issue as verified.
Comment 19 errata-xmlrpc 2015-12-04 16:57:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2546.html
Note You need to log in before you can comment on or make changes to this bug.