Will try on x86 to see if this bug is aarch64 only.

Didn't reproduce this bug on x86 with repeating 200 times:
1) tried 100 times on qemu-kvm-7.2.0-4.el9.x86_64
2) tried 100 times on qemu-kvm-7.2.0-2.el9.x86_64



Will provide the network info of aarch64 machines later.

Also hit this issue on s390x, the host system of s390x is a KVM guest.


You can reserve them from beaker.
src host name: s390x-kvm-042.lab.eng.rdu2.redhat.com
dst host name: s390x-kvm-055.lab.eng.rdu2.redhat.com



The network info is same on src and dst:
# ethtool -k enc1 | grep scatter-gather
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
# lshw -c network -businfo
Bus info      Device     Class      Description
===============================================
virtio@1      enc1       network    Ethernet interface
ccw.0000             network    Virtual network device
ccw.0001             network    Virtual network device
[root@s390x-kvm-042 ~]# hostname
s390x-kvm-042.lab.eng.rdu2.redhat.com

The network info of aarch64 machine is (both src and dst machine are same):
# lshw -c network -businfo
Bus info          Device       Class      Description
=====================================================
pci@0000:01:00.0  enp1s0f0np0  network    MT27710 Family [ConnectX-4 Lx]
pci@0000:01:00.1  enp1s0f1np1  network    MT27710 Family [ConnectX-4 Lx]
pci@0002:01:00.0  enP2p1s0     network    I210 Gigabit Network Connection
# ethtool -k enP2p1s0 | grep scatter-gather
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]

Hi Yan, thanks for the update, I will try on x86 later.

Hi yafu

This is a different bug.
When we are on migration, there are two stages:
- iterative stage
- final stage.

During final stage, migration_cancel don't work.  We don't have the infrastructure to get back to iterative stage.  So it is a different bug.  We can open a new bug to try to fix it, but we don't have (upstream) the infrastructure to fix it.
Notice that you don't have any of the multifd threads working at that time.

So I think we are back to only s390 and arm to reproduce it.

(In reply to Juan Quintela from comment #8)
> Hi yafu
> 
> This is a different bug.
> When we are on migration, there are two stages:
> - iterative stage
> - final stage.
> 
> During final stage, migration_cancel don't work.  We don't have the
> infrastructure to get back to iterative stage.  So it is a different bug. 
> We can open a new bug to try to fix it, but we don't have (upstream) the
> infrastructure to fix it.
> Notice that you don't have any of the multifd threads working at that time.
> 
> So I think we are back to only s390 and arm to reproduce it.

Hi Juan,

Thanks for your clarification. File a bug to track this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=2167670

I could also reproduce this bug on the latest rhel 9.2.0 with 1/100 rate. 

So update hardware from aarch64 to all.

Also reproduce migration cancel hang issue when migrating with multifd enabled and multifd thread 4, but zerocopy is disabled with 1/200 rate.
(adjust VIRT-296187 case to disable zerocopy and add a sleep of 6 sceonds before canceling migration)

Qemu and libvirt: qemu-kvm-7.2.0-14.el9_2.x86_64 && libvirt-client-9.0.0-9.el9_2.x86_64

Try to find the difference between this bug and bug 2167657 by testing via libvirt and qemu:
1. Only enable multifd:
(1) when set multifd threads to a big value, such as 100, 250, migration cancel issue can be easily reproduced through libvirt.
(2) but if threads are 4 or 10, reproduce with a very low rate like my above tests (1/200)

2. Enable  zerocopy and multifd:
when set multifd threads to 4, it's easier to reproduce than zerocopy disable, like this bug's Description, 1/50 rate. 



So in summary, seems this bug and bug 2167657 are the same, they test under different scenarios, and the reproduction rate is not same. 
Easier reproduce when set multifd threads to be a bug value, and enable zerocopy.

*** Bug 2167657 has been marked as a duplicate of this bug. ***

Hi

This comment helps a lot to reproduce:
(1) when set multifd threads to a big value, such as 100, 250, migration cancel issue can be easily reproduced through libvirt.

I will try it and will told you if I can reproduce it easily that way (such a big number makes no sense at all), but if I helps debugging, that is very good.

Hi
I have been able to reproduce this bug with multifd channels set to 250.
It looks like there are memory corruption.  Trying to follow what is going on here.


Thanks, Juan.

(In reply to Juan Quintela from comment #16)
> Hi
> I have been able to reproduce this bug with multifd channels set to 250.
> It looks like there are memory corruption.  Trying to follow what is going
> on here.
> 
> 
> Thanks, Juan.

Nice.


Update the summary of this bug as zerocopy is not a must, multifd and many channels also reproduce it.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.