Qemu command [1]:
/usr/libexec/qemu-kvm  \
-name "mouse-vm" \
-sandbox on \
-machine s390-ccw-virtio,memory-backend=mem-machine_mem \
-cpu host \
-nodefaults  \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1,server=on,wait=off \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor,server=on,wait=off \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device '{"driver":"virtio-scsi-ccw","id":"virtio_scsi_ccw0","devno":"fe.0.0000"}' \
-device '{"driver":"scsi-hd","id":"image1","device_id":"drive-image1","drive":"drive_image1","bus":"virtio_scsi_ccw0.0","bootindex":0,"write-cache":"on"}' \
-device '{"driver":"virtio-net-ccw","mac":"9a:fc:6e:3c:a6:f0","id":"net0","netdev":"tap0","devno":"fe.0.0001"}' \
-device '{"driver":"virtio-mouse-ccw","id":"input_mouse1","devno":"fe.0.0002"}' \
-device '{"driver":"virtio-keyboard-ccw","id":"input_keyboard1","devno":"fe.0.0003"}' \
-device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0004"}' \
-blockdev '{"driver":"file","auto-read-only":true,"discard":"unmap","aio":"threads","cache":{"direct":true,"no-flush":false},"filename":"/mnt/xiaohli/rhel920-s390x-virtio-scsi.qcow2","node-name":"drive_sys1"}' \
-blockdev '{"driver":"qcow2","node-name":"drive_image1","read-only":false,"cache":{"direct":true,"no-flush":false},"file":"drive_sys1"}' \
-netdev tap,id=tap0,vhost=on \
-m 4096 \
-object '{"qom-type":"memory-backend-ram","id":"mem-machine_mem","size":4294967296}' \
-smp 1,maxcpus=1,cores=1,threads=1,sockets=1 \
-nographic  \
-rtc base=utc,clock=host,driftfix=slew \
-boot strict=on \
-enable-kvm  \
-qmp tcp:0:3333,server=on,wait=off \
-qmp tcp:0:9999,server=on,wait=off \
-qmp tcp:0:9888,server=on,wait=off \
-serial tcp:0:4444,server=on,wait=off \
-monitor stdio \

And also tried this case on x86 with repeating 300 times, didn't reproduce this bug.

we can refer to https://bugzilla.redhat.com/show_bug.cgi?id=2096143 which has fixed similar issue on RHEL 9.1

Li, do you know whether this is a regression? I.e. did this used to work in the past? If so, do you know which qemu-kvm version was still OK?

Peter, are you aware of any recent issues that could trigger this problem?

(In reply to Thomas Huth from comment #5)
> Peter, are you aware of any recent issues that could trigger this problem?

Yes we used to have bz2096143 (also quoted in comment 3) and we fixed it for all archs.

For this one it seems more like something special in s390 where the socket can be released slightly slower than the other archs?  Per Xiaohui's comment above, it doesn't happen on other archs (e.g. x86), and also on s390 the port will be released if we wait for 10 secs.

Copying Jason too just in case Jason has any input from the network kernel stack pov on why this could be happening too, or whether there's easy way to debug / justifity why it was releasing slower.  As I don't remember there's any arch-specific code in this path.

For whatever cause, the workaround will be extremely easy even if triggered in any customer - we just wait 10 seconds.  Not so bad for postcopy recovery since we don't even expect to use it unless there's a network failure already.

(In reply to Thomas Huth from comment #4)
> Li, do you know whether this is a regression? I.e. did this used to work in
> the past? If so, do you know which qemu-kvm version was still OK?

I have tried on qemu-kvm-6.2.0-13.el9 and qemu-kvm-7.0.0-13.el9.s390x.rpm, all hit this bug on s390x

Hi Peter,
I tried to test the case of this bug on x86 with RHEL 9.3, also hit this issue (kernel-5.14.0-316.el9.x86_64 & qemu-kvm-8.0.0-4.el9.x86_64).
But only hit once. Retry 10 times, didn't reproduce. But I would track this bug on x86 later again.

So I believe this bug should be for all hardware

Xiaohui,

Thanks for further providing the results on x86.  Maybe it's not really a bug, but just that the system will take time to shut off a port, and before that reusing is probably just not possible on the same one.  Then it seems for some reason s390 is just even slower on releasing so it's easier to reproduce.

Here for qmp "migrate-pause" it does a shutdown() syscall on src socket.  I'd boldly guess the port can be released even later if the network was interrupted not as graceful.

Daniel, according to above comment, do we still have chance to touch up the document update, removing the s390 specific wordings?

Thanks!
Peter

Hi Peter, that shouldn't be a problem, we can update the release notes pretty much at any point if there is some correction.
So, just keeping it general like this should work, right?

----
.Recovering an interrupted post-copy VM migration might fail
If a post-copy migration of a virtual machine (VM) is interrupted and then immediately resumed on the same incoming port, the migration might fail with the following error: `Address already in use`

To work around this problem, wait at least 10 seconds before resuming the post-copy migration or switch to another port for migration recovery.
----

Thanks for letting me know about it!
Daniel

ACK, thanks!

As per the previous discussion, I updated the release note text to be generic and removed the mention of any specific architecture.

(In reply to Peter Xu from comment #15)
> Xiaohui,
> 
> Thanks for further providing the results on x86.  Maybe it's not really a
> bug, but just that the system will take time to shut off a port, and before
> that reusing is probably just not possible on the same one.  Then it seems
> for some reason s390 is just even slower on releasing so it's easier to
> reproduce.

Thank you Peter.

Then shall we update the case in our test plan to add some sleep time if we think it's not really a bug?
How do you think if I add 10 seconds for it?

> 
> Here for qmp "migrate-pause" it does a shutdown() syscall on src socket. 
> I'd boldly guess the port can be released even later if the network was
> interrupted not as graceful.
> 
> Daniel, according to above comment, do we still have chance to touch up the
> document update, removing the s390 specific wordings?
> 
> Thanks!
> Peter

Update the hardware to all since also reproduce this bug on x86.

(In reply to Li Xiaohui from comment #19)
> Then shall we update the case in our test plan to add some sleep time if we
> think it's not really a bug?
> How do you think if I add 10 seconds for it?

A delay makes sense. Depending on whether you want to make it more efficient, instead of waiting 10s you can make a maximum timeout (say, 60sec?), but retry every 2 sec?

(In reply to Peter Xu from comment #21)
> (In reply to Li Xiaohui from comment #19)
> > Then shall we update the case in our test plan to add some sleep time if we
> > think it's not really a bug?
> > How do you think if I add 10 seconds for it?
> 
> A delay makes sense. Depending on whether you want to make it more
> efficient, instead of waiting 10s you can make a maximum timeout (say,
> 60sec?), but retry every 2 sec?

Good suggestion. I will do so. Thanks

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.