Xiaohui, a few questions/points:

1. What's the network issue you were encountering?
2. Have you tried waiting for some longer time before sending the "migrate-recover" command?  E.g., 10 min.
   I remember TCP has some timeout mechanism for some kind of network failures, I don't remember the details but I think 10 min will cover that.
3. Yes it'll be great if you could figure out whether it's a regression (as you mentioned in the "Additional Info").

Thanks,

(In reply to Peter Xu from comment #1)
> Xiaohui, a few questions/points:
> 
> 1. What's the network issue you were encountering?

I tried two methods to make postcopy active into pause status:
1) make network issue like down the migration network on destination host: 
# nmcli connection down enp96s0f1np1 
2) use '{"exec-oob":"migrate-pause"}' to pause the postcopy by manual on source host.

Above two ways all hit the bug.

> 2. Have you tried waiting for some longer time before sending the
> "migrate-recover" command?  E.g., 10 min.
>    I remember TCP has some timeout mechanism for some kind of network
> failures, I don't remember the details but I think 10 min will cover that.

I didn't tried to wait long time such as 10 mins. If need I can do it.
But I rem when we down the network, we need to wait such as 30 mins to tcp timeout so that we can get postcopy pause. 
For network recovery, I don't think we need wait(I also didn't wait in previous tests but they pass and didn't hit such a bug)

> 3. Yes it'll be great if you could figure out whether it's a regression (as
> you mentioned in the "Additional Info").

I will try tmr.

> 
> Thanks,

(In reply to Li Xiaohui from comment #2)
> (In reply to Peter Xu from comment #1)
> > Xiaohui, a few questions/points:
> > 
> > 1. What's the network issue you were encountering?
> 
> I tried two methods to make postcopy active into pause status:
> 1) make network issue like down the migration network on destination host: 
> # nmcli connection down enp96s0f1np1 
> 2) use '{"exec-oob":"migrate-pause"}' to pause the postcopy by manual on
> source host.
> 
> Above two ways all hit the bug.

Oh that's weird, then obviously I cannot reproduce this with upstream QEMU because I use 2) a lot.

> 
> > 2. Have you tried waiting for some longer time before sending the
> > "migrate-recover" command?  E.g., 10 min.
> >    I remember TCP has some timeout mechanism for some kind of network
> > failures, I don't remember the details but I think 10 min will cover that.
> 
> I didn't tried to wait long time such as 10 mins. If need I can do it.
> But I rem when we down the network, we need to wait such as 30 mins to tcp
> timeout so that we can get postcopy pause. 
> For network recovery, I don't think we need wait(I also didn't wait in
> previous tests but they pass and didn't hit such a bug)

Yes, migrate-pause is the safe/fast/clean way to close it and IIUC it does not require any major timeout.  But I think I used to try fail the network in strange ways and conditionally I could have dead tcp port which requires the timeout, I don't think that's as long as 30min but it could be that I mis-remembered.

So maybe turning down the NIC may hit it.  I didn't really check.  Anyway, if it can recover after a few tens of minutes then at least it means it's not QEMU who leaked it.  Not urgently needed to verify, as..

> 
> > 3. Yes it'll be great if you could figure out whether it's a regression (as
> > you mentioned in the "Additional Info").
> 
> I will try tmr.

.. this could help us to figure out something already, since I'd hope rhel8 doesn't have this issue.

Also hit this bug on RHEL 8.7.0 (kernel-4.18.0-393.el8.x86_64 && qemu-kvm-6.2.0-14.module+el8.7.0+15289+26b4351e.x86_64) with {"exec-oob":"migrate-pause"} method:

waiting 2 hours after postcopy paused, try again using the same migration port, still hit error:
{"exec-oob":"migrate-recover", "arguments":{"uri":"tcp:$dst_host_ip:1234"}}
{"error": {"class": "GenericError", "desc": "Failed to find an available port: Address already in use"}}

Didn't hit this bug on same hosts with qemu-kvm-6.0.0-33.module+el8.5.0+13740+349232b6.2.x86_64. So it should be a regression bug.


Shall I clone this bug for RHEL 8.7.0?

(In reply to Li Xiaohui from comment #5)
> Didn't hit this bug on same hosts with
> qemu-kvm-6.0.0-33.module+el8.5.0+13740+349232b6.2.x86_64. So it should be a
> regression bug.

How about rhel8.6?  

Would it be easy for you to figure out which package got it regressed first (aka, an initial bisection)?

> 
> Shall I clone this bug for RHEL 8.7.0?

Feel free to.  Thanks,

(In reply to Peter Xu from comment #6)
> (In reply to Li Xiaohui from comment #5)
> > Didn't hit this bug on same hosts with
> > qemu-kvm-6.0.0-33.module+el8.5.0+13740+349232b6.2.x86_64. So it should be a
> > regression bug.
> 
> How about rhel8.6?  

qemu-kvm-6.1.0-1.module+el8.6.0+12535+4e2af250.x86_64 brought out this bug. Can't reproduce on/before qemu-kvm-6.0.0-29.module+el8.6.0+12490+ec3e565c.x86_64.

> 
> Would it be easy for you to figure out which package got it regressed first
> (aka, an initial bisection)?

See above.

> 
> > 
> > Shall I clone this bug for RHEL 8.7.0?
> 
> Feel free to.  Thanks,

Thanks

thanks Xiaohui.

I think it's because of this commit that was introduced in v6.0.0-v6.1.0:

a59136f3b1 migration/socket: Close the listener at the end

And the latest upstream doesn't hit that because there's already a fix:

08401c0426 migration: Allow migrate-recover to run multiple times
(though we may also need to fetch some of its deps)

I'll cook a rhel9 build soon to verify.

I have downloaded the qemu-kvm scratch from Comment 9, will test it in next week because I'm on PTO today.

Retest this bug according to the Description, migrate-recover succeeds with qemu-img-7.0.0-6.el9.postcopy_port_reset.x86_64:
1.After postcopy is paused, check migration port 1234 on dst host, still exists:
[root@lenovo-sr630-01 ~]# netstat -tunap | grep 1234
tcp        0      0 10.73.178.67:1234       0.0.0.0:*               LISTEN      2145614/qemu-kvm   
2.Recover postcopy migration, relaunch the migration port 1234 successfully on dst host:
{"exec-oob":"migrate-recover", "arguments":{"uri":"tcp:$dst_host_ip:1234"}}
{"return": {}}
3.Start postcopy again on src host:
{"execute":"migrate", "arguments":{"uri":"tcp:$dst_host_ip:1234", "resume":true}}
{"return": {}}

Thanks, Xiaohui.  I'll prepare a MR soon.

Note: I can reproduce this bug on s390x (kernel-5.14.0-114.el9.s390x && qemu-kvm-7.0.0-6.el9.s390x), so it's not x86 only bug

(In reply to Li Xiaohui from comment #16)
> Note: I can reproduce this bug on s390x (kernel-5.14.0-114.el9.s390x &&
> qemu-kvm-7.0.0-6.el9.s390x), so it's not x86 only bug

Yes, it's a generic QEMU bug.

Verify this bug on the latest RHEL 9.1.0 (kernel-5.14.0-121.el9.x86_64 & qemu-kvm-7.0.0-8.el9.x86_64)


Test steps same as Comment 11, after recover postcopy and the status of migration is postcopy-active, found 1234(migration incoming port) on destination host like:
# netstat -tunap | grep 1234
tcp        0      0 10.73.130.69:1234       0.0.0.0:*               LISTEN      38502/qemu-kvm      
tcp        0      0 10.73.130.69:1234       10.73.130.67:37408      ESTABLISHED 38502/qemu-kvm 

By the way, postcopy recovery succeeds with reused migration port, and migration finishes successfully.


So I will mark this bug as verified per above test result after we get verified:Tested.


Peter, can we adjust the postcopy recovery case:
1) current scenario: use real network issue to make postcopy pause. it's not much convenient for test and it's not automated
2) I want to update the scenario, use '{"exec-oob":"migrate-pause"}' qmp command to make postcopy pause. It will help test and automation.

(In reply to Li Xiaohui from comment #19)
> Peter, can we adjust the postcopy recovery case:
> 1) current scenario: use real network issue to make postcopy pause. it's not
> much convenient for test and it's not automated
> 2) I want to update the scenario, use '{"exec-oob":"migrate-pause"}' qmp
> command to make postcopy pause. It will help test and automation.

I think it's okay to use "migrate-pause" to trigger it for auto test.  Note that I think you don't even need to use OOB message for the pause command, only the "migrate-recover" command requires to be an OOB message.

It's just that "migrate-pause" will cleanly shutdown the sockets so it's so friendly.  Some other auto-able options: (1) use one "nc" process to forward the migration, then kill the "nc" program anytime using SIGKILL, or (2) turn down the port of migration using nmcli.

But we can leave any option for later.

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

We have stable recurrence steps for this bug, so remove 'SanityOnly' from Verified.

Hi Camilla, can you help move the bug to on_qa according to Comment 21? 

The verification for this bug (see Comment 19) has been finished by me, I'm waiting to mark it as verified after this bug is in errata

Mark verfied per Comment 19

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7967