Update spice-server on RHEL7.3 host to spice-server-0.12.4-19.el7.x86_64, and test again, the problem still exists

Created attachment 1200914 [details]
log and domain xml

1) 7.3 -> 7.3 works because libvirtd is not (correctly) waiting for the SPICE_MIGRATE_COMPLETED event

2) in the 7.2 -> 7.2 case libvirtd is waiting for the event and QEMU sends it

3) 7.3 -> 7.2 is broken since libvirtd is waiting for the event, but QEMU doesn't send it

I wasn't able to reproduce the third case; QEMU was sending the event. I'm not sure why it didn't work for you, but we should fix libvirt to behave similarly to 1. That is, libvirt should not even try to do seamless SPICE migration.

Fixed upstream by

commit 2e164b451ee6a33e2d9ba33d7bf2121f3f4817a0
Refs: v2.2.0-141-g2e164b4
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Sep 20 17:27:03 2016 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Wed Sep 21 14:11:26 2016 +0200

    qemu: Ignore graphics cookie if port == 0

    Old libvirt represents

        <graphics type='spice'>
          <listen type='none'/>
        </graphics>

    as

        <graphics type='spice' autoport='no'/>

    In this mode, QEMU doesn't listen for SPICE connection anywhere and
    clients have to use virDomainOpenGraphics* APIs to attach to the domain.
    That is, the client has to run on the same host where the domains runs
    and it's impossible to tell the client to reconnect to the destination
    QEMU during migration (unless there is some kind of proxy on the host).

    While current libvirt correctly ignores such graphics devices when
    creating graphics migration cookie, old libvirt just sends

        <graphics type='spice' port='0' listen='0.0.0.0' tlsPort='-1'/>

    in the cookie. After seeing this cookie, we happily would call
    client_migrate_info QMP command and wait for SPICE_MIGRATE_COMPLETED
    event, which is quite pointless since the doesn't know where to connecti
    anyway. We should just ignore such cookies.

    https://bugzilla.redhat.com/show_bug.cgi?id=1376083

    Signed-off-by: Jiri Denemark <jdenemar>

Hi Jirka

I tried to verify this bug with the following builds, but the issue still exists:

Source host:
libvirt-1.2.17-13.el7_2.6.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.25.x86_64
spice-server-0.12.4-15.el7_2.2.x86_64

Target host:
libvirt-3.2.0-6.virtcov.el7.x86_64
qemu-kvm-rhev-2.9.0-6.el7.x86_64
spice-server-0.12.8-2.el7.x86_64

Well, the fix was on the source and you're still using an old unfixed version there. Thus this is definitely not something the patch from comment 5 could fix. However, from its commit message I'd say the new libvirt on the target host should not even send a graphics cookie, but apparently it did. Could you be so kind and share debug logs from both sides with us?

Created attachment 1283695 [details]
libvirtd log for comment 7

Hmm, this seems to be a different issue according to logs. There's no graphics cookie sent by the target libvirtd and there's no message in the log which would indicate the source is waiting for spice migration to finish. It seems the migration just hangs somewhere in qemuMigrationConfirmPhase just after parsing the migration cookie. Could you please attach gdb to the source daemon once it hangs and attach a complete backtrace of it?

Created attachment 1283714 [details]
backtrace of libvirtd

(In reply to Jiri Denemark from comment #10)
> Hmm, this seems to be a different issue according to logs. There's no
> graphics cookie sent by the target libvirtd and there's no message in the
> log which would indicate the source is waiting for spice migration to
> finish. It seems the migration just hangs somewhere in
> qemuMigrationConfirmPhase just after parsing the migration cookie. Could you
> please attach gdb to the source daemon once it hangs and attach a complete
> backtrace of it?

Maybe there is a misunderstanding between us?  I'm always testing vm migration from RHEL 7.2 to newer release from the beginning.

OK so the source is really waiting for SPICE migration to finish according to the backtrace even though the new libvirt on the destination does not (correctly) send anything about graphics in the migration cookie.

And this bug is about fixing the source, so you can't really test it by using the old unfixed libvirt on the source. You could try to update the source libvirtd, but keep old qemu-kvm-rhev there.

(In reply to Jiri Denemark from comment #13)
> OK so the source is really waiting for SPICE migration to finish according
> to the backtrace even though the new libvirt on the destination does not
> (correctly) send anything about graphics in the migration cookie.
> 
> And this bug is about fixing the source, so you can't really test it by
> using the old unfixed libvirt on the source. You could try to update the
> source libvirtd, but keep old qemu-kvm-rhev there.

Migration between newer libvirt versions or between older libvirt versions or from newer libvirt version to older libvirt version can always succeed, this bug only exists when migration from older libvirt version(RHEL7.2) to newer libvirt version. 
So I think if I can update libvirt version before do migration, the migration can succeed even before the fix.

I don't really remember any details about this issue, but commit 4 suggests QEMU version matters too. Which is why I suggested updating just libvirt and using it with older QEMU.

(In reply to Jiri Denemark from comment #15)
> I don't really remember any details about this issue, but commit 4 suggests
> QEMU version matters too. Which is why I suggested updating just libvirt and
> using it with older QEMU.

After update libvirt to latest version, spice migration can be completed. But I still think this bug can't be verified, because usually we don't update libvirt version when testing cross migration

Hi Jirka

Now, the test results are as below. It seems that the problem can only be fixed on source side(which should be not possible), should we just close this bug as won'tfix?

Direction      Result
7.2->7.3       Fail    ===> This one is what's reported in comment 0.
7.2->7.4       Fail

7.3->7.2       Succeed
7.4->7.2       Succeed

7.2->7.2       Succeed
7.3->7.3       Succeed
7.4->7.4       Succeed

I think CURRENTRELEASE would be a better reason.

(In reply to Jiri Denemark from comment #18)
> I think CURRENTRELEASE would be a better reason.

OK, I have no objection. Will you do it? Or should I do it?