RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1442929 - virt-viewer cannot keep connection while seamless migration
Summary: virt-viewer cannot keep connection while seamless migration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virt-viewer
Version: 7.4
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Pavel Grunt
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1449577
TreeView+ depends on / blocked
 
Reported: 2017-04-18 06:17 UTC by Yanqiu Zhang
Modified: 2017-08-01 15:06 UTC (History)
17 users (show)

Fixed In Version: virt-viewer-5.0-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 15:06:41 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1849 0 normal SHIPPED_LIVE virt-viewer bug fix and enhancement update 2017-08-01 17:49:46 UTC

Description Yanqiu Zhang 2017-04-18 06:17:24 UTC 
Description of problem:
virt-viewer cannot keep connection while seamless migration

Version-Release number of selected component (if applicable):
virt-viewer-5.0-2.el7.x86_64
spice-gtk3-0.33-3.el7.x86_64

How reproducible:
100%

Steps to Reproduce:

1.prepare migration env, and a domain with images on nfs storage

2.Dispatch ssh public key of source host to target host.
# ssh-keygen -t rsa
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target_ip}

3.Define a domain with default spice graphic on source host.
...
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
    </graphics>
…
4.add the following line in /etc/libvirt/qemu.conf both in source host and target host, and restart libvirtd
# echo 'spice_listen = "0.0.0.0"' >> /etc/libvirt/qemu.conf
# systemctl restart libvirtd

5.Start the domain on source host
# virsh start V
Domain V started

# virsh dumpxml V|grep graphic -A3
    <graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
...

6.Use virt-viewer to connect the domain on source host.
# virt-viewer V --debug --spice-debug

7.Migrate the domain to destination host and check virt-viewer connection
# virsh migrate V qemu+ssh://{target_ip}/system --live --unsafe --verbose
Migration: [100 %]

virt-viewer disconnect after migration.

# cat seamless-spice-debug.log|grep WARNING
(virt-viewer:7106): Spice-WARNING **: channel-display-gst.c:303:create_pipeline: GStreamer error: no element "avdec_h264"
(virt-viewer:7106): Gtk-WARNING **: Allocating size to SpiceDisplay 0x55b5489a63c0 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?
(virt-viewer:7106): Spice-WARNING **: channel-display-gst.c:303:create_pipeline: GStreamer error: no element "avdec_h264"
(virt-viewer:7106): Spice-WARNING **: channel-display-gst.c:303:create_pipeline: GStreamer error: no element "avdec_h264"
(virt-viewer:7106): Spice-WARNING **: channel-display-gst.c:303:create_pipeline: GStreamer error: no element "avdec_h264"


Actual results:
As in step7, virt-viewer cannot keep connection while seamless migration


Expected results:
virt-viewer should keep connection while seamless migration

Additional info:
"remote-viewer spice://{source_ip}:5900" works well
Comment 4 Pavel Grunt 2017-04-18 07:48:39 UTC 
Since you say that it is a regression, which version of spice-gtk & virt-viewer works ?

Also try using --direct:
  virt-viewer V --debug --spice-debug --direct
Comment 5 Yanqiu Zhang 2017-04-18 08:26:38 UTC 
Last time, we tested rhel7.4 libvirt-3.1.0-1 with virt-viewer-2.0-13.el7.x86_64.rpm & spice-gtk3-0.31-8.el7.x86_64.rpm, test passed.


Also, now rhel7.3 testing on libvirt-2.0.0-10 passed, with following pkgs version:
virt-viewer-2.0-12.el7.x86_64
spice-gtk3-0.31-6.el7_3.2.x86_64
Comment 6 Victor Toso 2017-04-24 12:31:03 UTC 
Hi,

I used remote-viewer to connect to a rhv 4.1 instance + migration between hosts and I was not able to reproduce it virt-viewer-5.0-2.el7.x86_64 and spice-gtk3-0.33-3.el7.x86_64.

Both spice-gtk and spice server were rebased this cycle. From the logs nothing poped up from the client side.

Could you provide the qemu+spice logs from both src & target hosts? (/var/log/libvirt/qemu/<domain>.log)

Could you try downgrading spice to rhel-7.3 version, which is 0.12.4-20 (both src/target hosts) and trying again?

The only bit I found suspicious in the client-side log is:

> (virt-viewer:9553): GSpice-DEBUG: spice-session.c:1701
> migration channels left:8 (in migration:8)

But this is *AFTER* the migration was already completed. Something seems off indeed. I'm still looking into the reason.
Comment 8 Yanqiu Zhang 2017-04-25 09:16:26 UTC 
(In reply to Victor Toso from comment #6)

Hi

> I used remote-viewer to connect to a rhv 4.1 instance + migration between
> hosts and I was not able to reproduce it virt-viewer-5.0-2.el7.x86_64 and
> spice-gtk3-0.33-3.el7.x86_64.

Yes, remote-viewer will not reproduce it, only reproduce with virt-viewer. As descripted in comment 0 additional info:
"remote-viewer spice://{source_ip}:5900" works well

> Both spice-gtk and spice server were rebased this cycle. From the logs
> nothing poped up from the client side.

all 7.4 test are with spice-server-0.12.8-1.el7.x86_64


> Could you provide the qemu+spice logs from both src & target hosts?
> (/var/log/libvirt/qemu/<domain>.log)

Pls refer to the new attachment "qemu-log" in comment 7.

> Could you try downgrading spice to rhel-7.3 version, which is 0.12.4-20
> (both src/target hosts) and trying again?

Retest by downgrading some pkgs, here are additional test scenarios:
a)works well on :
libvirt-3.1.0-1.el7.x86_64
virt-viewer-2.0-13.el7.x86_64
spice-gtk3-0.31-8.el7.x86_64

b)issue can also reproduce on:
libvirt-3.1.0-1.el7.x86_64
virt-viewer-5.0-2.el7.x86_64
spice-gtk3-0.33-3.el7.x86_64

So the issue is caused by virt-viewer-5.0-2.el7.x86_64 and spice-gtk3-0.33-3.el7.x86_64. the versions of virt-viewer and spice-gtk3 are interdependent, can not individually downgrade one and leave another one high version.

Why need to downgrade spice to rhel-7.3 version(0.12.4-20) ?


btw:also test with '--direct', no help, still reproduce.
Comment 9 Victor Toso 2017-04-25 09:31:30 UTC 
> Yes, remote-viewer will not reproduce it, only reproduce with virt-viewer.
> As descripted in comment 0 additional info:
> "remote-viewer spice://{source_ip}:5900" works well

Sorry, I missed that!

> > Could you provide the qemu+spice logs from both src & target hosts?
> > (/var/log/libvirt/qemu/<domain>.log)
> 
> Pls refer to the new attachment "qemu-log" in comment 7.

If possible, can you enable debug using SPICE_DEBUG=1 ? I usually do that by including export SPICE_DEBUG=1 in a custom qemu-kvm script.

> 
> > Could you try downgrading spice to rhel-7.3 version, which is 0.12.4-20
> > (both src/target hosts) and trying again?
> 
> Retest by downgrading some pkgs, here are additional test scenarios:
> a)works well on :
> libvirt-3.1.0-1.el7.x86_64
> virt-viewer-2.0-13.el7.x86_64
> spice-gtk3-0.31-8.el7.x86_64
> 
> b)issue can also reproduce on:
> libvirt-3.1.0-1.el7.x86_64
> virt-viewer-5.0-2.el7.x86_64
> spice-gtk3-0.33-3.el7.x86_64
> 
> So the issue is caused by virt-viewer-5.0-2.el7.x86_64 and
> spice-gtk3-0.33-3.el7.x86_64. the versions of virt-viewer and spice-gtk3 are
> interdependent, can not individually downgrade one and leave another one
> high version.
> 
> Why need to downgrade spice to rhel-7.3 version(0.12.4-20) ?

In 7.3 last version is 0.12.4-20, in 7.4 last version is 0.12.8-2 (today). The bump from 0.12.4 to 0.12.8 might have changed something too (but I'm skeptical now as this is not reproducible with remote-viewer, only with virt-viewer).

The migration messages comes from spice.

> btw:also test with '--direct', no help, still reproduce.

Thanks
Comment 11 Yanqiu Zhang 2017-04-26 09:38:59 UTC 
> If possible, can you enable debug using SPICE_DEBUG=1 ? I usually do that by
> including export SPICE_DEBUG=1 in a custom qemu-kvm script.

I tried with "export SPICE_DEBUG=1" in terminal, or add following xml in domain:
# virsh dumpxml V|grep kvm
<domain type='kvm' id='12' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...
  <qemu:commandline>
    <qemu:env name='SPICE_DEBUG' value='1'/>
  </qemu:commandline>

can see "SPICE_DEBUG=1" was set successfully in the qemu log(refer to new attachment "SPICE_DEBUG_qemu_log"), but still no any SPICE_DEBUG messages . Could you help to check?


> In 7.3 last version is 0.12.4-20, in 7.4 last version is 0.12.8-2 (today).
> The bump from 0.12.4 to 0.12.8 might have changed something too (but I'm
> skeptical now as this is not reproducible with remote-viewer, only with
> virt-viewer).
>

As in commment 8, all rhel7.4 scenarios were tested with the same spice-server-0.12.8-1.el7.x86_64, if as following, virt-viewer will keep connection well while seamless migration:
    virt-viewer-2.0-13.el7.x86_64
    spice-gtk3-0.31-8.el7.x86_64

if as following, issue reproduces:
    virt-viewer-5.0-2.el7.x86_64
    spice-gtk3-0.33-3.el7.x86_64

Both pass and fail result can occur on the same spice-server-0.12.8, so, I don't think need to downgrade to spice-server-0.12.4-20.
Comment 14 Pavel Grunt 2017-05-16 15:49:04 UTC 
Hi,

it regression in virt-viewer. I have bisected it to:
first bad commit: [a62827d28c6b69e90102e4c1c8043cbddad8929a] virt-viewer: ensure we close when seeing domain stop event
commit a62827d28c6b69e90102e4c1c8043cbddad8929a
Author: Daniel P. Berrange <berrange>
Date:   Tue Jul 12 11:27:09 2016 +0100

    virt-viewer: ensure we close when seeing domain stop event
    
    Normally virt-viewer relies on the VNC/SPICE widget seeing
    an EOF on its underlying connection to detect when the
    session is closed.
    
    When tunnelling to a remote guest over SSH though, this
    EOF can be delayed for a very long time, leaving a dead
    session open.
    
    This can be seen with
    
       virt-viewer -c qemu+ssh://remotehost/system guestname
    
    when on the remote shell run
    
       virsh destroy guestname
    
    and notice that virt-viewer does not see the shutdown
    immediately.
    
    When we get a domain stopped event we know the session
    should be dead, so forceably close it, if not already
    closed.
    
    Signed-off-by: Daniel P. Berrange <berrange>
Comment 16 Pavel Grunt 2017-05-18 09:33:25 UTC 
Posted: https://www.redhat.com/archives/virt-tools-list/2017-May/msg00072.html
Comment 19 Xiaodai Wang 2017-05-22 07:24:14 UTC 
I can reproduce it with virt-viewer-5.0-2.el7.x86_64.

then i verified it with virt-viewer-5.0-4.el7.x86_64, virt-viewer can still keep connecting after migration.

so move the bug from ON_QA to VERIFIED.
Comment 21 errata-xmlrpc 2017-08-01 15:06:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1849
Note You need to log in before you can comment on or make changes to this bug.