Bug 1238153 - guest job locked after OpenGraphics fail because qemu have been killed
Summary: guest job locked after OpenGraphics fail because qemu have been killed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.2
Hardware: x86_64
OS: Linux
low
low
Target Milestone: rc
: ---
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-01 09:59 UTC by Luyao Huang
Modified: 2015-11-19 06:47 UTC (History)
6 users (show)

Fixed In Version: libvirt-1.2.17-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 06:47:15 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2202 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2015-11-19 08:17:58 UTC

Description Luyao Huang 2015-07-01 09:59:17 UTC
Description of problem:
guest job locked after OpenGraphics fail because qemu have been killed

Version-Release number of selected component (if applicable):
libvirt-1.2.16-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. prepare a small c progrem
# cat libvirt-test-openGraphic.c
#include <stdio.h>
#include "libvirt/libvirt.h"

int main() {
    virDomainPtr dom = NULL;
    virConnectPtr conn = NULL;
    int ret = -1;

    if (!(conn = virConnectOpen("qemu:///system")))
        return ret;

    dom = virDomainLookupByName(conn, "test3");

    ret = virDomainOpenGraphicsFD(dom, 0, 0);
 cleanup:
    virConnectClose(conn);
    return ret;
}

# gcc libvirt-test-openGraphic.c -lvirt -o libvirt-test-openGraphic.o

2. open another terminal to attach libvirtd and set breakpoint at qemuDomainOpenGraphicsFD:

(gdb) br qemuDomainOpenGraphicsFD
Breakpoint 1 at 0x7feca8b25590: file qemu/qemu_driver.c, line 17595.
(gdb) c
Continuing.
[Switching to Thread 0x7fecb208f700 (LWP 3234)]


3. call  qemuDomainOpenGraphicsFD function via c progrem:
# ./libvirt-test-openGraphic.o

4. kill the qemu when the libvirtd in this step:

17647	    if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_MODIFY) < 0)
(gdb) 
17649	    qemuDomainObjEnterMonitor(driver, vm);
(gdb) 
17650	    ret = qemuMonitorOpenGraphics(priv->mon, protocol, pair[1], "graphicsfd",                            <-------kill qemu then continue
(gdb) n
17652	    if (qemuDomainObjExitMonitor(driver, vm) < 0) {
(gdb) 
17650	    ret = qemuMonitorOpenGraphics(priv->mon, protocol, pair[1], "graphicsfd",
(gdb) 
17652	    if (qemuDomainObjExitMonitor(driver, vm) < 0) {
(gdb) 
[Thread 0x7feca73d7700 (LWP 18028) exited]

5. restart guest:

# virsh start test3
error: Failed to start domain test3
error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainOpenGraphicsFd)


Actual results:
guest job locked after OpenGraphics fail because qemu have been killed

Expected results:
I used gdb to reproduce this issue as we have some bugs like this(qemu crash when libvirtd enter the monitor), so it will be a possible issue in the future.

Additional info:

This upstream patch will fix this issue:

commit 91c9e4d92099dab178de9f204f0fdb91028f071a
Author: Luyao Huang <lhuang@redhat.com>
Date:   Tue Jun 30 11:35:13 2015 +0800

    qemu: End job even if exiting monitor after OpenGraphics(FD) fails

v1.2.17-rc2-3-g91c9e4d

Comment 2 hongming 2015-07-07 10:19:03 UTC
Verify it as follows. The result is expected. Move its status to VERIFIED.

# rpm -q libvirt
libvirt-1.2.17-1.el7.x86_64

# virsh net-edit ipv6
error: XML error: Invalid character 'x' in id '0:4:7e:7d:f0:7d:a8:bc:c5:d2:13:32:11:ed:16:ea:84:aaxxx' of network 'ipv6'
Failed. Try again? [y,n,f,?]: 
error: XML error: Invalid character 'x' in id '0:4:7e:7d:f0:7d:a8:bc:c5:d2:13:32:11:ed:16:ea:84:aax66' of network 'ipv6'
Failed. Try again? [y,n,f,?]: 


# virsh net-define ipv6.xml
error: Failed to define network from ipv6.xml
error: XML error: Invalid character 'x' in id '0:4:7e:7d:f0:7d:a8:bc:c5:d2:13:32:11:ed:16:ea:84:aaxx3' of network 'ipv6'


# virsh net-define ipv6.xml
error: Failed to define network from ipv6.xml
error: XML error: Invalid character 'z' in id '0:4:7e:7d:f0:7d:a8:bc:c5:d2:13:32:11:ed:16:ea:84:z3' of network 'ipv6'

Comment 3 hongming 2015-07-07 10:20:31 UTC
Sorry . I paste the wrong verification .Change the status to ON_QA

Comment 4 Shanzhi Yu 2015-07-14 01:19:32 UTC
I can reproduce it with libvirt-1.2.16-1.el7.x86_64
Step as in comment 0.

# virsh start r7 
error: Failed to start domain r7
error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainOpenGraphicsFd)

error log:

2015-07-14 01:06:23.922+0000: 23122: error : qemuDomainObjBeginJobInternal:1478 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainOpenGraphicsFd)

Verify with libvirt-1.2.17-2.el7.x86_64

17379	    if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_MODIFY) < 0)
(gdb) 
17381	    qemuDomainObjEnterMonitor(driver, vm);
(gdb) 
17382	    ret = qemuMonitorOpenGraphics(priv->mon, protocol, pair[1], "graphicsfd",

kill qemu-kvm process 

start guest again

# virsh start r7 
Domain r7 started

So no any more job locked again.

Comment 6 errata-xmlrpc 2015-11-19 06:47:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html


Note You need to log in before you can comment on or make changes to this bug.