Bug 743843 - Deadlock if client sends an RPC message with unknown program
Summary: Deadlock if client sends an RPC message with unknown program
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 747120
TreeView+ depends on / blocked
 
Reported: 2011-10-06 09:46 UTC by Daniel Berrangé
Modified: 2011-12-06 11:37 UTC (History)
8 users (show)

Fixed In Version: libvirt-0.9.4-17.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-06 11:37:05 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:1513 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2011-12-06 01:23:30 UTC

Description Daniel Berrangé 2011-10-06 09:46:56 UTC
Description of problem:
if the client sends an RPC message to libvirtd with an unknown RPC program or version number, it will cause libvirtd to deadlock in the main event loop. This is obviously bad, because it can happen before any authentication has even taken place.

This is a regression introduced by the patches for bug 692663.

Version-Release number of selected component (if applicable):


How reproducible:
libvirt-0.9.4-14.el6

Steps to Reproduce:
1. Hack the libvirt client source to change the RPC program number
2. Run any command in the client
3.
  
Actual results:
Daemon is deadlocked

Expected results:
Client receives an RPC reply about an unknown program number.

Additional info:

Comment 1 Daniel Berrangé 2011-10-06 09:47:14 UTC
Patch upstream

https://www.redhat.com/archives/libvir-list/2011-October/msg00158.html

Comment 4 Alex Jia 2011-10-09 09:13:16 UTC
(In reply to comment #1)
> Patch upstream
> 
> https://www.redhat.com/archives/libvir-list/2011-October/msg00158.html

Michal has given a ACK, Should move this bug to POST status?

Daniel, I tried to reproduce this issue by hacking client source, however, I'm not clear whether it's a right place and modification:

* src/rpc/virnetclient.c
......
int virNetClientAddProgram(virNetClientPtr client,
                           virNetClientProgramPtr prog)
{
    virNetClientLock(client);

    if (VIR_EXPAND_N(client->programs, client->nprograms, 1) < 0)
        goto no_memory;

    client->programs[client->nprograms-1] = prog;
......

Should I change 'client->nprograms' to a constant? it seems this is a wrong behaviour.

Thanks,
Alex

Comment 7 Daniel Berrangé 2011-10-10 09:54:02 UTC
Find the place in daemon/libvirtd.c which does

    if (virNetServerAddProgram(srv, qemuProgram) < 0) {
        ret = VIR_DAEMON_ERR_INIT;
        goto cleanup;
    }


and comment out those 4 lines & rebuild libvirtd.

Then try to run 'virsh qemu-monitor-command --hmp $GUESTNAME  'info cpus'

Comment 8 Alex Jia 2011-10-10 09:59:36 UTC
(In reply to comment #7) 
> Then try to run 'virsh qemu-monitor-command --hmp $GUESTNAME  'info cpus'

Daniel, thanks, I will do this now.

Comment 9 Alex Jia 2011-10-10 10:36:29 UTC
I can reproduce this issue when I roll back current git to fd52b96 then rebuild libvirtd, virsh command will be hang. 

And virsh client works well after I apply patch 696becb (Don't send back unknown program errors for async messages):

$ ./daemon/libvirtd --daemon
$ ./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
* CPU #0: pc=0xffffffff8006b2f5 (halted) thread_id=15405

So current upstream is fine and has resolved this issue.

Alex

Comment 10 Alex Jia 2011-10-11 02:25:11 UTC
(In reply to comment #9)
> 
> $ ./daemon/libvirtd --daemon
> $ ./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
> * CPU #0: pc=0xffffffff8006b2f5 (halted) thread_id=15405

I forgot to comment out previous 4 lines again, the actual result should like this:

./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
error: Cannot find program 536903815 version 1

Notes, libvirt raises a appropriate error information.

Comment 12 Alex Jia 2011-10-12 02:42:55 UTC
This issue has been resolved on libvirt-0.9.4-17.el6.src.rpm, the test result is the same to Comment 10, the function virNetServerProgramUnknownError is hit and libvirt raises a appropriate error information when run libvirt client programming:
# ./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
error: Cannot find program 536903815 version 1

So move the bug to VERIFIED status.

Comment 13 errata-xmlrpc 2011-12-06 11:37:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html


Note You need to log in before you can comment on or make changes to this bug.