Bug 743843

Summary: Deadlock if client sends an RPC message with unknown program
Product: Red Hat Enterprise Linux 6 Reporter: Daniel Berrangé <berrange>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.0CC: acathrow, ajia, dallan, mjenner, mzhan, rwu, veillard, weizhan
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.9.4-17.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 11:37:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 747120    

Description Daniel Berrangé 2011-10-06 09:46:56 UTC
Description of problem:
if the client sends an RPC message to libvirtd with an unknown RPC program or version number, it will cause libvirtd to deadlock in the main event loop. This is obviously bad, because it can happen before any authentication has even taken place.

This is a regression introduced by the patches for bug 692663.

Version-Release number of selected component (if applicable):


How reproducible:
libvirt-0.9.4-14.el6

Steps to Reproduce:
1. Hack the libvirt client source to change the RPC program number
2. Run any command in the client
3.
  
Actual results:
Daemon is deadlocked

Expected results:
Client receives an RPC reply about an unknown program number.

Additional info:

Comment 1 Daniel Berrangé 2011-10-06 09:47:14 UTC
Patch upstream

https://www.redhat.com/archives/libvir-list/2011-October/msg00158.html

Comment 4 Alex Jia 2011-10-09 09:13:16 UTC
(In reply to comment #1)
> Patch upstream
> 
> https://www.redhat.com/archives/libvir-list/2011-October/msg00158.html

Michal has given a ACK, Should move this bug to POST status?

Daniel, I tried to reproduce this issue by hacking client source, however, I'm not clear whether it's a right place and modification:

* src/rpc/virnetclient.c
......
int virNetClientAddProgram(virNetClientPtr client,
                           virNetClientProgramPtr prog)
{
    virNetClientLock(client);

    if (VIR_EXPAND_N(client->programs, client->nprograms, 1) < 0)
        goto no_memory;

    client->programs[client->nprograms-1] = prog;
......

Should I change 'client->nprograms' to a constant? it seems this is a wrong behaviour.

Thanks,
Alex

Comment 7 Daniel Berrangé 2011-10-10 09:54:02 UTC
Find the place in daemon/libvirtd.c which does

    if (virNetServerAddProgram(srv, qemuProgram) < 0) {
        ret = VIR_DAEMON_ERR_INIT;
        goto cleanup;
    }


and comment out those 4 lines & rebuild libvirtd.

Then try to run 'virsh qemu-monitor-command --hmp $GUESTNAME  'info cpus'

Comment 8 Alex Jia 2011-10-10 09:59:36 UTC
(In reply to comment #7) 
> Then try to run 'virsh qemu-monitor-command --hmp $GUESTNAME  'info cpus'

Daniel, thanks, I will do this now.

Comment 9 Alex Jia 2011-10-10 10:36:29 UTC
I can reproduce this issue when I roll back current git to fd52b96 then rebuild libvirtd, virsh command will be hang. 

And virsh client works well after I apply patch 696becb (Don't send back unknown program errors for async messages):

$ ./daemon/libvirtd --daemon
$ ./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
* CPU #0: pc=0xffffffff8006b2f5 (halted) thread_id=15405

So current upstream is fine and has resolved this issue.

Alex

Comment 10 Alex Jia 2011-10-11 02:25:11 UTC
(In reply to comment #9)
> 
> $ ./daemon/libvirtd --daemon
> $ ./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
> * CPU #0: pc=0xffffffff8006b2f5 (halted) thread_id=15405

I forgot to comment out previous 4 lines again, the actual result should like this:

./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
error: Cannot find program 536903815 version 1

Notes, libvirt raises a appropriate error information.

Comment 12 Alex Jia 2011-10-12 02:42:55 UTC
This issue has been resolved on libvirt-0.9.4-17.el6.src.rpm, the test result is the same to Comment 10, the function virNetServerProgramUnknownError is hit and libvirt raises a appropriate error information when run libvirt client programming:
# ./tools/virsh qemu-monitor-command --hmp vr-rhel5u4-x86_64-kvm 'info cpus'
error: Cannot find program 536903815 version 1

So move the bug to VERIFIED status.

Comment 13 errata-xmlrpc 2011-12-06 11:37:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html