RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1054437 - libvirt should use server-side filtering for event registration
Summary: libvirt should use server-side filtering for event registration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Eric Blake
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-16 20:02 UTC by Eric Blake
Modified: 2015-11-19 05:44 UTC (History)
7 users (show)

Fixed In Version: libvirt-1.2.17-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 05:44:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2202 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2015-11-19 08:17:58 UTC

Description Eric Blake 2014-01-16 20:02:59 UTC
Description of problem:
Consider what happens when a host is managing lots of vms, but a particular client only cares about events happening on a small subset of those vms.  Right now, as soon as the client registers for an event tied to a particular domain, the server is told to send every event for every domain over the connection, and the client then has to filter which events pertain to the domain that was originally requested, so that the user's callback only fires for events tied to the domain the user cares about.  But this implies a lot of asynchronous RPC traffic, and wasted efforts in packaging, unpackaging, and discarding unrelated events.  Efficiency would be better if the server knew which domains a client cares about, and only sends events to the connections that care.

To a minor degree, the use of ACL domain:getattr filtering can prevent the delivery of wasted events to some users (see bug 1047964), but using ACLs in this manner is not ideal.  The problem itself is baked into the RPC protocol (in the original implementation, the client doesn't send enough information to the server for the server to know that per-domain filtering is in use, and the server doesn't reply with enough information for the client to track parallel event registrations); thankfully, we don't require a .so bump to fix the problem, but can merely add new RPC calls, along with feature detection for client/server negotiation of whether to use old style or new style RPCs.

Version-Release number of selected component (if applicable):
libvirt-1.1.1-18.el7

How reproducible:
100%

Steps to Reproduce:
1. Set up multiple VMs on a host
2. Register to receive events for a single domain in a client, with debugging turned on to track async messages from the server
3. Trigger the event for both the domain the client cares about and for other domains

Actual results:
debug traffic shows that the server sent every event, even the ones that weren't delivered to the callback function because of client-side filtering

Expected results:
debug traffic should show that the client sends the domain to the server, and that events coming from the server are limited to the ones for the domain in question

Additional info:
Patches have been posted upstream:
https://www.redhat.com/archives/libvir-list/2014-January/msg00689.html
and an initial attempt at a backport was made while fixing a CVE:
https://post-office.corp.redhat.com/mailman/private/virt-devel/2014-January/msg00079.html
While I raised this issue in the 7.0 timeframe, I'm not sure if there are any existing users that make heavy use of per-domain events (that is, VDSM is just fine with global events, because it wants to manage all vms rather than just a subset).  I'm not familiar enough with boxes to know if it would be a better fit for wanting events on only a subset of domains.  Depending on whether anyone determines a client that would benefit from the speedup now, it may be better to just delay this feature addition until 7.1.  I also plan to make heavier use of server-side filtering when I add support in libvirt-qemu.so for tracking qemu monitor events as a development aid (since qemu events may be more frequent than libvirt events).

Comment 4 Eric Blake 2015-07-28 15:34:47 UTC
Server side filtering exists as of upstream v1.2.2 with commits such as:

commit 158795d20ea0ae928feb8a3691e6d8340cb8a92a
Author: Eric Blake <eblake>
Date:   Thu Jan 9 11:22:53 2014 -0700

    event: convert remaining domain events to new style
    
    Following the patterns established by lifecycle events, this
    creates all the new RPC calls needed to pass callback IDs
    for every domain event, and changes the limits in client and
    server codes to use modern style when possible.
    
    I've tested all combinations: both 'old client and new server'
    and 'new client and old server' continue to work with the old
    RPCs, and 'new client and new server' benefit from server-side
    filtering with the new RPCs.
    
    * src/remote/remote_protocol.x (REMOTE_PROC_DOMAIN_EVENT_*): Add
    REMOTE_PROC_DOMAIN_EVENT_CALLBACK_* counterparts.
    * daemon/remote.c (remoteRelayDomainEvent*): Send callbackID via
    newer RPC when used with new-style registration.
    (remoteDispatchConnectDomainEventCallbackRegisterAny): Extend to
    cover all domain events.
    * src/remote/remote_driver.c (remoteDomainBuildEvent*): Add new
    Callback and Helper functions.
    (remoteEvents): Match order of RPC numbers, register new handlers.
    (remoteConnectDomainEventRegisterAny)
    (remoteConnectDomainEventDeregisterAny): Extend to cover all
    domain events.
    * src/remote_protocol-structs: Regenerate.
    
    Signed-off-by: Eric Blake <eblake>

Comment 6 zhenfeng wang 2015-09-15 10:23:34 UTC
Hi Eric
I'm verify this bug right now, Unluckly, i couldn't reproduce it following the comment0's steps, so can you show me some more details steps or a reproducer for this bug reproduce? so that i could verify this bug sufficiently, Thanks in advance.

Comment 7 zhenfeng wang 2015-09-17 08:16:40 UTC
Hi Eric
The following steps were my reproduce and verify steps, can you help check whether they're ok to verify this bug thanks

Reproduce steps
1.Prepare 2 host, 1 libvirtd server with a old packet and 1 libvirtd client with the latest packet
Server:
libvirt-1.1.1-29.el7.x86_64
client:
libvirt-client-1.2.17-9.el7.x86_64

2.Prepare 3 guests in the libvirtd server
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 7     vm1                            running
 10    vm2                            running
 -     win7                           shut off


3.Connect the libvirtd server from the libvirtd client,Register to receive events for a single domain in a client, with debugging turned on to track async messages from the server

client#LIBVIRT_DEBUG=1 virsh -c qemu+ssh://$server_ip/system
virsh# event vm1 --all


4.Connect the libvirtd server with the server's local virsh client, then do some operates on the vm2 or win7
server#virsh -c qemu:///system 
virsh # start win7
Domain win7 started

virsh # destroy vm2
Domain vm2 destroyed

virsh # start vm2
Domain vm2 started


5.Check the debug logs on the step3's libvirtd client, the debug traffic show that libvirtd server also send the other domain's event to this client

2015-09-17 07:42:43.279+0000: 5575: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed04000a20 classname=virDomainEventLifecycle
--
2015-09-17 07:42:49.437+0000: 5575: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed04000a00 classname=virDomainEventLifecycle
--
2015-09-17 07:43:12.350+0000: 5575: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed04000a20 classname=virDomainEventLifecycle


Verify steps
1.Both server and client use the latest libvirt packet
 Server:
libvirt-1.2.17-9.el7.x86_64
client:
libvirt-client-1.2.17-9.el7.x86_64

2.Re-excute step 2~4
3.check the debug logs on the libvirtd client, the events for other domain won't show in this client 

4.do some operations with vm1, could get the corresponding event for vm1

server#virsh -c qemu:///system 
virsh#destroy vm1


2015-09-17 07:55:27.174+0000: 5700: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fb1b0000a90 classname=virDomainEventLifecycle

Comment 8 zhenfeng wang 2015-09-17 08:18:55 UTC
Hi Eric
Please ignore the comment6 and check comment7 directly, thanks

Comment 9 Eric Blake 2015-09-18 14:18:02 UTC
Yes, comment 7 confirms that the old server did not do server-side filtering, while the new server does; it also confirms that the new client is interoperable with either version of server.  You've come up with a good test.

Comment 10 zhenfeng wang 2015-09-21 02:13:17 UTC
Thanks for Eric's response, so mark this bug verified

Comment 12 errata-xmlrpc 2015-11-19 05:44:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html


Note You need to log in before you can comment on or make changes to this bug.