Bug 500712

Summary: QMF queries to the broker may return records for deleted objects
Product: Red Hat Enterprise MRG Reporter: Ted Ross <tross>
Component: distributionAssignee: Ted Ross <tross>
Status: CLOSED ERRATA QA Contact: Jan Sarenik <jsarenik>
Severity: high Docs Contact:
Priority: high    
Version: 1.1.1CC: gsim, jsarenik
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, when a QMF get-query was sent to the broker, the broker replied with indications for all matching objects plus objects that have been deleted since the last mgmt-publish-interval. This could result in inconsistent management traffic in clusters because the publish-intervals were not synchronized across all cluster nodes even though the states of the nodes were completely consistent. Note that this was not an issue for API users because the console API filters result for deleted objects. With this update, clusters no longer suffer from inconsistent management traffic.
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-14 16:10:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Reproducer script
none
Updated reproducer/verifier none

Description Ted Ross 2009-05-13 19:23:05 UTC
When a qmf get-query is sent to the broker, the broker will reply with indications for all matching objects plus objects that have been deleted since the last mgmt-publish-interval.

This is an issue for clusters because the publish-intervals are not synchronized across cluster nodes and this can result in inconsistent management traffic even though the state of the nodes is completely consistent.

This is not seen by console API users because the console API filters results for deleted objects.

Comment 1 Ted Ross 2009-05-13 19:25:25 UTC
A fix for this issue has been committed upstream at revision 774438.

Comment 3 Jan Sarenik 2010-02-19 11:32:19 UTC
How can I test this if it's not seen by the API?

Comment 4 Ted Ross 2010-02-19 14:29:56 UTC
You can apply the following patch to the API code to suppress this filtering.  With this patch, you can create an object (say a queue), delete it, then query for all the queues to see if it's been removed from the set.  If this bug is in effect, the deleted queue is very likely to be in the list of results from the getObjects query.

Index: extras/qmf/src/py/qmf/console.py
===================================================================
--- extras/qmf/src/py/qmf/console.py	(revision 911184)
+++ extras/qmf/src/py/qmf/console.py	(working copy)
@@ -915,7 +915,7 @@
     try:
       self.cv.acquire()
       if seq in self.syncSequenceList:
-        if object.getTimestamps()[2] == 0 and self._selectMatch(object):
+        if self._selectMatch(object):
           self.getResult.append(object)
         return
     finally:

Comment 5 Jan Sarenik 2010-02-22 20:22:03 UTC
Created attachment 395561 [details]
Reproducer script

Comment 6 Jan Sarenik 2010-02-22 20:34:44 UTC
Reproduced on RHEL5 (i386):
  qpidd-0.5.752581-34.el5
  python-qpid-0.5.752581-4.el5

Verified on
  RHEL5 (i386, x86_64):
    qpid-cpp-server-0.7.908272-1.el5
    python-qpid-0.5.904641-1.el5

The bug is still there on RHEL4, packages
  qpidd-0.5.752581-33.el4
  python-qpid-0.5.760500-5.el4
so I will wait for update here before re-testing.

Comment 7 Jan Sarenik 2010-03-04 15:46:15 UTC
Created attachment 397832 [details]
Updated reproducer/verifier

Verified on RHEL4 as well (i386, x86_64)
  qpid-cpp-server-0.7.916826-2.el4
  qpid-tools-0.7.917557-4.el4

Comment 9 Martin Prpič 2010-10-06 12:29:40 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, when a QMF get-query was sent to the broker, the broker replied with indications for all matching objects plus objects that have been deleted since the last mgmt-publish-interval. This could result in inconsistent management traffic in clusters because the publish-intervals were not synchronized across all cluster nodes even though the states of the nodes were completely consistent. Note that this was not an issue for API users because the console API filters result for deleted objects. With this update, clusters no longer suffer from inconsistent management traffic.

Comment 11 errata-xmlrpc 2010-10-14 16:10:46 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html