Bug 1225173 - Not capturing events properly from RHOS (QPID)
Summary: Not capturing events properly from RHOS (QPID)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.4.0
Assignee: Greg Blomquist
QA Contact: Pete Savage
URL:
Whiteboard:
Depends On: 1224389
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-26 18:36 UTC by Greg Blomquist
Modified: 2015-06-24 20:39 UTC (History)
8 users (show)

Fixed In Version: 5.4.0.5
Doc Type: Bug Fix
Doc Text:
Clone Of: 1224389
Environment:
Last Closed: 2015-06-16 13:02:50 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:1100 0 normal SHIPPED_LIVE CFME 5.4.0 bug fixes, and enhancement update 2015-06-16 16:28:42 UTC

Comment 1 CFME Bot 2015-05-26 19:36:09 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=0b9b0c2c913a093abfabd6a312c6abe603354eb3

commit 0b9b0c2c913a093abfabd6a312c6abe603354eb3
Author:     Greg Blomquist <gblomqui>
AuthorDate: Fri May 22 16:07:28 2015 -0400
Commit:     Greg Blomquist <gblomqui>
CommitDate: Tue May 26 15:13:47 2015 -0400

    Whitespace cleanup
    
    Cleanup some bad use of whitespce in OpenstackQpidReceiver.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1225173

 lib/openstack/amqp/openstack_qpid_receiver.rb | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

Comment 2 CFME Bot 2015-05-26 19:36:13 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=b2c9f69777d52dac4956533c7a008c4829453f2c

commit b2c9f69777d52dac4956533c7a008c4829453f2c
Author:     Greg Blomquist <gblomqui>
AuthorDate: Sat May 23 10:16:25 2015 -0400
Commit:     Greg Blomquist <gblomqui>
CommitDate: Tue May 26 15:14:13 2015 -0400

    Get rid of required "options" in Qpid connection
    
    OpenstackQpidConnection#initialize accepted an options hash but required some of
    the "options".
    
    Now the initialize method takes in actual parameters making it clearer what
    needs to be used to create a Qpid connection.
    
    At this point, this refactoring isn't being propagated to the
    OpenstackEventMonitor so as not to break the existing interface.  But, I think
    that refactoring the OpenstackEventMonitor initializer is needed as well to
    clear up that interface as well.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1225173

 lib/openstack/amqp/openstack_qpid_connection.rb    | 35 ++++++++++------------
 lib/openstack/amqp/openstack_qpid_event_monitor.rb | 23 ++++++++++++--
 2 files changed, 36 insertions(+), 22 deletions(-)

Comment 3 CFME Bot 2015-05-26 19:36:16 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=5fa43c82bcbfd159de99b1cf894447b56b1e4f50

commit 5fa43c82bcbfd159de99b1cf894447b56b1e4f50
Author:     Greg Blomquist <gblomqui>
AuthorDate: Sat May 23 10:38:04 2015 -0400
Commit:     Greg Blomquist <gblomqui>
CommitDate: Tue May 26 15:14:30 2015 -0400

    Accommodate qpid exchange names
    
    Openstack supports two different types of Qpid Exchange names:
    
     1. <service_name> (e.g., "nova)
     2. amq.topic/topic/<service_name> (e.g., "amq.topic/topic/nova")
    
    Originally, ManageIQ only supported the first type of exchange name.  Now the
    OpenstackQpidReceiver can support both type of exchange names.
    
    Additionally, when creating the real qpid receiver, a uniquely named queue was
    automatically created by the qpid library.  The qpid receiver would connect to
    this queue, which receives the actual messages from the exchange.  The queue
    name looked like:
    
     * <service_name>_<uuid>
       - e.g., "nova_123-456-789"
    
    Now the name of the queue is controlled by the OpenstackQpidReceiver and looks
    like:
    
     * miq-<host>-<exchange_name>
       - e.g., "miq-10.10.10.10-amq.topic_topic_nova
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1225173

 lib/openstack/amqp/openstack_qpid_event_monitor.rb | 22 +++++++++++++---
 lib/openstack/amqp/openstack_qpid_receiver.rb      | 30 ++++++++++++++--------
 2 files changed, 38 insertions(+), 14 deletions(-)

Comment 4 CFME Bot 2015-05-26 19:36:19 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=b907f2b383276984b504f812ffd0d976511d7032

commit b907f2b383276984b504f812ffd0d976511d7032
Author:     Greg Blomquist <gblomqui>
AuthorDate: Sat May 23 11:00:09 2015 -0400
Commit:     Greg Blomquist <gblomqui>
CommitDate: Tue May 26 15:14:45 2015 -0400

    Update Qpid tests
    
    The various qpid tests need to be updated to work with the changes to the
    OpenstackQpidEventMonitor and associated classes.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1225173

 .../openstack/amqp/openstack_qpid_connection_spec.rb  |  7 +------
 .../amqp/openstack_qpid_event_monitor_spec.rb         | 19 +++++++++++++++----
 .../openstack/amqp/openstack_qpid_receiver_spec.rb    |  6 +++++-
 3 files changed, 21 insertions(+), 11 deletions(-)

Comment 6 Pete Savage 2015-06-02 21:55:54 UTC
It appears as though we make multiple connections to the same queue, as such it needs to be established this way. It appears this is not the case as per the logs below.

/opt/rh/ruby200/root/usr/share/gems/gems/qpid_messaging-0.20.2/lib/qpid_messaging/receiver.rb:137:in `available'
/var/www/miq/lib/openstack/amqp/openstack_qpid_receiver.rb:44:in `get_notifications'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:73:in `block in each_batch'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each_batch'
[----] E, [2015-06-02T17:50:18.914945 #30805:354ecd4] ERROR -- : MIQ(EventCatcherOpenstack) EMS [xx.xx.xx.xx] as [admin] Event Monitor Thread aborted because [resource-locked: resource-locked: Cannot grant exclusive access to queue miq-xx.xx.xx.xx-cinder (qpid/broker/SessionAdapter.cpp:332)]
[----] E, [2015-06-02T17:50:18.915216 #30805:354ecd4] ERROR -- : [MessagingError]: resource-locked: resource-locked: Cannot grant exclusive access to queue miq-xx.xx.xx.xx-cinder (qpid/broker/SessionAdapter.cpp:332)  Method:[rescue in block in start_event_monitor]
[----] E, [2015-06-02T17:50:18.915349 #30805:354ecd4] ERROR -- : /opt/rh/ruby200/root/usr/share/gems/gems/qpid_messaging-0.20.2/lib/qpid_messaging/receiver.rb:137:in `getAvailable'
/opt/rh/ruby200/root/usr/share/gems/gems/qpid_messaging-0.20.2/lib/qpid_messaging/receiver.rb:137:in `available'
/var/www/miq/lib/openstack/amqp/openstack_qpid_receiver.rb:44:in `get_notifications'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:73:in `block in each_batch'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each_batch'
[----] E, [2015-06-02T17:50:33.917395 #30805:355d4dc] ERROR -- : MIQ(EventCatcherOpenstack) EMS [xx.xx.xx.xx] as [admin] Event Monitor Thread aborted because [resource-locked: resource-locked: Cannot grant exclusive access to queue miq-xx.xx.xx.xx-cinder (qpid/broker/SessionAdapter.cpp:332)]
[----] E, [2015-06-02T17:50:33.917648 #30805:355d4dc] ERROR -- : [MessagingError]: resource-locked: resource-locked: Cannot grant exclusive access to queue miq-xx.xx.xx.xx-cinder (qpid/broker/SessionAdapter.cpp:332)  Method:[rescue in block in start_event_monitor]
[----] E, [2015-06-02T17:50:33.918055 #30805:355d4dc] ERROR -- : /opt/rh/ruby200/root/usr/share/gems/gems/qpid_messaging-0.20.2/lib/qpid_messaging/receiver.rb:137:in `getAvailable'
/opt/rh/ruby200/root/usr/share/gems/gems/qpid_messaging-0.20.2/lib/qpid_messaging/receiver.rb:137:in `available'
/var/www/miq/lib/openstack/amqp/openstack_qpid_receiver.rb:44:in `get_notifications'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:73:in `block in each_batch'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each_batch'
[----] E, [2015-06-02T17:50:48.920380 #30805:35659d4] ERROR -- : MIQ(EventCatcherOpenstack) EMS [xx.xx.xx.xx] as [admin] Event Monitor Thread aborted because [resource-locked: resource-locked: Cannot grant exclusive access to queue miq-xx.xx.xx.xx-cinder (qpid/broker/SessionAdapter.cpp:332)]
[----] E, [2015-06-02T17:50:48.920698 #30805:35659d4] ERROR -- : [MessagingError]: resource-locked: resource-locked: Cannot grant exclusive access to queue miq-xx.xx.xx.xx-cinder (qpid/broker/SessionAdapter.cpp:332)  Method:[rescue in block in start_event_monitor]
[----] E, [2015-06-02T17:50:48.921152 #30805:35659d4] ERROR -- : /opt/rh/ruby200/root/usr/share/gems/gems/qpid_messaging-0.20.2/lib/qpid_messaging/receiver.rb:137:in `getAvailable'
/opt/rh/ruby200/root/usr/share/gems/gems/qpid_messaging-0.20.2/lib/qpid_messaging/receiver.rb:137:in `available'
/var/www/miq/lib/openstack/amqp/openstack_qpid_receiver.rb:44:in `get_notifications'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:73:in `block in each_batch'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each'
/var/www/miq/lib/openstack/amqp/openstack_qpid_event_monitor.rb:72:in `each_batch'

Comment 7 CFME Bot 2015-06-04 18:01:54 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=0b6cb3a145a4bc2fa994f2a8a9f2d91a2b9d8a24

commit 0b6cb3a145a4bc2fa994f2a8a9f2d91a2b9d8a24
Author:     Greg Blomquist <gblomqui>
AuthorDate: Wed Jun 3 16:16:05 2015 -0400
Commit:     Greg Blomquist <gblomqui>
CommitDate: Thu Jun 4 13:52:33 2015 -0400

    Fix custom naming for AMQP binding queues
    
    The original fixes for the bugs linked below used the OpenStack server's IP
    address as the hostname information in the binding queue name.  This meant that
    any two appliances that attempted to connect to a single OpenStack env would
    create the same named binding queues.
    
    This fix uses the appliance's ip address in the binding queue name, making the
    name of the binding queue unique per appliance.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1225178
    https://bugzilla.redhat.com/show_bug.cgi?id=1225173
    
    Upstream PR: https://github.com/ManageIQ/manageiq/pull/3050

 lib/openstack/amqp/openstack_qpid_event_monitor.rb             | 10 ++++++----
 lib/openstack/amqp/openstack_qpid_receiver.rb                  |  5 +++--
 lib/openstack/amqp/openstack_rabbit_event_monitor.rb           |  6 ++++--
 lib/spec/openstack/amqp/openstack_qpid_event_monitor_spec.rb   |  2 +-
 lib/spec/openstack/amqp/openstack_qpid_receiver_spec.rb        |  2 +-
 lib/spec/openstack/amqp/openstack_rabbit_event_monitor_spec.rb |  2 +-
 vmdb/lib/workers/mixins/event_catcher_openstack_mixin.rb       |  4 +++-
 7 files changed, 19 insertions(+), 12 deletions(-)

Comment 8 John Prause 2015-06-04 18:06:09 UTC
This fix will be in the 5.4.0.5 build.

Comment 9 Pete Savage 2015-06-09 12:44:29 UTC
Verified in 5.4.0.5

Comment 11 errata-xmlrpc 2015-06-16 13:02:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1100.html

Comment 12 CFME Bot 2015-06-24 20:39:22 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=bdc878afed77d8dcb539191dadb1d05dbb875421

commit bdc878afed77d8dcb539191dadb1d05dbb875421
Author:     Greg Blomquist <gblomqui>
AuthorDate: Tue Jun 23 13:30:07 2015 -0400
Commit:     Greg Blomquist <gblomqui>
CommitDate: Wed Jun 24 16:05:37 2015 -0400

    Include miq_server when retrieving worker
    
    To try to make the way the OpenStack event catcher creates binding queues
    work a little better, the appliance's IP address was looked up and used as part
    of the binding queue's name.
    
    However, there were a couple of things working against this fix.  First, the
    appliance's IP address was not readily available to the worker process.  Second,
    ManageIQ has a DB connection pool with only one connection.  And, threads (i.e.,
    where event catcher workers do all their work) that attempt to run queries are
    opening a new DB connection.
    
    The original fix never actually tried opening the a new connection.  Instead, it
    was perfectly happy to get back a nil value for the appliance and try to lookup
    Nil#ipaddress.
    
    This fix gets around this problem by throwing the appliance record (miq_server,
    actually) into an ivar and making that available to the thread.  This keeps the
    thread from having to query for the miq_server, while still giving it access to
    the MiqServer#ipaddress.
    
    Original PR:
    https://github.com/ManageIQ/manageiq/pull/3050
    
    Fixes:
    https://bugzilla.redhat.com/show_bug.cgi?id=1233798
    
    References:
    https://bugzilla.redhat.com/show_bug.cgi?id=1225173
    https://bugzilla.redhat.com/show_bug.cgi?id=1225178

 vmdb/lib/workers/mixins/event_catcher_openstack_mixin.rb |  2 +-
 vmdb/lib/workers/worker_base.rb                          | 13 +++++++------
 2 files changed, 8 insertions(+), 7 deletions(-)

Comment 13 CFME Bot 2015-06-24 20:39:35 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=36efb1c837f179d701422ea3a4aae77e84c8d6bd

commit 36efb1c837f179d701422ea3a4aae77e84c8d6bd
Merge: 3ae711d bdc878a
Author:     Joe Rafaniello <jrafanie>
AuthorDate: Wed Jun 24 16:37:18 2015 -0400
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Wed Jun 24 16:37:18 2015 -0400

    Merge branch 'bz1233798-backport_worker_base_for_bz1232484' into '5.4.z'
    
    Include miq_server when retrieving worker
    
    To try to make the way the OpenStack event catcher creates binding queues work a little better, the appliance's IP address was looked up and used as part
    of the binding queue's name.
    
    However, there were a couple of things working against this fix.  First, the appliance's IP address was not readily available to the worker process.  Second,
    ManageIQ has a DB connection pool with only one connection.  And, threads (i.e., where event catcher workers do all their work) that attempt to run queries are
    opening a new DB connection.
    
    The original fix never actually tried opening the a new connection.  Instead, it was perfectly happy to get back a nil value for the appliance and try to lookup
    Nil#ipaddress.
    
    This fix gets around this problem by throwing the appliance record (miq_server, actually) into an ivar and making that available to the thread.  This keeps the
    thread from having to query for the miq_server, while still giving it access to the MiqServer#ipaddress.
    
    Upstream PR:
    https://github.com/ManageIQ/manageiq/pull/3244
    
    No cherry-pick conflicts
    
    Fixes:
    https://bugzilla.redhat.com/show_bug.cgi?id=1233798
    
    References:
    https://bugzilla.redhat.com/show_bug.cgi?id=1225173
    https://bugzilla.redhat.com/show_bug.cgi?id=1225178
    
    See merge request !145

 vmdb/lib/workers/mixins/event_catcher_openstack_mixin.rb |  2 +-
 vmdb/lib/workers/worker_base.rb                          | 13 +++++++------
 2 files changed, 8 insertions(+), 7 deletions(-)


Note You need to log in before you can comment on or make changes to this bug.