Bug 1358963 - ec2 event monitor spams database connection errors
Summary: ec2 event monitor spams database connection errors
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.5.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: GA
: 5.5.5
Assignee: Joe Rafaniello
QA Contact: Matouš Mojžíš
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-21 23:33 UTC by Matouš Mojžíš
Modified: 2016-09-06 14:26 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-26 15:37:36 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)
evm.log (4.65 KB, text/plain)
2016-07-21 23:33 UTC, Matouš Mojžíš
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1488 normal SHIPPED_LIVE CFME 5.5.5 bug fixes and enhancement update 2016-07-26 19:28:27 UTC

Description Matouš Mojžíš 2016-07-21 23:33:41 UTC
Created attachment 1182672 [details]
evm.log

Description of problem:
I couldn't reproduce it in 5.5.4.2, so it's broken in 5.5.5.x only.
EventCatcher is not working properly, so I have to do manual refresh and automatic refresh/discovery doesn't work.
It doesn't spam it every five seconds, but it's like 1ten times per minute.

Version-Release number of selected component (if applicable):
5.5.5.0

How reproducible:
Always

Steps to Reproduce:
1. Add an ec2 provider
2.
3.

Actual results:
evm.log in attachment.



Expected results:


Additional info:

Comment 5 CFME Bot 2016-07-22 17:41:56 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a2e0ab9f1ae842ae6eac987b6c8b1c4b20cf63be

commit a2e0ab9f1ae842ae6eac987b6c8b1c4b20cf63be
Author:     Joe Rafaniello <jrafanie@redhat.com>
AuthorDate: Fri Jul 22 12:44:53 2016 -0400
Commit:     Joe Rafaniello <jrafanie@redhat.com>
CommitDate: Fri Jul 22 12:44:53 2016 -0400

    Accept an input vmdb config instead of always hitting the db
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963

 lib/vmdb/util.rb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 6 CFME Bot 2016-07-22 17:42:00 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=eff2db4d0ca449481db3db0950ae2ff4cd19dcce

commit eff2db4d0ca449481db3db0950ae2ff4cd19dcce
Author:     Joe Rafaniello <jrafanie@redhat.com>
AuthorDate: Fri Jul 22 12:36:59 2016 -0400
Commit:     Joe Rafaniello <jrafanie@redhat.com>
CommitDate: Fri Jul 22 12:36:59 2016 -0400

    Bump the connection pool size since 1 is excessive
    
    1 connection means you can't create a second thread and do any database query.
    It's just too easy to accidentally issue a query while not knowing this constraint.
    
    Note, this change would require you update the configuration for each appliance
    so, the next commit will prevent the db connection used in a thread.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963

 config/database.pg.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 7 CFME Bot 2016-07-22 17:42:04 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=7da8181a338c77f55ff160f09e8a496a74609fd0

commit 7da8181a338c77f55ff160f09e8a496a74609fd0
Author:     Joe Rafaniello <jrafanie@redhat.com>
AuthorDate: Fri Jul 22 12:49:29 2016 -0400
Commit:     Joe Rafaniello <jrafanie@redhat.com>
CommitDate: Fri Jul 22 13:00:37 2016 -0400

    Create the Stream with the http_proxy_uri using the existing @vmdb_config
    
    The @vmdb_config is created/updated in the MAIN thread of all
    MiqWorker::Runner during sync_config[1] and contains the http_proxy_uri we need.
    We already have this configuration from the main thread, so we can pass along
    the calculated http_proxy_uri to the Stream.
    
    This prevents the Thread created in
    ManageIQ::Providers::BaseManager::EventCatcher::Runner#start_event_monitor[2]
    from connecting to the database.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    [1] https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/blob/63646d7bf3472f2034eaf6b8510ced3afaddcd97/app/models/miq_worker/runner.rb#L284
    [2] https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/blob/77ef3f32c330d45414e9b3e3857ef1d2cb97fd73/app/models/manageiq/providers/base_manager/event_catcher/runner.rb#L114-128

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 3 ++-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 6 ++++--
 2 files changed, 6 insertions(+), 3 deletions(-)

Comment 8 CFME Bot 2016-07-22 17:42:09 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=c3cd7e2bade964fddd22acc5454ad5e683174f72

commit c3cd7e2bade964fddd22acc5454ad5e683174f72
Merge: c917133 7da8181
Author:     Gregg Tanzillo <gtanzill@redhat.com>
AuthorDate: Fri Jul 22 13:38:07 2016 -0400
Commit:     Gregg Tanzillo <gtanzill@redhat.com>
CommitDate: Fri Jul 22 13:38:07 2016 -0400

    Merge branch 'aws_event_catcher_dont_connect_to_db_in_thread' into '5.5.z'
    
    Don't connect to the database during the AWS event catcher subthread
    
    Note, this occurs only in 5.5.5+ via https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/commit/77ef3f32c330d45414e9b3e3857ef1d2cb97fd73
    
    A different solution will need to be done in upstream.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    
    * Bump the connection pool size since 1 is excessive  (this forces users/customers to update all appliance configurations, so follow up commits were done)
    
    * Update a method so you can accept an input vmdb config instead of always hitting the db to get one
    
    * Create the Stream with the http_proxy_uri using the existing @vmdb_config (avoids hitting the db in the subthread)
    
    Please review each commit.
    
    I tested this manually by creating a bogus aws cloud provider, changed verify_credentials? to always return true for aws, then restarting the appliance.  It fails at boot each time it tries to start the amazon event catcher worker.
    
    
    See merge request !1022

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 3 ++-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 6 ++++--
 config/database.pg.yml                                              | 2 +-
 lib/vmdb/util.rb                                                    | 4 ++--
 4 files changed, 9 insertions(+), 6 deletions(-)

Comment 10 Matouš Mojžíš 2016-07-25 13:01:46 UTC
Verified in: 5.5.5.4. After adding ec2 provider and waiting for 30 minutes no errors appeared in log.

Comment 12 errata-xmlrpc 2016-07-26 15:37:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1488.html

Comment 13 CFME Bot 2016-09-06 14:26:52 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=62e98284023e17d4466258857ede942983e6db69

commit 62e98284023e17d4466258857ede942983e6db69
Merge: 75afa46 ed6cc46
Author:     Greg Blomquist <gblomqui@redhat.com>
AuthorDate: Tue Sep 6 10:25:40 2016 -0400
Commit:     Greg Blomquist <gblomqui@redhat.com>
CommitDate: Tue Sep 6 10:25:40 2016 -0400

    Merge branch 'fix_aws_sns_topic_lost' into '5.5.z'
    
    fix wrong not working event catcher for aws
    
    before the default aws_config topic would be set to nil by a wrong call to Stream.new
    
    this was introduced in 7da8181a338c77f55ff160f09e8a496a74609fd0
    
    the original was a fix for
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    this one fixes
    https://bugzilla.redhat.com/show_bug.cgi?id=1367872
    
    See merge request !1041

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb   | 2 +-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

Comment 14 CFME Bot 2016-09-06 14:26:58 UTC
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=ed6cc46062232bc3545bb279a71c97cc773bd87f

commit ed6cc46062232bc3545bb279a71c97cc773bd87f
Author:     Marcel Hild <hild@b4mad.net>
AuthorDate: Wed Aug 17 19:09:22 2016 +0200
Commit:     Marcel Hild <hild@b4mad.net>
CommitDate: Wed Aug 17 19:09:22 2016 +0200

    before the default aws_config topic would be set to nil by a wrong
    call to Stream.new
    
    this was introduced 7da8181a338c77f55ff160f09e8a496a74609fd0
    
    the original was a fix for
    https://bugzilla.redhat.com/show_bug.cgi?id=1358963
    
    this one fixes
    https://bugzilla.redhat.com/show_bug.cgi?id=1367872

 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb   | 2 +-
 .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)


Note You need to log in before you can comment on or make changes to this bug.