Created attachment 1182672 [details] evm.log Description of problem: I couldn't reproduce it in 5.5.4.2, so it's broken in 5.5.5.x only. EventCatcher is not working properly, so I have to do manual refresh and automatic refresh/discovery doesn't work. It doesn't spam it every five seconds, but it's like 1ten times per minute. Version-Release number of selected component (if applicable): 5.5.5.0 How reproducible: Always Steps to Reproduce: 1. Add an ec2 provider 2. 3. Actual results: evm.log in attachment. Expected results: Additional info:
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a2e0ab9f1ae842ae6eac987b6c8b1c4b20cf63be commit a2e0ab9f1ae842ae6eac987b6c8b1c4b20cf63be Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Jul 22 12:44:53 2016 -0400 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Jul 22 12:44:53 2016 -0400 Accept an input vmdb config instead of always hitting the db https://bugzilla.redhat.com/show_bug.cgi?id=1358963 lib/vmdb/util.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=eff2db4d0ca449481db3db0950ae2ff4cd19dcce commit eff2db4d0ca449481db3db0950ae2ff4cd19dcce Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Jul 22 12:36:59 2016 -0400 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Jul 22 12:36:59 2016 -0400 Bump the connection pool size since 1 is excessive 1 connection means you can't create a second thread and do any database query. It's just too easy to accidentally issue a query while not knowing this constraint. Note, this change would require you update the configuration for each appliance so, the next commit will prevent the db connection used in a thread. https://bugzilla.redhat.com/show_bug.cgi?id=1358963 config/database.pg.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=7da8181a338c77f55ff160f09e8a496a74609fd0 commit 7da8181a338c77f55ff160f09e8a496a74609fd0 Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Jul 22 12:49:29 2016 -0400 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Jul 22 13:00:37 2016 -0400 Create the Stream with the http_proxy_uri using the existing @vmdb_config The @vmdb_config is created/updated in the MAIN thread of all MiqWorker::Runner during sync_config[1] and contains the http_proxy_uri we need. We already have this configuration from the main thread, so we can pass along the calculated http_proxy_uri to the Stream. This prevents the Thread created in ManageIQ::Providers::BaseManager::EventCatcher::Runner#start_event_monitor[2] from connecting to the database. https://bugzilla.redhat.com/show_bug.cgi?id=1358963 [1] https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/blob/63646d7bf3472f2034eaf6b8510ced3afaddcd97/app/models/miq_worker/runner.rb#L284 [2] https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/blob/77ef3f32c330d45414e9b3e3857ef1d2cb97fd73/app/models/manageiq/providers/base_manager/event_catcher/runner.rb#L114-128 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 3 ++- .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-)
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=c3cd7e2bade964fddd22acc5454ad5e683174f72 commit c3cd7e2bade964fddd22acc5454ad5e683174f72 Merge: c917133 7da8181 Author: Gregg Tanzillo <gtanzill> AuthorDate: Fri Jul 22 13:38:07 2016 -0400 Commit: Gregg Tanzillo <gtanzill> CommitDate: Fri Jul 22 13:38:07 2016 -0400 Merge branch 'aws_event_catcher_dont_connect_to_db_in_thread' into '5.5.z' Don't connect to the database during the AWS event catcher subthread Note, this occurs only in 5.5.5+ via https://gitlab.cloudforms.lab.eng.rdu2.redhat.com/cloudforms/cfme/commit/77ef3f32c330d45414e9b3e3857ef1d2cb97fd73 A different solution will need to be done in upstream. https://bugzilla.redhat.com/show_bug.cgi?id=1358963 * Bump the connection pool size since 1 is excessive (this forces users/customers to update all appliance configurations, so follow up commits were done) * Update a method so you can accept an input vmdb config instead of always hitting the db to get one * Create the Stream with the http_proxy_uri using the existing @vmdb_config (avoids hitting the db in the subthread) Please review each commit. I tested this manually by creating a bogus aws cloud provider, changed verify_credentials? to always return true for aws, then restarting the appliance. It fails at boot each time it tries to start the amazon event catcher worker. See merge request !1022 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 3 ++- .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 6 ++++-- config/database.pg.yml | 2 +- lib/vmdb/util.rb | 4 ++-- 4 files changed, 9 insertions(+), 6 deletions(-)
Verified in: 5.5.5.4. After adding ec2 provider and waiting for 30 minutes no errors appeared in log.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1488.html
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=62e98284023e17d4466258857ede942983e6db69 commit 62e98284023e17d4466258857ede942983e6db69 Merge: 75afa46 ed6cc46 Author: Greg Blomquist <gblomqui> AuthorDate: Tue Sep 6 10:25:40 2016 -0400 Commit: Greg Blomquist <gblomqui> CommitDate: Tue Sep 6 10:25:40 2016 -0400 Merge branch 'fix_aws_sns_topic_lost' into '5.5.z' fix wrong not working event catcher for aws before the default aws_config topic would be set to nil by a wrong call to Stream.new this was introduced in 7da8181a338c77f55ff160f09e8a496a74609fd0 the original was a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1358963 this one fixes https://bugzilla.redhat.com/show_bug.cgi?id=1367872 See merge request !1041 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 2 +- .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=ed6cc46062232bc3545bb279a71c97cc773bd87f commit ed6cc46062232bc3545bb279a71c97cc773bd87f Author: Marcel Hild <hild> AuthorDate: Wed Aug 17 19:09:22 2016 +0200 Commit: Marcel Hild <hild> CommitDate: Wed Aug 17 19:09:22 2016 +0200 before the default aws_config topic would be set to nil by a wrong call to Stream.new this was introduced 7da8181a338c77f55ff160f09e8a496a74609fd0 the original was a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1358963 this one fixes https://bugzilla.redhat.com/show_bug.cgi?id=1367872 .../manageiq/providers/amazon/cloud_manager/event_catcher/runner.rb | 2 +- .../manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)