Bug 1354054 - Errno::ECONNREFUSED: Connection refused when dynamic dialog menus are set to refresh
Summary: Errno::ECONNREFUSED: Connection refused when dynamic dialog menus are set to ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Automate
Version: 5.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: GA
: 5.7.0
Assignee: mkanoor
QA Contact: Shveta
URL:
Whiteboard: service:dialog
Depends On:
Blocks: 1356704
TreeView+ depends on / blocked
 
Reported: 2016-07-08 21:03 UTC by Jared Deubel
Modified: 2017-01-12 05:12 UTC (History)
13 users (show)

Fixed In Version: 5.7.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1356704 (view as bug list)
Environment:
Last Closed: 2017-01-11 20:03:32 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)
Proposed Patch (3.66 KB, application/x-gzip)
2016-07-19 18:03 UTC, mkanoor
no flags Details

Description Jared Deubel 2016-07-08 21:03:04 UTC
Description of problem:
This is an issue related to dynamic dialog menus. If the menu item is set to refresh after updating another field this connection problem occurs (shown below). If refresh is set to manual I do not see this issue. This was not a problem in the 4.0 version

=================
[----] E, [2016-07-07T16:59:29.988334 #3089:14c9ae0] ERROR -- : <AutomationEngine> Method STDERR: /opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:744:in `rescue in block in open': druby://127.0.0.1:33938 - #<Errno::ECONNREFUSED: Connection refused - connect(2) for "127.0.0.1" port 33938> (DRb::DRbConnError)
====================

Noticed that this seems to be a similar issue to: https://github.com/ManageIQ/manageiq/issues/8626


Upping priority on this ticket. Database connection errors are preventing provisioning


Version-Release number of selected component (if applicable):
5.6

Comment 6 mkanoor 2016-07-19 18:03:14 UTC
Created attachment 1181756 [details]
Proposed Patch

The gzipped tar archive has a single file lib/miq_automation_engine/engine/miq_ae_engine.rb
On a Test appliance please make a backup of the existing file lib/miq_automation_engine/engine/miq_ae_engine.rb and replace it with the enclosed file.

Comment 7 mkanoor 2016-07-19 18:04:34 UTC
The problem was happening because of multiple requests coming into the Automate Engine. Now we have a mutex that would serialize the requests so that only one request can run at the time.

Comment 8 CFME Bot 2016-07-19 19:05:54 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/cbb5661645090d377341d72567697b6ee1d1784a

commit cbb5661645090d377341d72567697b6ee1d1784a
Author:     Madhu Kanoor <mkanoor@redhat.com>
AuthorDate: Mon Jul 18 17:44:44 2016 -0400
Commit:     Madhu Kanoor <mkanoor@redhat.com>
CommitDate: Tue Jul 19 12:46:08 2016 -0400

    Use a mutex to prevent concurrent deliver calls
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1354054

 lib/miq_automation_engine/engine/miq_ae_engine.rb | 5 +++++
 1 file changed, 5 insertions(+)

Comment 9 mkanoor 2016-07-21 14:24:02 UTC
Hi Jared,
Did the customer get a chance to test out the patch?
Thanks,
Madhu

Comment 11 CFME Bot 2016-07-26 01:45:49 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/a1656fce7e861a92a671390648535533d753f6a6

commit a1656fce7e861a92a671390648535533d753f6a6
Author:     Madhu Kanoor <mkanoor@redhat.com>
AuthorDate: Mon Jul 25 12:06:51 2016 -0400
Commit:     Madhu Kanoor <mkanoor@redhat.com>
CommitDate: Mon Jul 25 12:06:51 2016 -0400

    Allow symbols to load from DRb threads
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1354054
    
    use ActiveSupport::Dependencies.interlock.permit_concurrent_loads
    to allow for Automate Methods/Drb to load symbols

 lib/miq_automation_engine/engine/miq_ae_engine.rb | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

Comment 12 Shveta 2016-09-19 19:54:53 UTC
Fixed.
verified in 5.7.0.0.20160906172503_12f65cb

Comment 13 tachoi 2016-10-28 04:53:00 UTC
Hi Team

One of customer applied this patch in their testbed
But after that, customer is not able to login CFME any more.

Problem Description
#############################
Customer version: 5.6.0.13-20160624114606_13a9153
CFME login is stuck on spinning wheel status when user try to login

Action Taken
#############################
Once we rolled-back to original miq_ae_engine.rb file, the login was successful within a min
We have tested with userid 756691 with both scenarios
We did restart evmserver around 2016-10-28T13:37:26 system time to rollback to original rb file(I will attach log file separately)

This might be related or not
When patched ruby file is in use, I observed below error from ssl_error.log

[Fri Oct 28 13:29:57.240835 2016] [proxy_http:error] [pid 3537] (70007)The timeout specified has expired: [client 10.67.119.246:60057] AH01102: error reading status line from remote server 0.0.0.0:3001, referer: https://lrlcfa50/
[Fri Oct 28 13:29:57.240901 2016] [proxy:error] [pid 3537] [client 10.67.119.246:60057] AH00898: Error reading from remote server returned by /dashboard/authenticate, referer: https://lrlcfa50/
[Fri Oct 28 13:34:37.675795 2016] [proxy_http:error] [pid 3537] (70007)The timeout specified has expired: [client 10.67.119.246:60287] AH01102: error reading status line from remote server 0.0.0.0:3001
[Fri Oct 28 13:34:37.675862 2016] [proxy:error] [pid 3537] [client 10.67.119.246:60287] AH00898: Error reading from remote server returned by /
[Fri Oct 28 13:35:21.341204 2016] [proxy_http:error] [pid 3562] (70007)The timeout specified has expired: [client 10.67.119.246:60317] AH01102: error reading status line from remote server 0.0.0.0:3001, referer: https://lrlcfa50/
[Fri Oct 28 13:35:21.341298 2016] [proxy:error] [pid 3562] [client 10.67.119.246:60317] AH00898: Error reading from remote server returned by /dashboard/authenticate, referer: https://lrlcfa50/

Once we rolled-back to original ruby one, I don't see this msg any more.


Note You need to log in before you can comment on or make changes to this bug.