Bug 847685 - Sometimes update domain was successful but CLI returns timeout error
Sometimes update domain was successful but CLI returns timeout error
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Command Line Interface (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: John (J5) Palmieri
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-13 05:36 EDT by Jianwei Hou
Modified: 2015-05-14 22:02 EDT (History)
3 users (show)

See Also:
Fixed In Version: rhc-0.97.13-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-17 17:29:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jianwei Hou 2012-08-13 05:36:51 EDT
Description of problem:
update domain was successful, but CLI returns timeout error 

Version-Release number of selected component (if applicable):
on devenv_1975
cli: rhc-0.97.5

How reproducible:
sometimes

Steps to Reproduce:
1.Create a domain and create applications in domain
2.Update domain's namespace
  rhc domain update -n 1975t7 -d --trace
3.Run rhc domain show to get domain information, and check all apps are avaiable after updating namespace
  
Actual results:
After step 2:
Password: **

D, [2012-08-13T05:10:29.816554 #7099] DEBUG -- : Connecting to https://ec2-23-22-28-192.compute-1.amazonaws.com/broker/rest/api
D, [2012-08-13T05:10:33.299544 #7099] DEBUG -- : Getting all domains

Updating domain '1975t6' to namespace '1975t7'

RESULT:
/usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest.rb:75:in `rescue in request': Connection to server timed out or got interrupted: Request Timeout (Rhc::Rest::ConnectionException)
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest.rb:66:in `request'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest/domain.rb:56:in `update'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands/domain.rb:42:in `block in update'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:182:in `call'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:182:in `section'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:199:in `paragraph'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands/domain.rb:40:in `update'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands.rb:156:in `block (3 levels) in to_commander'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:155:in `run'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/runner.rb:402:in `run_active_command'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands.rb:65:in `run!'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/delegates.rb:11:in `run!'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/cli.rb:37:in `start'
	from /usr/local/share/gems/gems/rhc-0.97.5/bin/rhc:62:in `<top (required)>'
	from /usr/local/bin/rhc:23:in `load'
	from /usr/local/bin/rhc:23:in `<main>'

After step 3:
User Info
=========
Namespace: 1975t7

Login: jhou@redhat.com

Namespace 1975t7's Applications
===============================

Domain name is updated, and all apps are available.
The actual result and the returned messages are not consistent.

Expected results:
Domain update was successful, if timeout error occurs, the namespace should not be changed.

Additional info:
Create more applications and update domain with --timeout option, will make this problem more reproducible.
eg:Create more than 5 gears in a domain and then update namespace
   rhc domain update -n $newnamespace -d --trace --timeout 1
Comment 1 Rob Millner 2012-08-14 13:21:55 EDT
The fact that it times out while a successful operation is running is a bug; but I believe the right choice is to allow app operations to complete in the background if the client disconnects.

John, please follow up and change this ticket to deal with the timeout if you agree.
Comment 2 John (J5) Palmieri 2012-08-14 13:23:33 EDT
I upped the default timeout to 5 minutes but we are not in control of what gets rolled back.  If the client times out waiting for a response it would be up to the server to see the broken connection and revert.  I don't think this important though (a bit confusing perhaps).  We really should be moving to tight response times and any process taking longer should be async anyway.
Comment 3 joycezhang 2012-08-15 04:14:18 EDT
This bug will be verified with the build with fix is available.
Comment 4 Jianwei Hou 2012-08-16 03:51:28 EDT
Verifed on devenv_1997
CLI: rhc-0.97.7

Steps:
1.Create domain and create more than 5 apps in domain
2.Update namespace
  rhc domain update -n $new_name

Result:
Domain update was successfull. No timeout errors occur.


Updating domain '1997t1' to namespace '1997t2'

RESULT:
Success!
Comment 5 joycezhang 2012-08-20 03:43:03 EDT
This bug is reproduced again ,so reopen it. 

Please refer to the details as below:

Build:
rhc-0.97.12.gem
devenv_2029

1. Create domain and some apps (I have created 8 scalable apps on it)
2. Update name space
# rhc domain update joycedev5 -p redhat --trace

Updating domain 'joycedev4' to namespace 'joycedev5'

/Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc-rest.rb:82:in `request': Connection to server timed out or got interrupted: Request Timeout (Rhc::Rest::ConnectionException)
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc-rest/domain.rb:57:in `update'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands/domain.rb:40:in `update'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:164:in `send'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:164:in `to_commander'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:155:in `run'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/runner.rb:402:in `run_active_command'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:73:in `run!'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/delegates.rb:7:in `run!'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/cli.rb:37:in `start'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/bin/rhc:62
	from /usr/bin/rhc:23:in `load'
	from /usr/bin/rhc:23

The name space has been changed successfully even if this error displayed, all the apps can be accessed with the new name space.
Comment 6 John (J5) Palmieri 2012-08-20 11:07:26 EDT
Can you time how long it takes?  There is a 5 minute timeout on the client for this operation in which we might be butting up against the proxy timeouts.  There isn't really anything I can do about that in the client other than print out the the operation may have succeeded despite the timeout.
Comment 7 John (J5) Palmieri 2012-08-20 18:33:08 EDT
Updated timeout message to reflect that operation may have succeeded
Comment 8 Jianwei Hou 2012-08-21 05:28:34 EDT
I can not reproduce it every time, sometimes when timeout occur, CLI returns timeout but domain was updated successfully.

Steps:
Create over 11 apps and then run 'rhc domain alter'

hjw@my ~$ time rhc domain update -n jhou7
Password: ******


Updating domain 'jhou6' to namespace 'jhou7'

RESULT:
Connection to server timed out or got interrupted: Request Timeout


real	5m7.609s
user	0m0.595s
sys	0m0.278s

run 'rhc domain show' the domain name is updated.

User Info
=========
Namespace: jhou7

Login: jhou@redhat.com

Namespace jhou7's Applications
Comment 9 Jianwei Hou 2012-08-22 06:03:08 EDT
Verified on devenv_2046

Steps:
1.Create domain and create more than 10 apps
2.Alter namespace

time rhc domain update -n jhou16
Password: ******

Updating domain 'jhou12' to namespace 'jhou16'

RESULT:
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications.


real	5m6.794s
user	0m0.644s
sys	0m0.508s

3.Run 'rhc domain show'

User Info
=========
Namespace: jhou16

Login: jhou@redhat.com

Namespace jhou16's Applications
===============================
...................

Result:
Timeout occur, namespace is updated but the given message is now more friendly and will not lead to misunderstanding

Note You need to log in before you can comment on or make changes to this bug.