Bug 847685 - Sometimes update domain was successful but CLI returns timeout error
Summary: Sometimes update domain was successful but CLI returns timeout error
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OKD
Classification: Red Hat
Component: oc
Version: 2.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: John (J5) Palmieri
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-08-13 09:36 UTC by Jianwei Hou
Modified: 2015-05-15 02:02 UTC (History)
3 users (show)

Fixed In Version: rhc-0.97.13-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-09-17 21:29:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jianwei Hou 2012-08-13 09:36:51 UTC
Description of problem:
update domain was successful, but CLI returns timeout error 

Version-Release number of selected component (if applicable):
on devenv_1975
cli: rhc-0.97.5

How reproducible:
sometimes

Steps to Reproduce:
1.Create a domain and create applications in domain
2.Update domain's namespace
  rhc domain update -n 1975t7 -d --trace
3.Run rhc domain show to get domain information, and check all apps are avaiable after updating namespace
  
Actual results:
After step 2:
Password: **

D, [2012-08-13T05:10:29.816554 #7099] DEBUG -- : Connecting to https://ec2-23-22-28-192.compute-1.amazonaws.com/broker/rest/api
D, [2012-08-13T05:10:33.299544 #7099] DEBUG -- : Getting all domains

Updating domain '1975t6' to namespace '1975t7'

RESULT:
/usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest.rb:75:in `rescue in request': Connection to server timed out or got interrupted: Request Timeout (Rhc::Rest::ConnectionException)
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest.rb:66:in `request'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest/domain.rb:56:in `update'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands/domain.rb:42:in `block in update'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:182:in `call'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:182:in `section'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:199:in `paragraph'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands/domain.rb:40:in `update'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands.rb:156:in `block (3 levels) in to_commander'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:155:in `run'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/runner.rb:402:in `run_active_command'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands.rb:65:in `run!'
	from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/delegates.rb:11:in `run!'
	from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/cli.rb:37:in `start'
	from /usr/local/share/gems/gems/rhc-0.97.5/bin/rhc:62:in `<top (required)>'
	from /usr/local/bin/rhc:23:in `load'
	from /usr/local/bin/rhc:23:in `<main>'

After step 3:
User Info
=========
Namespace: 1975t7

Login: jhou

Namespace 1975t7's Applications
===============================

Domain name is updated, and all apps are available.
The actual result and the returned messages are not consistent.

Expected results:
Domain update was successful, if timeout error occurs, the namespace should not be changed.

Additional info:
Create more applications and update domain with --timeout option, will make this problem more reproducible.
eg:Create more than 5 gears in a domain and then update namespace
   rhc domain update -n $newnamespace -d --trace --timeout 1

Comment 1 Rob Millner 2012-08-14 17:21:55 UTC
The fact that it times out while a successful operation is running is a bug; but I believe the right choice is to allow app operations to complete in the background if the client disconnects.

John, please follow up and change this ticket to deal with the timeout if you agree.

Comment 2 John (J5) Palmieri 2012-08-14 17:23:33 UTC
I upped the default timeout to 5 minutes but we are not in control of what gets rolled back.  If the client times out waiting for a response it would be up to the server to see the broken connection and revert.  I don't think this important though (a bit confusing perhaps).  We really should be moving to tight response times and any process taking longer should be async anyway.

Comment 3 joycezhang 2012-08-15 08:14:18 UTC
This bug will be verified with the build with fix is available.

Comment 4 Jianwei Hou 2012-08-16 07:51:28 UTC
Verifed on devenv_1997
CLI: rhc-0.97.7

Steps:
1.Create domain and create more than 5 apps in domain
2.Update namespace
  rhc domain update -n $new_name

Result:
Domain update was successfull. No timeout errors occur.


Updating domain '1997t1' to namespace '1997t2'

RESULT:
Success!

Comment 5 joycezhang 2012-08-20 07:43:03 UTC
This bug is reproduced again ,so reopen it. 

Please refer to the details as below:

Build:
rhc-0.97.12.gem
devenv_2029

1. Create domain and some apps (I have created 8 scalable apps on it)
2. Update name space
# rhc domain update joycedev5 -p redhat --trace

Updating domain 'joycedev4' to namespace 'joycedev5'

/Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc-rest.rb:82:in `request': Connection to server timed out or got interrupted: Request Timeout (Rhc::Rest::ConnectionException)
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc-rest/domain.rb:57:in `update'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands/domain.rb:40:in `update'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:164:in `send'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:164:in `to_commander'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:180:in `call'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:155:in `run'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/runner.rb:402:in `run_active_command'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:73:in `run!'
	from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/delegates.rb:7:in `run!'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/cli.rb:37:in `start'
	from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/bin/rhc:62
	from /usr/bin/rhc:23:in `load'
	from /usr/bin/rhc:23

The name space has been changed successfully even if this error displayed, all the apps can be accessed with the new name space.

Comment 6 John (J5) Palmieri 2012-08-20 15:07:26 UTC
Can you time how long it takes?  There is a 5 minute timeout on the client for this operation in which we might be butting up against the proxy timeouts.  There isn't really anything I can do about that in the client other than print out the the operation may have succeeded despite the timeout.

Comment 7 John (J5) Palmieri 2012-08-20 22:33:08 UTC
Updated timeout message to reflect that operation may have succeeded

Comment 8 Jianwei Hou 2012-08-21 09:28:34 UTC
I can not reproduce it every time, sometimes when timeout occur, CLI returns timeout but domain was updated successfully.

Steps:
Create over 11 apps and then run 'rhc domain alter'

hjw@my ~$ time rhc domain update -n jhou7
Password: ******


Updating domain 'jhou6' to namespace 'jhou7'

RESULT:
Connection to server timed out or got interrupted: Request Timeout


real	5m7.609s
user	0m0.595s
sys	0m0.278s

run 'rhc domain show' the domain name is updated.

User Info
=========
Namespace: jhou7

Login: jhou

Namespace jhou7's Applications

Comment 9 Jianwei Hou 2012-08-22 10:03:08 UTC
Verified on devenv_2046

Steps:
1.Create domain and create more than 10 apps
2.Alter namespace

time rhc domain update -n jhou16
Password: ******

Updating domain 'jhou12' to namespace 'jhou16'

RESULT:
Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications.


real	5m6.794s
user	0m0.644s
sys	0m0.508s

3.Run 'rhc domain show'

User Info
=========
Namespace: jhou16

Login: jhou

Namespace jhou16's Applications
===============================
...................

Result:
Timeout occur, namespace is updated but the given message is now more friendly and will not lead to misunderstanding


Note You need to log in before you can comment on or make changes to this bug.