Description of problem: update domain was successful, but CLI returns timeout error Version-Release number of selected component (if applicable): on devenv_1975 cli: rhc-0.97.5 How reproducible: sometimes Steps to Reproduce: 1.Create a domain and create applications in domain 2.Update domain's namespace rhc domain update -n 1975t7 -d --trace 3.Run rhc domain show to get domain information, and check all apps are avaiable after updating namespace Actual results: After step 2: Password: ** D, [2012-08-13T05:10:29.816554 #7099] DEBUG -- : Connecting to https://ec2-23-22-28-192.compute-1.amazonaws.com/broker/rest/api D, [2012-08-13T05:10:33.299544 #7099] DEBUG -- : Getting all domains Updating domain '1975t6' to namespace '1975t7' RESULT: /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest.rb:75:in `rescue in request': Connection to server timed out or got interrupted: Request Timeout (Rhc::Rest::ConnectionException) from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest.rb:66:in `request' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc-rest/domain.rb:56:in `update' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands/domain.rb:42:in `block in update' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:182:in `call' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:182:in `section' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/helpers.rb:199:in `paragraph' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands/domain.rb:40:in `update' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands.rb:156:in `block (3 levels) in to_commander' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:155:in `run' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/runner.rb:402:in `run_active_command' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/commands.rb:65:in `run!' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/delegates.rb:11:in `run!' from /usr/local/share/gems/gems/rhc-0.97.5/lib/rhc/cli.rb:37:in `start' from /usr/local/share/gems/gems/rhc-0.97.5/bin/rhc:62:in `<top (required)>' from /usr/local/bin/rhc:23:in `load' from /usr/local/bin/rhc:23:in `<main>' After step 3: User Info ========= Namespace: 1975t7 Login: jhou Namespace 1975t7's Applications =============================== Domain name is updated, and all apps are available. The actual result and the returned messages are not consistent. Expected results: Domain update was successful, if timeout error occurs, the namespace should not be changed. Additional info: Create more applications and update domain with --timeout option, will make this problem more reproducible. eg:Create more than 5 gears in a domain and then update namespace rhc domain update -n $newnamespace -d --trace --timeout 1
The fact that it times out while a successful operation is running is a bug; but I believe the right choice is to allow app operations to complete in the background if the client disconnects. John, please follow up and change this ticket to deal with the timeout if you agree.
I upped the default timeout to 5 minutes but we are not in control of what gets rolled back. If the client times out waiting for a response it would be up to the server to see the broken connection and revert. I don't think this important though (a bit confusing perhaps). We really should be moving to tight response times and any process taking longer should be async anyway.
This bug will be verified with the build with fix is available.
Verifed on devenv_1997 CLI: rhc-0.97.7 Steps: 1.Create domain and create more than 5 apps in domain 2.Update namespace rhc domain update -n $new_name Result: Domain update was successfull. No timeout errors occur. Updating domain '1997t1' to namespace '1997t2' RESULT: Success!
This bug is reproduced again ,so reopen it. Please refer to the details as below: Build: rhc-0.97.12.gem devenv_2029 1. Create domain and some apps (I have created 8 scalable apps on it) 2. Update name space # rhc domain update joycedev5 -p redhat --trace Updating domain 'joycedev4' to namespace 'joycedev5' /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc-rest.rb:82:in `request': Connection to server timed out or got interrupted: Request Timeout (Rhc::Rest::ConnectionException) from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc-rest/domain.rb:57:in `update' from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands/domain.rb:40:in `update' from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:164:in `send' from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:164:in `to_commander' from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:180:in `call' from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:180:in `call' from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/command.rb:155:in `run' from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/runner.rb:402:in `run_active_command' from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/commands.rb:73:in `run!' from /Library/Ruby/Gems/1.8/gems/commander-4.1.2/lib/commander/delegates.rb:7:in `run!' from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/lib/rhc/cli.rb:37:in `start' from /Library/Ruby/Gems/1.8/gems/rhc-0.97.12/bin/rhc:62 from /usr/bin/rhc:23:in `load' from /usr/bin/rhc:23 The name space has been changed successfully even if this error displayed, all the apps can be accessed with the new name space.
Can you time how long it takes? There is a 5 minute timeout on the client for this operation in which we might be butting up against the proxy timeouts. There isn't really anything I can do about that in the client other than print out the the operation may have succeeded despite the timeout.
Updated timeout message to reflect that operation may have succeeded
I can not reproduce it every time, sometimes when timeout occur, CLI returns timeout but domain was updated successfully. Steps: Create over 11 apps and then run 'rhc domain alter' hjw@my ~$ time rhc domain update -n jhou7 Password: ****** Updating domain 'jhou6' to namespace 'jhou7' RESULT: Connection to server timed out or got interrupted: Request Timeout real 5m7.609s user 0m0.595s sys 0m0.278s run 'rhc domain show' the domain name is updated. User Info ========= Namespace: jhou7 Login: jhou Namespace jhou7's Applications
Verified on devenv_2046 Steps: 1.Create domain and create more than 10 apps 2.Alter namespace time rhc domain update -n jhou16 Password: ****** Updating domain 'jhou12' to namespace 'jhou16' RESULT: Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. real 5m6.794s user 0m0.644s sys 0m0.508s 3.Run 'rhc domain show' User Info ========= Namespace: jhou16 Login: jhou Namespace jhou16's Applications =============================== ................... Result: Timeout occur, namespace is updated but the given message is now more friendly and will not lead to misunderstanding