Description of problem: Update domain is failed with time out error displayed, and no apps are shown if using rhc domain show. In addition,it's failed to update if running update namespace command again: # rhc domain show Password: * User Info ========= Namespace: accep5 Login: jinzhang+6 Namespace accep5's Applications =============================== RESULT: Domain accep6 not found Version-Release number of selected component (if applicable): devenv_2085 rhc-0.98.3.gem How reproducible: Not always (It's happened when domain contains more apps, please refer to my app info list) Steps to Reproduce: 1. Create some scalable and non-scalable apps 2. Update domain #rhc domain update accep6 3. Run "rhc domain show" if time out error displayed 4. Run update domain again same as step2 Actual results: 1. Update domain is timeout as below: # rhc domain update accep6 -d Password: * D, [2012-08-28T15:26:59.397752 #1907] DEBUG -- : Connecting to https://ec2-23-20-37-22.compute-1.amazonaws.com/broker/rest/api D, [2012-08-28T15:27:01.789260 #1907] DEBUG -- : Getting all domains Updating domain 'accep5' to namespace 'accep6' RESULT: Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. 2. No apps are displayed when using "rhc domain show": # rhc domain show Password: * User Info ========= Namespace: accep5 Login: jinzhang+6 Namespace accep5's Applications =============================== RESULT: Domain accep6 not found 3. The domain name exists if trying to update domain again rhc domain update accep6 -d Password: * D, [2012-08-28T15:33:26.857864 #1946] DEBUG -- : Connecting to https://ec2-23-20-37-22.compute-1.amazonaws.com/broker/rest/api D, [2012-08-28T15:33:29.081908 #1946] DEBUG -- : Getting all domains Updating domain 'accep6' to namespace 'accep6' RESULT: Namespace 'accep6' already in use. Please choose another. Expected results: 1.Update domain is successful. 2.All apps are displayed even if namespace haven't been updated competely. 3.It should be successful if trying to run update domain again as namespace hasn't been updated successfully in step2. Additional info:
Here are my apps list for the reference: User Info ========= Namespace: accep5 Login: jinzhang+6 Namespace accep5's Applications =============================== myscapp1 Framework: jbossas-7 Creation: 2012-08-28T01:18:29-04:00 UUID: 1b0c09429af2440f93a6df20687c7ced Git URL: ssh://1b0c09429af2440f93a6df20687c7ced.rhcloud.com/~/git/myscapp1.git/ Public URL: http://myscapp1-accep5.dev.rhcloud.com/ Cartridges: haproxy-1.4 mongodb-2.0 - mongodb://e97b99a172-accep5.dev.rhcloud.com:35581/ myscapp2 Framework: perl-5.10 Creation: 2012-08-28T01:22:10-04:00 UUID: a272700d1eb646ef90298140d16111a2 Git URL: ssh://a272700d1eb646ef90298140d16111a2.rhcloud.com/~/git/myscapp2.git/ Public URL: http://myscapp2-accep5.dev.rhcloud.com/ Cartridges: haproxy-1.4 mongodb-2.0 - mongodb://96e1746617-accep5.dev.rhcloud.com:35671/ myscapp3 Framework: php-5.3 Creation: 2012-08-28T01:26:39-04:00 UUID: c890b5592987445dabed5d0a382bea9b Git URL: ssh://c890b5592987445dabed5d0a382bea9b.rhcloud.com/~/git/myscapp3.git/ Public URL: http://myscapp3-accep5.dev.rhcloud.com/ Cartridges: postgresql-8.4 - postgresql://591716586b-accep5.dev.rhcloud.com:35596/ haproxy-1.4 myscapp5 Framework: nodejs-0.6 Creation: 2012-08-28T01:35:55-04:00 UUID: 16096e85a6954bd995adb90ec515e98a Git URL: ssh://16096e85a6954bd995adb90ec515e98a.rhcloud.com/~/git/myscapp5.git/ Public URL: http://myscapp5-accep5.dev.rhcloud.com/ Cartridges: postgresql-8.4 - postgresql://8628d935cb-accep5.dev.rhcloud.com:35641/ haproxy-1.4 mongodb-2.0 - mongodb://72c7211e7b-accep5.dev.rhcloud.com:35561/ myscapp6 Framework: ruby-1.8 Creation: 2012-08-28T01:39:16-04:00 UUID: 33d4044ccfa44ecf90151dade7393bb7 Git URL: ssh://33d4044ccfa44ecf90151dade7393bb7.rhcloud.com/~/git/myscapp6.git/ Public URL: http://myscapp6-accep5.dev.rhcloud.com/ Cartridges: postgresql-8.4 - postgresql://995fe09210-accep5.dev.rhcloud.com:35656/ haproxy-1.4 mongodb-2.0 - mongodb://d6e53c630a-accep5.dev.rhcloud.com:35681/ myscapp8 Framework: ruby-1.9 Creation: 2012-08-28T02:05:18-04:00 UUID: c42963d4506c45fcb8891ccbc7f27807 Git URL: ssh://c42963d4506c45fcb8891ccbc7f27807.rhcloud.com/~/git/myscapp8.git/ Public URL: http://myscapp8-accep5.dev.rhcloud.com/ Cartridges: postgresql-8.4 - postgresql://2219e14d1d-accep5.dev.rhcloud.com:35661/ jenkins-client-1.4 - https://jenk-accep5.dev.rhcloud.com/job/myscapp8-build/ haproxy-1.4 mongodb-2.0 - mongodb://2961cec604-accep5.dev.rhcloud.com:35626/ myscapp9 Framework: jbosseap-6.0 Creation: 2012-08-28T02:09:19-04:00 UUID: 045c12388e1a47939d5118c01af3e6d7 Git URL: ssh://045c12388e1a47939d5118c01af3e6d7.rhcloud.com/~/git/myscapp9.git/ Public URL: http://myscapp9-accep5.dev.rhcloud.com/ Cartridges: postgresql-8.4 - postgresql://525ea25869-accep5.dev.rhcloud.com:35696/ haproxy-1.4 jenk Framework: jenkins-1.4 Creation: 2012-08-28T02:41:05-04:00 UUID: ca8f2fa502584be49b90c85ed304a88a Git URL: ssh://ca8f2fa502584be49b90c85ed304a88a.rhcloud.com/~/git/jenk.git/ Public URL: http://jenk-accep5.dev.rhcloud.com/ Cartridges: None
Update domain will be successful if updating domain with a new name instead of the name which was failed in step2. So set this bug as low serverity.
This bug is still reproduced on devenv_2108. All app info are lost after updating domain time out. # rhc domain update joyceaccp3 Password: * Updating domain 'accep2' to namespace 'joyceaccp3' RESULT: Connection to server timed out. It is possible the operation finished without being able to report success. Use 'rhc domain show' or 'rhc app status' to check the status of your applications. # rhc domain show --trace Password: * User Info ========= Namespace: accep2 Login: jinzhang+5 Namespace accep2's Applications =============================== /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/rest.rb:186:in `block in process_error_response': Domain 'accep2' not found (RHC::Rest::ResourceNotFoundException) from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/rest.rb:184:in `each' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/rest.rb:184:in `process_error_response' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/rest.rb:160:in `rescue in request' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/rest.rb:147:in `request' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/rest/domain.rb:39:in `applications' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/commands/domain.rb:71:in `block (2 levels) in show' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/helpers.rb:168:in `call' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/helpers.rb:168:in `section' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/helpers.rb:185:in `paragraph' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/commands/domain.rb:67:in `block in show' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/commands/domain.rb:66:in `each' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/commands/domain.rb:66:in `show' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/commands.rb:160:in `block (3 levels) in to_commander' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:180:in `call' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/command.rb:155:in `run' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/runner.rb:402:in `run_active_command' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/commands.rb:73:in `run!' from /usr/local/share/gems/gems/commander-4.1.2/lib/commander/delegates.rb:11:in `run!' from /usr/local/share/gems/gems/rhc-0.98.6/lib/rhc/cli.rb:37:in `start' from /usr/local/share/gems/gems/rhc-0.98.6/bin/rhc:62:in `<top (required)>' from /usr/local/bin/rhc:23:in `load' from /usr/local/bin/rhc:23:in `<main>'
The rest api needs to be able to handle disconnects during updates. We can not judge how long this is going to take and a timeout will happen in the most extreme cases. Once an update is issued it should complete on the server even if we disconnect.
https://github.com/openshift/li/pull/484 Fixed and merged.
li pull request is merged but the crankcase pull request https://github.com/openshift/origin-server/pull/676 is yet to be merged. Will put ON_QA status once it is merged.
This bug has been verified and fixed on devenv_2340. All apps are shown with old domain name if domain update failed. Mark it as Verified. Thanks.