Bug 1038881 - Old applications can not work correctly after server upgrade on both INT and devenv_4012
Summary: Old applications can not work correctly after server upgrade on both INT and ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Pod
Version: 2.x
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Abhishek Gupta
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks: 1038376
TreeView+ depends on / blocked
 
Reported: 2013-12-06 03:04 UTC by Yujie Zhang
Modified: 2015-05-15 00:23 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-30 00:52:29 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Yujie Zhang 2013-12-06 03:04:13 UTC
Description of problem:

Checked the old existing applications on INT, visited the appilcation url and checked the application details page, the gear cost of app is "0 started / 0" and the status value is empty, the "Gears" info is "None", and "Source Code" info is like "ssh:///~/git/jbosseap.git/", "Remote Access" info is "ssh".

Check existing app using "rhc app show --gears" or other rhc commands will also meet errors, details are as following.

Version-Release number of selected component (if applicable):
INT(devenv_4101)

How reproducible:
always

Steps to Reproduce:
1.Login to INT 
2.Visit old existing application
3.Check the application details page
4.Do rhc app show --gears , rhc setup, rhc ssh

Actual results:

The status info is empty and "Gears" info is "None", gear cost is all 0, and "Source Code" info is like "ssh:///~/git/jbosseap.git/", "Remote Access" info is "ssh".Check existing app using "rhc app show  --gears" or other commands will also meet errors, details are as following:

$ rhc setup
Checking common problems .
An SSH connection could not be established to kitchensink-lxiamigrate.int.rhcloud.com. Your SSH configuration may not be
correct, or the application may not be responding. bad URI(absolute but no path): ssh:// (URI::InvalidURIError)

$ rhc ssh jbossas70s
Connecting to ssh:// ...
ssh: Could not resolve hostname ssh://: Name or service not known

[root@Daphne ~]# rhc app show n10map --state
Cartridge nodejs-0.10, mongodb-2.2 is 0/0 started

$ rhc app show jbossas70s --gears
/usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:363:in  `block (2 levels) in column_widths': undefined method `min' for  nil:NilClass (NoMethodError)
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:359:in `each'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:359:in `each_with_index'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:359:in `block in column_widths'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:358:in `each'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:358:in `column_widths'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:375:in `allocate_widths_for'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:433:in `widths'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:446:in `header_rows'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:453:in `rows'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/highline_extensions.rb:82:in `say'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/commands/app.rb:347:in `show'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/commands.rb:285:in `execute'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/commands.rb:276:in `block (3 levels) in to_commander'
    from /usr/local/lib/ruby/gems/2.0.0/gems/commander-4.1.4/lib/commander/command.rb:180:in `call'
    from /usr/local/lib/ruby/gems/2.0.0/gems/commander-4.1.4/lib/commander/command.rb:180:in `call'
    from /usr/local/lib/ruby/gems/2.0.0/gems/commander-4.1.4/lib/commander/command.rb:155:in `run'
    from /usr/local/lib/ruby/gems/2.0.0/gems/commander-4.1.4/lib/commander/runner.rb:400:in `run_active_command'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/command_runner.rb:73:in `run!'
    from /usr/local/lib/ruby/gems/2.0.0/gems/commander-4.1.4/lib/commander/delegates.rb:7:in `run!'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/lib/rhc/cli.rb:37:in `start'
    from /usr/local/lib/ruby/gems/2.0.0/gems/rhc-1.18.0/bin/rhc:20:in `<top (required)>'
    from /usr/local/bin/rhc:23:in `load'
    from /usr/local/bin/rhc:23:in `<main>'


Expected results:

Should display correct gears, status and ssh related info for all old applications.

Additional info:

Comment 1 Yujie Zhang 2013-12-06 03:20:23 UTC
Besides above issues, adding alias and cartridges also does not work, result is as following:

[root@dhcp-8-252 home]# rhc alias add www.nodejsalias.com -a  nodejs06787 -d
DEBUG: Using config file /root/.openshift/express.conf
DEBUG: Git config 'git config --get rhc.app-id' returned ''
DEBUG: Git config 'git config --get rhc.app-name' returned ''
DEBUG: Git config 'git config --get rhc.domain-name' returned ''
DEBUG: Authenticating with RHC::Auth::Basic
DEBUG: Connecting to https://int.openshift.redhat.com/broker/rest/api
DEBUG: Getting all domains
DEBUG: Client supports API versions 1.1, 1.2, 1.3, 1.4, 1.5
DEBUG: Created new httpclient
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/api
DEBUG:    code 200 1319 ms
DEBUG: Server supports API versions 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
DEBUG:    Using API version 1.5
DEBUG: Client API version 1.5 is not current. Refetching API
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/api
DEBUG:    code 200  321 ms
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/domains
DEBUG:    code 200 1159 ms
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/domains/stg999test/applications/nodejs06787
DEBUG:    code 200  869 ms
DEBUG: Running add_alias for nodejs06787
DEBUG: Request POST https://int.openshift.redhat.com/broker/rest/domain/stg999test/application/nodejs06787/aliases
DEBUG:    code 404 1201 ms
Alias 'www.nodejsalias.com' not found.


[root@dhcp-8-252 home]# rhc cartridge add mysql-5.1 -a  nodejs06787 -d
DEBUG: Using config file /root/.openshift/express.conf
DEBUG: Git config 'git config --get rhc.app-id' returned ''
DEBUG: Git config 'git config --get rhc.app-name' returned ''
DEBUG: Git config 'git config --get rhc.domain-name' returned ''
DEBUG: Authenticating with RHC::Auth::Basic
DEBUG: Connecting to https://int.openshift.redhat.com/broker/rest/api
DEBUG: Getting all cartridges
DEBUG: Client supports API versions 1.1, 1.2, 1.3, 1.4, 1.5
DEBUG: Created new httpclient
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/api
DEBUG:    code 200 2413 ms
DEBUG: Server supports API versions 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
DEBUG:    Using API version 1.5
DEBUG: Client API version 1.5 is not current. Refetching API
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/api
DEBUG:    code 200  357 ms
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/cartridges
DEBUG:    code 200  937 ms
Adding mysql-5.1 to application 'nodejs06787' ... DEBUG: Getting all domains
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/domains
DEBUG:    code 200 1145 ms
DEBUG: Request GET https://int.openshift.redhat.com/broker/rest/domains/stg999test/applications/nodejs06787?include=cartridges
DEBUG:    code 200 1142 ms
DEBUG: Adding cartridge nodejs06787
DEBUG: Request POST https://int.openshift.redhat.com/broker/rest/domain/stg999test/application/nodejs06787/cartridges
DEBUG:    code 422 28476 ms
Gear containing application dns not found

Comment 2 Meng Bo 2013-12-06 08:12:55 UTC
This issue can be reproduced on devenv_4102 as well.

After upgrade the packages to the latest and restart the services, the gear ssh info will be cleared when check the app info.

But app can be visited via web browser.

# rhc app show jbeap1
Password: 

jbeap1 @ http://jbeap1-bmeng1.dev.rhcloud.com/ (uuid: 52a17f18baf5abbe55000001)
-------------------------------------------------------------------------------
  Domain:     bmeng1
  Created:    3:39 PM
  Gears:      0 (defaults to small)
  Git URL:    ssh:///~/git/jbeap1.git/
  SSH:        ssh://
  Deployment: auto (on git push)

  jbosseap-6 (JBoss Enterprise Application Platform 6)
  ----------------------------------------------------
    Gears: Located with jenkins-client-1
  This gear costs an additional $0.03 per gear after the first 3 gears.

  jenkins-client-1 (Jenkins Client)
  ---------------------------------
    Gears:   Located with jbosseap-6
    Job URL: https://jenkins-bmeng1.dev.rhcloud.com/job/jbeap1-build/


And when trying to upgrade the node, the gears cannot be found from the node.

# oo-admin-upgrade upgrade-node --version 2.0.37 --ignore-cartridge-version 
Upgrader started with options: {:version=>"2.0.37", :ignore_cartridge_version=>true, :target_server_identity=>nil, :upgrade_position=>1, :num_upgraders=>1, :max_threads=>12, :gear_whitelist=>[], :num_tries=>2}
Building new upgrade queues and cluster metadata
Getting all active gears...
Getting all logins...
Writing node queue to /tmp/oo-upgrade/node_queue
Writing cluster metadata to /tmp/oo-upgrade/cluster_metadata
Loading cluster metadata from /tmp/oo-upgrade/cluster_metadata
Loading node queue from /tmp/oo-upgrade/node_queue
Node queue is empty; there's nothing for the upgrader to do. Exiting.

Comment 3 Jianwei Hou 2013-12-06 09:42:13 UTC
On the devenv_4102 environment testing, the gears document in applications collection of rhc-datastore was empty, which caused the problem. Not sure why this happened since this problem does not happen in previous testings.

<------------------------->
	"gears" : [ ],
	"group_instances" : [
		{
			"_id" : ObjectId("52a16034ab7615edd3000009")
		}
	],
<------------------------->

Comment 4 Jianwei Hou 2013-12-09 09:04:11 UTC
This problem is still on INT.
Wasn't able to reproduce this problem after upgrading devenv-stage_601 to devenv-stage_604 and migrate.

Comment 5 Ravi Sankar 2013-12-09 18:54:13 UTC
INT is broken, data related to 'gears' field is lost during migration (migration script was not re-entrant) and we did not had a good backup to restore INT. So ignore INT and test it on STG/DEV/TEST environment.

Comment 6 Abhishek Gupta 2013-12-09 18:56:16 UTC
There was a bug in the migration script that was fixed with --> 
https://github.com/openshift/li/pull/2216

The fix should have made it into devenv_4108. Please try again on the latest devenv or on STG once it is built/ready today.

As far as INT is concerned, it is not in a state to allow for testing the migration. Refer email from Dan and me on Friday on this topic as well as the comment above from Ravi.

Comment 7 Yujie Zhang 2013-12-10 07:56:41 UTC
Tested on STG(devenv-stage_604), old applications can work correctly after the migration of STG, so verify this bug, thanks.


Note You need to log in before you can comment on or make changes to this bug.