Bug 1005007 - (INT)Meet 'undefined method `categories' for nil:NilClass.' while connect to server or operate existing apps with old accounts having phpmyadmin cartridge after upgrading and migration
(INT)Meet 'undefined method `categories' for nil:NilClass.' while connect to ...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Kubernetes (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-05 22:42 EDT by Qiushui Zhang
Modified: 2015-05-14 20:20 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-19 12:49:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
upgrade log (8.99 KB, text/plain)
2013-09-06 00:51 EDT, Xiaoli Tian
no flags Details
broker log (5.18 KB, text/plain)
2013-09-06 00:59 EDT, Xiaoli Tian
no flags Details

  None (edit)
Description Qiushui Zhang 2013-09-05 22:42:07 EDT
Description of problem:
Int connection to accounts with old apps before upgrading fails, with error prompts "Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass.
Reference ID: 38f62689ff90e6923aaafbc0fcb36cde"
App is not accessable via browser
App is not accessable via command line, including app show, rhc ssh, etc.

Version-Release number of selected component (if applicable):
INT(devenv_3749)

How reproducible:
always

Steps to Reproduce:
1. rhc setup to int.openshift.redhat.com using accounts with old apps before upgrading
2. 
3.

Actual results:
It prompts "Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass.
Reference ID: 38f62689ff90e6923aaafbc0fcb36cde"

Expected results:
Connect successfully.

Additional info:
Accouts impacted:
qiuzhang+migrate@redhat.com
lxia+migrate@redhat.com
chunchen+migrate@redhat.com
wsun+migrate@redhat.com
wsun+1@redhat.com
wsun+2@redhat.com
wjiang@redhat.com
Comment 1 Xiaoli Tian 2013-09-06 00:31:58 EDT
Could be reproduced via upgrading from devenv-stage_461 to devenv_3749 as well:
After upgrading, running mongo migration scripts and oo-admin-upgrade, it will fail with the following error: 

stop operation failed against php1
Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass.
Comment 2 Xiaoli Tian 2013-09-06 00:39:06 EDT
Datastore migrate log:
Starting migration: non_compatible
Updating cartridge phpmyadmin-3 to phpmyadmin-4
Updating component instances for all applications...    Done.
Updating group overrides for all applications...        Done.
Updating cartridge version for all applications...      Done.
Updating cartridge version for premium cartridges in usage...   Done.
Verifying cartridge update for all applications...      Successful.


Migration complete
Done!
Comment 3 Xiaoli Tian 2013-09-06 00:51:40 EDT
Created attachment 794575 [details]
upgrade log
Comment 4 Xiaoli Tian 2013-09-06 00:59:21 EDT
Created attachment 794576 [details]
broker log

Find the following error in broker log ( more log is attached) 

DEBUG: Output of parallel execute: [{:tag=>{"op_id"=>"52295fcf473119cf85000006"}, :gear=>"927512607618356641529856", :job=>{:cartridge=>"openshift-or
igin-node", :action=>"stop", :args=>{"--with-app-uuid"=>"52294a9b473119758c000241", "--with-app-name"=>"jenkins1", "--with-container-uuid"=>"927512607618356641529856", "--with-container-name"=>"jenkins1", "--wit
h-namespace"=>"migration2", "--with-request-id"=>"f0a9acfe96869df2e4b02ebfd0005307", "--cart-name"=>"jenkins-1", "--component-name"=>"jenkins-1", "--with-software-version"=>"1", "--cartridge-vendor"=>"redhat"}},
 :result_stdout=>"Stopping jenkins cartridge\nWaiting for stop to finish\n", :result_stderr=>"", :result_exit_code=>0}], exitcode: 0, from: ip-10-152-163-89  (Request ID: f0a9acfe96869df2e4b02ebfd0005307) (pid:9
068)
ESC[0;37m2013-09-06 00:53:36.711ESC[0m [ESC[0;37mDEBUGESC[0m] DEBUG: MCollective Response Time (execute_parallel): 1.341103574s  (Request ID: f0a9acfe96869df2e4b02ebfd0005307) (pid:9068)
ESC[0;37m2013-09-06 00:53:36.796ESC[0m [ESC[31mERRORESC[0m] DEPRECATED: Access to Application#uuid has been removed
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.14.2/lib/openshift/controller/api_responses.rb:233:in `get_log_args
Comment 5 Meng Bo 2013-09-06 04:04:33 EDT
On devenv-stage ami, after remove the gears with phpmyadmin added, the account can be used.

It should be caused by the phpmyadmin cartridge.
Comment 6 Abhishek Gupta 2013-09-06 14:15:39 EDT
Can you make sure that this is not a cache problem? Please run the command below after the migration is complete.

oo-admin-broker-cache -c
Comment 7 Qiushui Zhang 2013-09-09 06:31:18 EDT
After "oo-admin-broker-cache -c", it works now. The apps are accessable and no error message like that on INT.
Comment 8 Abhishek Gupta 2013-09-09 16:45:15 EDT
Ravi: Can you please add exception handling to this scenario and provide a more user friendly error message.
Comment 9 Jianwei Hou 2013-09-10 03:14:18 EDT
Tried to figure out whether this is a broker cache issue, here is the find out:

Here is my steps:
1. Create apps on devenv-stage_465
2. Upgrade it to latest, and the version is same as devenv_3762
3. Clear broker cache after yum update
4. Migrate datastore with rhc-admin-migrate-datastore, this will migrate phpmyadmin-3 to phpmyadmin-4
rhc-admin-migrate-datastore --non-compatible --version 2.0.32 (**the version of  datastore migration is hard coded to 2.0.32, is ops using the same version?**)
5. Migrate applications with oo-admin-upgrade, all the gears are migrated successfully
6. Clear broker cache again.
7. Try to connect from client, but I couldn't reproduce this issue.

However, this problem is still seen on INT, even after broker cache is cleared again after migration. 
Here is what is returned during rhc setup.
"Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass.
Reference ID: e36d5b82c2ad809efd9fee63ffe143fb"

If I'm having the same steps as ops, and couldn't reproduce this bug, I think this bug is possibly not caused by the broker cache issue. I'm curious about my step 4, is the migration steps causing problems?
Comment 10 Abhishek Gupta 2013-09-10 19:25:18 EDT
Can you please try it again on INT?
Comment 11 Abhishek Gupta 2013-09-10 19:27:01 EDT
Also, I have fixed (pulled request yet to merge as of now) the error message that gets displayed to provide a little more information.

https://github.com/openshift/origin-server/pull/3613
Comment 13 Yan Du 2013-09-10 21:55:11 EDT
The issue still can be reproduced on INT (devenv_3769) with old account.

[root@Daphne home]# rhc setup --server int.openshift.redhat.com -l yadu+m@redhat.com -p XXXX
<------------snip------------------>
Saving configuration to /root/.openshift/express.conf ... done

Checking for git ... found git version 1.7.1

Checking common problems .

Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass.
Reference ID: 069f968ea4da9508d61f1582d6e2d957

Checking for a domain ... dy

Checking for applications ... 
Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass.
Reference ID: bcaf18435cd48ea8218648ea14d9b74a
Comment 14 Xiaoli Tian 2013-09-10 22:12:12 EDT
I guess the real bug for the INT issue is that:

The version specified in rhc-admin-migrate-datastore is still using 2.0.32.

But guess ops migration scripts is using 2.0.33, that cause the datastore migration is actually not run
Comment 15 Wesley Hearn 2013-09-11 09:48:13 EDT
That is correct. All migration scripts that need a version number are passed the current release number.
Comment 16 Abhishek Gupta 2013-09-11 13:32:03 EDT
Xiaoli: Excellent catch!!! 

It has now been fixed --> https://github.com/openshift/li/pull/1893
Comment 17 Qiushui Zhang 2013-09-11 22:23:53 EDT
Tested with latest INT (devenv_3775).

The accounts for migration work now.

On the instance, the version number is changed to 2.0.33 in rhc-admin-migrate-datastore
So mark the bug as verified.

Note You need to log in before you can comment on or make changes to this bug.