Description of problem: Int connection to accounts with old apps before upgrading fails, with error prompts "Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass. Reference ID: 38f62689ff90e6923aaafbc0fcb36cde" App is not accessable via browser App is not accessable via command line, including app show, rhc ssh, etc. Version-Release number of selected component (if applicable): INT(devenv_3749) How reproducible: always Steps to Reproduce: 1. rhc setup to int.openshift.redhat.com using accounts with old apps before upgrading 2. 3. Actual results: It prompts "Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass. Reference ID: 38f62689ff90e6923aaafbc0fcb36cde" Expected results: Connect successfully. Additional info: Accouts impacted: qiuzhang+migrate lxia+migrate chunchen+migrate wsun+migrate wsun+1 wsun+2 wjiang
Could be reproduced via upgrading from devenv-stage_461 to devenv_3749 as well: After upgrading, running mongo migration scripts and oo-admin-upgrade, it will fail with the following error: stop operation failed against php1 Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass.
Datastore migrate log: Starting migration: non_compatible Updating cartridge phpmyadmin-3 to phpmyadmin-4 Updating component instances for all applications... Done. Updating group overrides for all applications... Done. Updating cartridge version for all applications... Done. Updating cartridge version for premium cartridges in usage... Done. Verifying cartridge update for all applications... Successful. Migration complete Done!
Created attachment 794575 [details] upgrade log
Created attachment 794576 [details] broker log Find the following error in broker log ( more log is attached) DEBUG: Output of parallel execute: [{:tag=>{"op_id"=>"52295fcf473119cf85000006"}, :gear=>"927512607618356641529856", :job=>{:cartridge=>"openshift-or igin-node", :action=>"stop", :args=>{"--with-app-uuid"=>"52294a9b473119758c000241", "--with-app-name"=>"jenkins1", "--with-container-uuid"=>"927512607618356641529856", "--with-container-name"=>"jenkins1", "--wit h-namespace"=>"migration2", "--with-request-id"=>"f0a9acfe96869df2e4b02ebfd0005307", "--cart-name"=>"jenkins-1", "--component-name"=>"jenkins-1", "--with-software-version"=>"1", "--cartridge-vendor"=>"redhat"}}, :result_stdout=>"Stopping jenkins cartridge\nWaiting for stop to finish\n", :result_stderr=>"", :result_exit_code=>0}], exitcode: 0, from: ip-10-152-163-89 (Request ID: f0a9acfe96869df2e4b02ebfd0005307) (pid:9 068) ESC[0;37m2013-09-06 00:53:36.711ESC[0m [ESC[0;37mDEBUGESC[0m] DEBUG: MCollective Response Time (execute_parallel): 1.341103574s (Request ID: f0a9acfe96869df2e4b02ebfd0005307) (pid:9068) ESC[0;37m2013-09-06 00:53:36.796ESC[0m [ESC[31mERRORESC[0m] DEPRECATED: Access to Application#uuid has been removed /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.14.2/lib/openshift/controller/api_responses.rb:233:in `get_log_args
On devenv-stage ami, after remove the gears with phpmyadmin added, the account can be used. It should be caused by the phpmyadmin cartridge.
Can you make sure that this is not a cache problem? Please run the command below after the migration is complete. oo-admin-broker-cache -c
After "oo-admin-broker-cache -c", it works now. The apps are accessable and no error message like that on INT.
Ravi: Can you please add exception handling to this scenario and provide a more user friendly error message.
Tried to figure out whether this is a broker cache issue, here is the find out: Here is my steps: 1. Create apps on devenv-stage_465 2. Upgrade it to latest, and the version is same as devenv_3762 3. Clear broker cache after yum update 4. Migrate datastore with rhc-admin-migrate-datastore, this will migrate phpmyadmin-3 to phpmyadmin-4 rhc-admin-migrate-datastore --non-compatible --version 2.0.32 (**the version of datastore migration is hard coded to 2.0.32, is ops using the same version?**) 5. Migrate applications with oo-admin-upgrade, all the gears are migrated successfully 6. Clear broker cache again. 7. Try to connect from client, but I couldn't reproduce this issue. However, this problem is still seen on INT, even after broker cache is cleared again after migration. Here is what is returned during rhc setup. "Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass. Reference ID: e36d5b82c2ad809efd9fee63ffe143fb" If I'm having the same steps as ops, and couldn't reproduce this bug, I think this bug is possibly not caused by the broker cache issue. I'm curious about my step 4, is the migration steps causing problems?
Can you please try it again on INT?
Also, I have fixed (pulled request yet to merge as of now) the error message that gets displayed to provide a little more information. https://github.com/openshift/origin-server/pull/3613
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/28e5454ab7477c36360fe0e68d26ca7724ff503a Fix for bug 1005007 and bug 1006526
The issue still can be reproduced on INT (devenv_3769) with old account. [root@Daphne home]# rhc setup --server int.openshift.redhat.com -l yadu+m -p XXXX <------------snip------------------> Saving configuration to /root/.openshift/express.conf ... done Checking for git ... found git version 1.7.1 Checking common problems . Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass. Reference ID: 069f968ea4da9508d61f1582d6e2d957 Checking for a domain ... dy Checking for applications ... Unable to complete the requested operation due to: undefined method `categories' for nil:NilClass. Reference ID: bcaf18435cd48ea8218648ea14d9b74a
I guess the real bug for the INT issue is that: The version specified in rhc-admin-migrate-datastore is still using 2.0.32. But guess ops migration scripts is using 2.0.33, that cause the datastore migration is actually not run
That is correct. All migration scripts that need a version number are passed the current release number.
Xiaoli: Excellent catch!!! It has now been fixed --> https://github.com/openshift/li/pull/1893
Tested with latest INT (devenv_3775). The accounts for migration work now. On the instance, the version number is changed to 2.0.33 in rhc-admin-migrate-datastore So mark the bug as verified.