Created attachment 679460 [details] broker log Description of problem: After migration, the broker is unable to handle scalabe applications. Unable to create, show, delete scalable applications, rest api(gear_groups) for scalable applications returns exception. However, the scalable apps themselves are accessible. Version-Release number of selected component (if applicable): From stage_272 updated to fork_ami_refctr1_405 How reproducible: Always Steps to Reproduce: 1. Launch devenv-stage_272 instance and a fork_ami_refctr1_405 instance 2. Create scalable app against devenv-stage_272 instance 2. Copy pacakges under ~/devenv-local and local.repo to devenv-stage_272 instance 3. Modify devenv.repo, replace 'stage' with 'candidate' in order to update to latest sed -i 's/stage/candidate/g' /etc/yum.repos.d/devenv.repo 4. yum update 5. ./migrate-mongo-2.1 6. On client, run rhc domain show Actual results: hjw@hjw devenv$ rhc domain show -px The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall).Please verify that you can access the OpenShift server https://ec2-50-19-48-55.compute-1.amazonaws.com/broker/rest/domains/272tjhou/applications development.log <------------------------------------------------> 013-01-16 05:07:05.600 [FATAL] NoMethodError (undefined method `[]' for nil:NilClass): openshift-origin-controller (1.3.3) app/models/application.rb:290:in `block in group_instances_with_scale' mongoid (3.0.5) lib/mongoid/relations/proxy.rb:117:in `map' mongoid (3.0.5) lib/mongoid/relations/proxy.rb:117:in `method_missing' mongoid (3.0.5) lib/mongoid/relations/embedded/many.rb:384:in `method_missing' openshift-origin-controller (1.3.3) app/models/application.rb:288:in `group_instances_with_scale' openshift-origin-controller (1.3.3) app/helpers/rest_model_helper.rb:15:in `get_application_rest_cartridges' openshift-origin-controller (1.3.3) app/helpers/rest_model_helper.rb:9:in `get_rest_application' openshift-origin-controller (1.3.3) app/controllers/applications_controller.rb:16:in `block in index' mongoid (3.0.5) lib/mongoid/relations/targets/enumerable.rb:395:in `map!' mongoid (3.0.5) lib/mongoid/relations/targets/enumerable.rb:395:in `method_missing' mongoid (3.0.5) lib/mongoid/relations/referenced/many.rb:390:in `method_missing' openshift-origin-controller (1.3.3) app/controllers/applications_controller.rb:16:in `index' <------------------------------------------------> Expected results: Should not have any error. Additional info: I have two accounts, the other one is filled with non-scalable applications, which works fine with domain show. So this problem is probably caused by scaling applications. development.log is attached.
Could not reproduce this problem. I created two apps - one scalable and one non-scalable, and post-migration I could do rhc domain show and it worked alright. Can you pass the mongo dump of the original application (its user)?
Created attachment 682244 [details] mongodump of openshfit_broker_dev This is still reproduced when I upgrade from stage_272 ami which was then updated to fork_ami_refctr1_434, and then run the migrate-mongo-2.1 script. I created a bunch of applications, covering every cartridge, covering scaling and non-scaling. I have dumped openshift_broker_dev database and uploaded it, hope this can help.
Created attachment 682245 [details] development.log Also adding today's development.log
Fixed with rev#4e5d635d67533dd6d4bb74dd7b4f01d16fec0e4f
Created attachment 685681 [details] tcpdump from broker Seems the fix is with jenkins, but I'm still able to reproduce this problem without jenkins. I got two accounts, the account with non-scaling apps seems work well, the other one with scalable apps are broken. I'm not sure what's going on, so I run a 'rhc domain show' from client and did a tcpdump from broker instance. With wireshark, I found there are some "HTTP Basic: Access denied." error. If the problem is not caused by mongo, it's probably caused by a yum update, can you see into this again? Thanks!
I cant reproduce this issue yet.. with the payload you sent I was able to do a 'rhc domain show' alright. Can you send the broker's development.log on the rhc domain show command?
(In reply to comment #6) > I cant reproduce this issue yet.. with the payload you sent I was able to do > a 'rhc domain show' alright. Can you send the broker's development.log on > the rhc domain show command? This is strange, each time after I upgrade and migrate mongodb, I hit the same problem if I got scalable app in domain. The development.log is as the first attachment each time I run 'rhc domain show', which always complains 'block in group_instances_with_scale'. <-----------------------------------------> openshift-origin-controller (1.3.3) app/models/application.rb:290:in `block in group_instances_with_scale' mongoid (3.0.5) lib/mongoid/relations/proxy.rb:117:in `map' mongoid (3.0.5) lib/mongoid/relations/proxy.rb:117:in `method_missing' mongoid (3.0.5) lib/mongoid/relations/embedded/many.rb:384:in `method_missing' openshift-origin-controller (1.3.3) app/models/application.rb:288:in `group_instances_with_scale' openshift-origin-controller (1.3.3) app/helpers/rest_model_helper.rb:15:in `get_application_rest_cartridges' openshift-origin-controller (1.3.3) app/helpers/rest_model_helper.rb:9:in `get_rest_application' openshift-origin-controller (1.3.3) app/controllers/applications_controller.rb:16:in `block in index' mongoid (3.0.5) lib/mongoid/relations/targets/enumerable.rb:395:in `map!' mongoid (3.0.5) lib/mongoid/relations/targets/enumerable.rb:395:in `method_missing' mongoid (3.0.5) lib/mongoid/relations/referenced/many.rb:390:in `method_missing' openshift-origin-controller (1.3.3) app/controllers/applications_controller.rb:16:in `index' actionpack (3.2.8) lib/action_controller/metal/implicit_render.rb:4:in `send_action' actionpack (3.2.8) lib/abstract_controller/base.rb:167:in `process_action' actionpack (3.2.8) lib/action_controller/metal/rendering.rb:10:in `process_action' actionpack (3.2.8) lib/abstract_controller/callbacks.rb:18:in `block in process_action' activesupport (3.2.8) lib/active_support/callbacks.rb:458:in `_run__1923736641651663583__process_action__169953413690722355__callbacks' activesupport (3.2.8) lib/active_support/callbacks.rb:405:in `__run_callback' activesupport (3.2.8) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks' activesupport (3.2.8) lib/active_support/callbacks.rb:81:in `run_callbacks' actionpack (3.2.8) lib/abstract_controller/callbacks.rb:17:in `process_action' actionpack (3.2.8) lib/action_controller/metal/rescue.rb:29:in `process_action' actionpack (3.2.8) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action' activesupport (3.2.8) lib/active_support/notifications.rb:123:in `block in instrument' activesupport (3.2.8) lib/active_support/notifications/instrumenter.rb:20:in `instrument' activesupport (3.2.8) lib/active_support/notifications.rb:123:in `instrument' actionpack (3.2.8) lib/action_controller/metal/instrumentation.rb:29:in `process_action' actionpack (3.2.8) lib/action_controller/metal/params_wrapper.rb:207:in `process_action' actionpack (3.2.8) lib/abstract_controller/base.rb:121:in `process' actionpack (3.2.8) lib/abstract_controller/rendering.rb:45:in `process' actionpack (3.2.8) lib/action_controller/metal.rb:203:in `dispatch' actionpack (3.2.8) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' actionpack (3.2.8) lib/action_controller/metal.rb:246:in `block in action' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:73:in `call' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:73:in `dispatch' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:36:in `call' journey (1.0.4) lib/journey/router.rb:68:in `block in call' journey (1.0.4) lib/journey/router.rb:56:in `each' journey (1.0.4) lib/journey/router.rb:56:in `call' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:600:in `call' mongo_mapper (0.11.1) lib/mongo_mapper/middleware/identity_map.rb:10:in `call' ...................... <-------------------------------------------------------> I want to make sure I'm following the correct steps to complete the upgrade and migration, so can you please check if the steps are valid? Thanks 1. Launch devenv-stage_272 instance and a fork_ami_refctr1_434 instance 2. Create scalable app against devenv-stage_272 instance 2. Copy pacakges under ~/devenv-local and local.repo from to fork_ami_refctr1_434 instance devenv-stage_272 instance 3. On devenv-stage_272 instance,modify devenv.repo, replacing 'stage' with 'candidate' in order to update to latest sed -i 's/stage/candidate/g' /etc/yum.repos.d/devenv.repo 4. yum update --enablerepo devenv-local 5. After yum update, clear broker cache cd /var/www/openshift/broker rake tmp:clear 6. execute migrate-mongo-2.1 script 7. run rhc domain show from client, and the problem is reproduced
Issue resolved. The fork ami #434 did not have the required fixes for this to work. Also several commits (all leading upto rev# noted in comment 4) will resolve the issue. Please verify against latest fork ami only.
After upgrading to the latest fork_ami, this problem is fixed Now that the refctr branch is merged in to master, I will also test against upgrading to latest devenv. But the packages are built yet. So once devenv is available to test, I will test and move this bug to verified, thanks!
This is reproduced again. This time, upgrade stage_ami to latest devenv_ami since refctr1 branch is merged into master now. 1. Launch devenv-stage_281 instance and a devenv_2737 instance 2. Create scalable app against devenv-stage_281 instance 2. Copy pacakges under ~/devenv-local and local.repo from devenv_2737 instance to devenv-stage_281 instance 3. On devenv-stage_281 instance,modify devenv.repo, replacing 'stage' with 'candidate' in order to update to latest sed -i 's/stage/candidate/g' /etc/yum.repos.d/devenv.repo 4. yum update --enablerepo devenv-local 5. After yum update, clear broker cache cd /var/www/openshift/broker rake tmp:clear 6. execute migrate-mongo-2.0.23 script 7. run 'rhc domain show' from client, and the problem is reproduced devenv_2737 is already the latest instance I have today. But according to comment 8, maybe the packages are not in the devenv instance?
Can we look at re-doing this? The devenv repos are probably built and pushed out, so a yum update will less likely fail. If, after the yum update, the latest code still does not get to the broker, we need to review whats going wrong with the upgrade process. If the latest code does get installed, can you send the latest broker development log to ensure its a bug related to migration.
I did a two step check: 1. sed -i 's/stage/candidate/g' /etc/yum.repos.d/devenv.repo Then, yum update After update, the schema of openshift_broker_dev database is like: libra_rs:PRIMARY> show collections; distributed_lock system.indexes system.users usages user 2. After copying all packages from latest devenv instanace to stage_ami(same as steps in comment 10), and yum update again, The schema of openshift_broker_dev database is like: libra_rs:PRIMARY> show collections; applications cloud_users distributed_lock distributed_locks districts domains gears locks system.indexes system.users usages user So After step 2, I think I have upgraded my stage instance to the latest, and I still got the problem same as before. Error log in development.log is still same as before, I will also attached updated packages as well. <----------------------------------------> 2013-01-30 04:55:43.678 [FATAL] NoMethodError (undefined method `[]' for nil:NilClass): openshift-origin-controller (1.4.2) app/models/application.rb:278:in `block in group_instances_with_scale' mongoid (3.0.5) lib/mongoid/relations/proxy.rb:117:in `map' mongoid (3.0.5) lib/mongoid/relations/proxy.rb:117:in `method_missing' mongoid (3.0.5) lib/mongoid/relations/embedded/many.rb:384:in `method_missing' openshift-origin-controller (1.4.2) app/models/application.rb:276:in `group_instances_with_scale' openshift-origin-controller (1.4.2) app/helpers/rest_model_helper.rb:15:in `get_application_rest_cartridges' openshift-origin-controller (1.4.2) app/helpers/rest_model_helper.rb:9:in `get_rest_application' openshift-origin-controller (1.4.2) app/controllers/applications_controller.rb:16:in `block in index' mongoid (3.0.5) lib/mongoid/relations/targets/enumerable.rb:395:in `map!' mongoid (3.0.5) lib/mongoid/relations/targets/enumerable.rb:395:in `method_missing' mongoid (3.0.5) lib/mongoid/relations/referenced/many.rb:390:in `method_missing' openshift-origin-controller (1.4.2) app/controllers/applications_controller.rb:16:in `index' actionpack (3.2.8) lib/action_controller/metal/implicit_render.rb:4:in `send_action' actionpack (3.2.8) lib/abstract_controller/base.rb:167:in `process_action' actionpack (3.2.8) lib/action_controller/metal/rendering.rb:10:in `process_action' actionpack (3.2.8) lib/abstract_controller/callbacks.rb:18:in `block in process_action' activesupport (3.2.8) lib/active_support/callbacks.rb:458:in `_run__3569841260964423227__process_action__1857483635049100958__callbacks' activesupport (3.2.8) lib/active_support/callbacks.rb:405:in `__run_callback' activesupport (3.2.8) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks' activesupport (3.2.8) lib/active_support/callbacks.rb:81:in `run_callbacks' actionpack (3.2.8) lib/abstract_controller/callbacks.rb:17:in `process_action' actionpack (3.2.8) lib/action_controller/metal/rescue.rb:29:in `process_action' actionpack (3.2.8) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action' activesupport (3.2.8) lib/active_support/notifications.rb:123:in `block in instrument' activesupport (3.2.8) lib/active_support/notifications/instrumenter.rb:20:in `instrument' activesupport (3.2.8) lib/active_support/notifications.rb:123:in `instrument' actionpack (3.2.8) lib/action_controller/metal/instrumentation.rb:29:in `process_action' actionpack (3.2.8) lib/action_controller/metal/params_wrapper.rb:207:in `process_action' actionpack (3.2.8) lib/abstract_controller/base.rb:121:in `process' actionpack (3.2.8) lib/abstract_controller/rendering.rb:45:in `process' actionpack (3.2.8) lib/action_controller/metal.rb:203:in `dispatch' actionpack (3.2.8) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch' actionpack (3.2.8) lib/action_controller/metal.rb:246:in `block in action' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:73:in `call' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:73:in `dispatch' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:36:in `call' journey (1.0.4) lib/journey/router.rb:68:in `block in call' journey (1.0.4) lib/journey/router.rb:56:in `each' journey (1.0.4) lib/journey/router.rb:56:in `call' actionpack (3.2.8) lib/action_dispatch/routing/route_set.rb:600:in `call' mongo_mapper (0.11.1) lib/mongo_mapper/middleware/identity_map.rb:10:in `call' mongoid (3.0.5) lib/rack/mongoid/middleware/identity_map.rb:33:in `block in call' mongoid (3.0.5) lib/mongoid/unit_of_work.rb:39:in `unit_of_work' mongoid (3.0.5) lib/rack/mongoid/middleware/identity_map.rb:33:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call' rack (1.4.1) lib/rack/etag.rb:23:in `call' rack (1.4.1) lib/rack/conditionalget.rb:25:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/head.rb:14:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/params_parser.rb:21:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/flash.rb:242:in `call' rack (1.4.1) lib/rack/session/abstract/id.rb:205:in `context' rack (1.4.1) lib/rack/session/abstract/id.rb:200:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/cookies.rb:339:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' activesupport (3.2.8) lib/active_support/callbacks.rb:405:in `_run__2345207533808026872__call__2221799604084675635__callbacks' activesupport (3.2.8) lib/active_support/callbacks.rb:405:in `__run_callback' activesupport (3.2.8) lib/active_support/callbacks.rb:385:in `_run_call_callbacks' activesupport (3.2.8) lib/active_support/callbacks.rb:81:in `run_callbacks' actionpack (3.2.8) lib/action_dispatch/middleware/callbacks.rb:27:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/remote_ip.rb:31:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call' railties (3.2.8) lib/rails/rack/logger.rb:26:in `call_app' railties (3.2.8) lib/rails/rack/logger.rb:16:in `call' actionpack (3.2.8) lib/action_dispatch/middleware/request_id.rb:22:in `call' rack (1.4.1) lib/rack/methodoverride.rb:21:in `call' rack (1.4.1) lib/rack/runtime.rb:17:in `call' activesupport (3.2.8) lib/active_support/cache/strategy/local_cache.rb:72:in `call' rack (1.4.1) lib/rack/lock.rb:15:in `call' rack-cache (1.2) lib/rack/cache/context.rb:136:in `forward' rack-cache (1.2) lib/rack/cache/context.rb:245:in `fetch' rack-cache (1.2) lib/rack/cache/context.rb:185:in `lookup' rack-cache (1.2) lib/rack/cache/context.rb:66:in `call!' rack-cache (1.2) lib/rack/cache/context.rb:51:in `call' railties (3.2.8) lib/rails/engine.rb:479:in `call' railties (3.2.8) lib/rails/application.rb:223:in `call' railties (3.2.8) lib/rails/railtie/configurable.rb:30:in `method_missing' passenger (3.0.17) lib/phusion_passenger/rack/request_handler.rb:96:in `process_request' passenger (3.0.17) lib/phusion_passenger/abstract_request_handler.rb:517:in `accept_and_process_next_request' passenger (3.0.17) lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop' passenger (3.0.17) lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler' passenger (3.0.17) lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application' passenger (3.0.17) lib/phusion_passenger/utils.rb:470:in `safe_fork' passenger (3.0.17) lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application' passenger (3.0.17) lib/phusion_passenger/spawn_manager.rb:264:in `spawn_rack_application' passenger (3.0.17) lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application' passenger (3.0.17) lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application' passenger (3.0.17) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop' passenger (3.0.17) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously' passenger (3.0.17) helper-scripts/passenger-spawn-server:102:in `<main>' (pid:32321) <---------------------------------------->
https://github.com/openshift/li/pull/817 rev#e821d77 in li.repo
Verified this bug after upgrading devenv-stage_281 to devenv_2749 The migrate-mongo-2.0.23 fixes this problem this time, thanks!