Created attachment 904588 [details] upgrade_errors Description of problem: oo-admin-upgrade failed to upgrade all gears due to one kind of error: 'Unable to obtain quota for user $UUID' Version-Release number of selected component (if applicable): On devenv_4850 How reproducible: Always, migrated twice, reproduced twice Steps to Reproduce: 1. Create applications with different cartridges on devenv-stage_861 2. Upgrade the devenv-stage_861 instance to current latest devenv_4850 version 3. Migrate nodes. Actual results: Failed to upgrade all gears, tracked log, all gears failed to upgrade with one kind of error: {"message":"CLIENT_ERROR: Unable to obtain quota for user 5395a3234b53dc7caa000001\n","backtrace":"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.1/lib/openshift-origin-node/model/node.rb:120:in `get_quota' .... For detail, see log attached Expected results: Should successfully upgraded the gears. Additional info:
The bug blocks migration testing.
I followed the 'steps to reproduce' but could not find any issue. I had three apps (scalable, non-scalable, with db cartridge), all upgrade nicely. Can I ask for a retry? Couple of commits related to the quota have already been made into master repo, maybe..?
Still can be reproduced for my side, from devenv-stage_861 to devenv_4869. + oo-broker --non-interactive oo-admin-upgrade upgrade-node --version 2.0.46 --ignore-cartridge-version Upgrader started with options: {:version=>"2.0.46", :ignore_cartridge_version=>true, :target_server_identity=>nil, :upgrade_position=>1, :num_upgraders=>1, :max_threads=>12, :gear_whitelist=>[], :num_tries=>2} Building new upgrade queues and cluster metadata Getting all active gears... Getting all logins... Writing 2 entries to gear queue for node ip-10-169-35-17 at /var/log/openshift/broker/upgrade/gear_queue_ip-10-169-35-17 Writing node queue to /var/log/openshift/broker/upgrade/node_queue Writing cluster metadata to /var/log/openshift/broker/upgrade/cluster_metadata Loading cluster metadata from /var/log/openshift/broker/upgrade/cluster_metadata Loading node queue from /var/log/openshift/broker/upgrade/node_queue Upgrading node ip-10-169-35-17 from gear queue file at /var/log/openshift/broker/upgrade/gear_queue_ip-10-169-35-17 1 of 1 nodes completed Writing updated node queue to /var/log/openshift/broker/upgrade/node_queue !!!!!!!!!!WARNING!!!!!!!!!!!!!WARNING!!!!!!!!!!!!WARNING!!!!!!!!!! The upgrade was incomplete due to unprocessed or failed gears remaining in node gear queues: 2 failed gears: ip-10-169-35-17: 2 You can run the upgrade again with the same arguments to continue. !!!!!!!!!!WARNING!!!!!!!!!!!!!WARNING!!!!!!!!!!!!WARNING!!!!!!!!!! ##################################################### Summary: # of users: 5 # of gears: 2 # of failures: 2 # of failures per plan: free: 2 # of failures per state: Active: 2 # of leftovers: 0 Gear counts per thread: [nil, nil, nil, 2] Timings: start=1402917734.476s total=47.048s Additional timings: gather_active_gears_total_time=20.787s gather_users_total_time=0.058s ##################################################### Some log in platform.log: ./node/platform.log:June 16 07:22:56 INFO openshift-agent: request end: action=upgrade, requestid=65ff48e6733f5c83ac1f2b2057609c33, senderid=3886b78581f0, statuscode=0, data={:time=>nil, :output=>nil, :exitcode=>0, :upgrade_result_json=>"{\"gear_uuid\":\"539ebeba4b53dc1ed2000020\",\"hostname\":\"ip-10-169-35-17\",\"steps\":{\"inspect_gear_state\":{\"status\":\"complete\",\"errors\":[],\"context\":{\"preupgrade_state\":\"started\"}},\"pre_upgrade\":{\"status\":\"complete\",\"errors\":[],\"context\":{}},\"compute_itinerary\":{\"status\":\"complete\",\"errors\":[],\"context\":{}}},\"upgrade_complete\":false,\"errors\":[{\"message\":\"CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"backtrace\":\"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/node.rb:121:in `get_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:565:in `relax_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:477:in `upgrade_cartridges'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:162:in `execute'\\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:280:in `upgrade_action'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'\"}],\"warnings\":[],\"itinerary\":{\"jbosseap-6\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}},\"postgresql-9.2\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}}},\"times\":{\"start_time\":1402917776148,\"upgrade_on_node_measured_from_node\":58},\"broker_directives\":[],\"log\":[\"Beginning 2.0.46 upgrade for 539ebeba4b53dc1ed2000020\",\"Inspecting gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Pre-upgrade state: started\",\"Marking step inspect_gear_state complete\",\"Marking step pre_upgrade complete\",\"Refreshing cartridge redhat:jbosseap:6:0.0.17, ignoring cartridge version.\",\"Creating itinerary entry for incompatible upgrade of redhat:jbosseap:6:0.0.17\",\"Refreshing cartridge redhat:postgresql:9.2:0.3.16, ignoring cartridge version.\",\"Creating itinerary entry for incompatible upgrade of redhat:postgresql:9.2:0.3.16\",\"Marking step compute_itinerary complete\",\"Migrating gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Caught an exception during upgrade: CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"Total upgrade time on node (ms): 58\"]}"} ./node/platform.log:June 16 07:23:00 INFO openshift-agent: request end: action=upgrade, requestid=35822b7c57eb5ad4b0fc7f3683b45e74, senderid=3886b78581f0, statuscode=0, data={:time=>nil, :output=>nil, :exitcode=>0, :upgrade_result_json=>"{\"gear_uuid\":\"539ebeba4b53dc1ed2000020\",\"hostname\":\"ip-10-169-35-17\",\"steps\":{\"inspect_gear_state\":{\"status\":\"complete\",\"errors\":[],\"context\":{}},\"pre_upgrade\":{\"status\":\"complete\",\"errors\":[],\"context\":{}}},\"upgrade_complete\":false,\"errors\":[{\"message\":\"CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"backtrace\":\"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/node.rb:121:in `get_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:565:in `relax_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:477:in `upgrade_cartridges'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:162:in `execute'\\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:280:in `upgrade_action'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'\"}],\"warnings\":[],\"itinerary\":{\"jbosseap-6\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}},\"postgresql-9.2\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}}},\"times\":{\"start_time\":1402917780324,\"upgrade_on_node_measured_from_node\":4},\"broker_directives\":[],\"log\":[\"Beginning 2.0.46 upgrade for 539ebeba4b53dc1ed2000020\",\"Inspecting gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Migrating gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Caught an exception during upgrade: CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"Total upgrade time on node (ms): 4\"]}"}
The devenv package overwrites the quota. Pull request - https://github.com/openshift/li/pull/2706
Verified on devenv_4882, the quota issue is fixed Building new upgrade queues and cluster metadata Getting all active gears... Getting all logins... Writing 6 entries to gear queue for node ip-10-136-92-228 at /var/log/openshift/broker/upgrade/gear_queue_ip-10-136-92-228 Writing node queue to /var/log/openshift/broker/upgrade/node_queue Writing cluster metadata to /var/log/openshift/broker/upgrade/cluster_metadata Loading cluster metadata from /var/log/openshift/broker/upgrade/cluster_metadata Loading node queue from /var/log/openshift/broker/upgrade/node_queue Upgrading node ip-10-136-92-228 from gear queue file at /var/log/openshift/broker/upgrade/gear_queue_ip-10-136-92-228 1 of 1 nodes completed Writing updated node queue to /var/log/openshift/broker/upgrade/node_queue ##################################################### Summary: # of users: 5 # of gears: 6 # of failures: 0 # of leftovers: 0 Gear counts per thread: [nil, nil, nil, 6] Timings: start=1403093877.785s total=93.01s Additional timings: gather_active_gears_total_time=20.717s gather_users_total_time=0.032s #####################################################