Bug 1106395

The bug blocks migration testing.

I followed the 'steps to reproduce' but could not find any issue. I had three apps (scalable, non-scalable, with db cartridge), all upgrade nicely. Can I ask for a retry? 
Couple of commits related to the quota have already been made into master repo, maybe..?

Still can be reproduced for my side, from devenv-stage_861 to devenv_4869.

+ oo-broker --non-interactive oo-admin-upgrade upgrade-node --version 2.0.46 --ignore-cartridge-version
Upgrader started with options: {:version=>"2.0.46", :ignore_cartridge_version=>true, :target_server_identity=>nil, :upgrade_position=>1, :num_upgraders=>1, :max_threads=>12, :gear_whitelist=>[], :num_tries=>2}
Building new upgrade queues and cluster metadata
Getting all active gears...
Getting all logins...
Writing 2 entries to gear queue for node ip-10-169-35-17 at /var/log/openshift/broker/upgrade/gear_queue_ip-10-169-35-17
Writing node queue to /var/log/openshift/broker/upgrade/node_queue
Writing cluster metadata to /var/log/openshift/broker/upgrade/cluster_metadata
Loading cluster metadata from /var/log/openshift/broker/upgrade/cluster_metadata
Loading node queue from /var/log/openshift/broker/upgrade/node_queue
Upgrading node ip-10-169-35-17 from gear queue file at /var/log/openshift/broker/upgrade/gear_queue_ip-10-169-35-17
1 of 1 nodes completed
Writing updated node queue to /var/log/openshift/broker/upgrade/node_queue
!!!!!!!!!!WARNING!!!!!!!!!!!!!WARNING!!!!!!!!!!!!WARNING!!!!!!!!!!
The upgrade was incomplete due to unprocessed or failed gears
remaining in node gear queues:  

2 failed gears:
  ip-10-169-35-17: 2

You can run the upgrade again with the same arguments to continue.
!!!!!!!!!!WARNING!!!!!!!!!!!!!WARNING!!!!!!!!!!!!WARNING!!!!!!!!!!
#####################################################
Summary:
# of users: 5
# of gears: 2
# of failures: 2
# of failures per plan:
    free: 2
# of failures per state:
    Active: 2
# of leftovers: 0
Gear counts per thread: [nil, nil, nil, 2]

Timings:
    start=1402917734.476s
    total=47.048s
Additional timings:
    gather_active_gears_total_time=20.787s
    gather_users_total_time=0.058s
#####################################################


Some log in platform.log:
./node/platform.log:June 16 07:22:56 INFO openshift-agent: request end: action=upgrade, requestid=65ff48e6733f5c83ac1f2b2057609c33, senderid=3886b78581f0, statuscode=0, data={:time=>nil, :output=>nil, :exitcode=>0, :upgrade_result_json=>"{\"gear_uuid\":\"539ebeba4b53dc1ed2000020\",\"hostname\":\"ip-10-169-35-17\",\"steps\":{\"inspect_gear_state\":{\"status\":\"complete\",\"errors\":[],\"context\":{\"preupgrade_state\":\"started\"}},\"pre_upgrade\":{\"status\":\"complete\",\"errors\":[],\"context\":{}},\"compute_itinerary\":{\"status\":\"complete\",\"errors\":[],\"context\":{}}},\"upgrade_complete\":false,\"errors\":[{\"message\":\"CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"backtrace\":\"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/node.rb:121:in `get_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:565:in `relax_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:477:in `upgrade_cartridges'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:162:in `execute'\\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:280:in `upgrade_action'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'\"}],\"warnings\":[],\"itinerary\":{\"jbosseap-6\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}},\"postgresql-9.2\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}}},\"times\":{\"start_time\":1402917776148,\"upgrade_on_node_measured_from_node\":58},\"broker_directives\":[],\"log\":[\"Beginning 2.0.46 upgrade for 539ebeba4b53dc1ed2000020\",\"Inspecting gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Pre-upgrade state: started\",\"Marking step inspect_gear_state complete\",\"Marking step pre_upgrade complete\",\"Refreshing cartridge redhat:jbosseap:6:0.0.17, ignoring cartridge version.\",\"Creating itinerary entry for incompatible upgrade of redhat:jbosseap:6:0.0.17\",\"Refreshing cartridge redhat:postgresql:9.2:0.3.16, ignoring cartridge version.\",\"Creating itinerary entry for incompatible upgrade of redhat:postgresql:9.2:0.3.16\",\"Marking step compute_itinerary complete\",\"Migrating gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Caught an exception during upgrade: CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"Total upgrade time on node (ms): 58\"]}"}   
./node/platform.log:June 16 07:23:00 INFO openshift-agent: request end: action=upgrade, requestid=35822b7c57eb5ad4b0fc7f3683b45e74, senderid=3886b78581f0, statuscode=0, data={:time=>nil, :output=>nil, :exitcode=>0, :upgrade_result_json=>"{\"gear_uuid\":\"539ebeba4b53dc1ed2000020\",\"hostname\":\"ip-10-169-35-17\",\"steps\":{\"inspect_gear_state\":{\"status\":\"complete\",\"errors\":[],\"context\":{}},\"pre_upgrade\":{\"status\":\"complete\",\"errors\":[],\"context\":{}}},\"upgrade_complete\":false,\"errors\":[{\"message\":\"CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"backtrace\":\"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/node.rb:121:in `get_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:565:in `relax_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:477:in `upgrade_cartridges'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:162:in `execute'\\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:280:in `upgrade_action'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'\"}],\"warnings\":[],\"itinerary\":{\"jbosseap-6\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}},\"postgresql-9.2\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}}},\"times\":{\"start_time\":1402917780324,\"upgrade_on_node_measured_from_node\":4},\"broker_directives\":[],\"log\":[\"Beginning 2.0.46 upgrade for 539ebeba4b53dc1ed2000020\",\"Inspecting gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Migrating gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Caught an exception during upgrade: CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"Total upgrade time on node (ms): 4\"]}"}

The devenv package overwrites the quota.
Pull request - https://github.com/openshift/li/pull/2706

Verified on devenv_4882, the quota issue is fixed

Building new upgrade queues and cluster metadata
Getting all active gears...
Getting all logins...
Writing 6 entries to gear queue for node ip-10-136-92-228 at /var/log/openshift/broker/upgrade/gear_queue_ip-10-136-92-228
Writing node queue to /var/log/openshift/broker/upgrade/node_queue
Writing cluster metadata to /var/log/openshift/broker/upgrade/cluster_metadata
Loading cluster metadata from /var/log/openshift/broker/upgrade/cluster_metadata
Loading node queue from /var/log/openshift/broker/upgrade/node_queue
Upgrading node ip-10-136-92-228 from gear queue file at /var/log/openshift/broker/upgrade/gear_queue_ip-10-136-92-228
1 of 1 nodes completed
Writing updated node queue to /var/log/openshift/broker/upgrade/node_queue
#####################################################
Summary:
# of users: 5
# of gears: 6
# of failures: 0
# of leftovers: 0
Gear counts per thread: [nil, nil, nil, 6]

Timings:
    start=1403093877.785s
    total=93.01s
Additional timings:
    gather_active_gears_total_time=20.717s
    gather_users_total_time=0.032s
#####################################################