Bug 1106395 - Unable obtain gear quota during gear upgrade
Summary: Unable obtain gear quota during gear upgrade
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 1.x
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Rajat Chopra
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-06-09 08:57 UTC by Jianwei Hou
Modified: 2014-07-15 10:30 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-07-15 10:30:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
upgrade_errors (199.99 KB, text/plain)
2014-06-09 08:57 UTC, Jianwei Hou
no flags Details

Description Jianwei Hou 2014-06-09 08:57:13 UTC
Created attachment 904588 [details]
upgrade_errors

Description of problem:
oo-admin-upgrade failed to upgrade all gears due to one kind of error: 'Unable to obtain quota for user $UUID'

Version-Release number of selected component (if applicable):
On devenv_4850

How reproducible:
Always, migrated twice, reproduced twice

Steps to Reproduce:
1. Create applications with different cartridges on devenv-stage_861
2. Upgrade the devenv-stage_861 instance to current latest devenv_4850 version
3. Migrate nodes.

Actual results:
Failed to upgrade all gears, tracked log, all gears failed to upgrade with one kind of error:
{"message":"CLIENT_ERROR: Unable to obtain quota for user 5395a3234b53dc7caa000001\n","backtrace":"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.1/lib/openshift-origin-node/model/node.rb:120:in `get_quota'
....
For detail, see log attached

Expected results:
Should successfully upgraded the gears.

Additional info:

Comment 1 Meng Bo 2014-06-12 07:28:10 UTC
The bug blocks migration testing.

Comment 2 Rajat Chopra 2014-06-13 18:34:17 UTC
I followed the 'steps to reproduce' but could not find any issue. I had three apps (scalable, non-scalable, with db cartridge), all upgrade nicely. Can I ask for a retry? 
Couple of commits related to the quota have already been made into master repo, maybe..?

Comment 3 Meng Bo 2014-06-16 08:31:54 UTC
Still can be reproduced for my side, from devenv-stage_861 to devenv_4869.

+ oo-broker --non-interactive oo-admin-upgrade upgrade-node --version 2.0.46 --ignore-cartridge-version
Upgrader started with options: {:version=>"2.0.46", :ignore_cartridge_version=>true, :target_server_identity=>nil, :upgrade_position=>1, :num_upgraders=>1, :max_threads=>12, :gear_whitelist=>[], :num_tries=>2}
Building new upgrade queues and cluster metadata
Getting all active gears...
Getting all logins...
Writing 2 entries to gear queue for node ip-10-169-35-17 at /var/log/openshift/broker/upgrade/gear_queue_ip-10-169-35-17
Writing node queue to /var/log/openshift/broker/upgrade/node_queue
Writing cluster metadata to /var/log/openshift/broker/upgrade/cluster_metadata
Loading cluster metadata from /var/log/openshift/broker/upgrade/cluster_metadata
Loading node queue from /var/log/openshift/broker/upgrade/node_queue
Upgrading node ip-10-169-35-17 from gear queue file at /var/log/openshift/broker/upgrade/gear_queue_ip-10-169-35-17
1 of 1 nodes completed
Writing updated node queue to /var/log/openshift/broker/upgrade/node_queue
!!!!!!!!!!WARNING!!!!!!!!!!!!!WARNING!!!!!!!!!!!!WARNING!!!!!!!!!!
The upgrade was incomplete due to unprocessed or failed gears
remaining in node gear queues:  

2 failed gears:
  ip-10-169-35-17: 2

You can run the upgrade again with the same arguments to continue.
!!!!!!!!!!WARNING!!!!!!!!!!!!!WARNING!!!!!!!!!!!!WARNING!!!!!!!!!!
#####################################################
Summary:
# of users: 5
# of gears: 2
# of failures: 2
# of failures per plan:
    free: 2
# of failures per state:
    Active: 2
# of leftovers: 0
Gear counts per thread: [nil, nil, nil, 2]

Timings:
    start=1402917734.476s
    total=47.048s
Additional timings:
    gather_active_gears_total_time=20.787s
    gather_users_total_time=0.058s
#####################################################


Some log in platform.log:
./node/platform.log:June 16 07:22:56 INFO openshift-agent: request end: action=upgrade, requestid=65ff48e6733f5c83ac1f2b2057609c33, senderid=3886b78581f0, statuscode=0, data={:time=>nil, :output=>nil, :exitcode=>0, :upgrade_result_json=>"{\"gear_uuid\":\"539ebeba4b53dc1ed2000020\",\"hostname\":\"ip-10-169-35-17\",\"steps\":{\"inspect_gear_state\":{\"status\":\"complete\",\"errors\":[],\"context\":{\"preupgrade_state\":\"started\"}},\"pre_upgrade\":{\"status\":\"complete\",\"errors\":[],\"context\":{}},\"compute_itinerary\":{\"status\":\"complete\",\"errors\":[],\"context\":{}}},\"upgrade_complete\":false,\"errors\":[{\"message\":\"CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"backtrace\":\"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/node.rb:121:in `get_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:565:in `relax_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:477:in `upgrade_cartridges'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:162:in `execute'\\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:280:in `upgrade_action'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'\"}],\"warnings\":[],\"itinerary\":{\"jbosseap-6\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}},\"postgresql-9.2\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}}},\"times\":{\"start_time\":1402917776148,\"upgrade_on_node_measured_from_node\":58},\"broker_directives\":[],\"log\":[\"Beginning 2.0.46 upgrade for 539ebeba4b53dc1ed2000020\",\"Inspecting gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Pre-upgrade state: started\",\"Marking step inspect_gear_state complete\",\"Marking step pre_upgrade complete\",\"Refreshing cartridge redhat:jbosseap:6:0.0.17, ignoring cartridge version.\",\"Creating itinerary entry for incompatible upgrade of redhat:jbosseap:6:0.0.17\",\"Refreshing cartridge redhat:postgresql:9.2:0.3.16, ignoring cartridge version.\",\"Creating itinerary entry for incompatible upgrade of redhat:postgresql:9.2:0.3.16\",\"Marking step compute_itinerary complete\",\"Migrating gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Caught an exception during upgrade: CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"Total upgrade time on node (ms): 58\"]}"}   
./node/platform.log:June 16 07:23:00 INFO openshift-agent: request end: action=upgrade, requestid=35822b7c57eb5ad4b0fc7f3683b45e74, senderid=3886b78581f0, statuscode=0, data={:time=>nil, :output=>nil, :exitcode=>0, :upgrade_result_json=>"{\"gear_uuid\":\"539ebeba4b53dc1ed2000020\",\"hostname\":\"ip-10-169-35-17\",\"steps\":{\"inspect_gear_state\":{\"status\":\"complete\",\"errors\":[],\"context\":{}},\"pre_upgrade\":{\"status\":\"complete\",\"errors\":[],\"context\":{}}},\"upgrade_complete\":false,\"errors\":[{\"message\":\"CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"backtrace\":\"/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/node.rb:121:in `get_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:565:in `relax_quota'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:477:in `upgrade_cartridges'\\n/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.26.4/lib/openshift-origin-node/model/upgrade.rb:162:in `execute'\\n/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:280:in `upgrade_action'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'\\n/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'\\n/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'\"}],\"warnings\":[],\"itinerary\":{\"jbosseap-6\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}},\"postgresql-9.2\":{\"upgrade_type\":\"incompatible\",\"upgrade_data\":{\"private_endpoints_remove\":[],\"public_endpoints_remove\":[],\"public_endpoints_add\":[]}}},\"times\":{\"start_time\":1402917780324,\"upgrade_on_node_measured_from_node\":4},\"broker_directives\":[],\"log\":[\"Beginning 2.0.46 upgrade for 539ebeba4b53dc1ed2000020\",\"Inspecting gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Migrating gear at /var/lib/openshift/539ebeba4b53dc1ed2000020\",\"Caught an exception during upgrade: CLIENT_ERROR: Unable to obtain quota for user 539ebeba4b53dc1ed2000020\\n\",\"Total upgrade time on node (ms): 4\"]}"}

Comment 5 Rajat Chopra 2014-06-17 19:43:17 UTC
The devenv package overwrites the quota.
Pull request - https://github.com/openshift/li/pull/2706

Comment 6 Jianwei Hou 2014-06-18 09:12:37 UTC
Verified on devenv_4882, the quota issue is fixed

Building new upgrade queues and cluster metadata
Getting all active gears...
Getting all logins...
Writing 6 entries to gear queue for node ip-10-136-92-228 at /var/log/openshift/broker/upgrade/gear_queue_ip-10-136-92-228
Writing node queue to /var/log/openshift/broker/upgrade/node_queue
Writing cluster metadata to /var/log/openshift/broker/upgrade/cluster_metadata
Loading cluster metadata from /var/log/openshift/broker/upgrade/cluster_metadata
Loading node queue from /var/log/openshift/broker/upgrade/node_queue
Upgrading node ip-10-136-92-228 from gear queue file at /var/log/openshift/broker/upgrade/gear_queue_ip-10-136-92-228
1 of 1 nodes completed
Writing updated node queue to /var/log/openshift/broker/upgrade/node_queue
#####################################################
Summary:
# of users: 5
# of gears: 6
# of failures: 0
# of leftovers: 0
Gear counts per thread: [nil, nil, nil, 6]

Timings:
    start=1403093877.785s
    total=93.01s
Additional timings:
    gather_active_gears_total_time=20.717s
    gather_users_total_time=0.032s
#####################################################


Note You need to log in before you can comment on or make changes to this bug.