Red Hat Bugzilla – Bug 1064631
Cannot change the starting UID for gears to a high number
Last modified: 2017-03-08 12:38 EST
Created attachment 862597 [details] platform-trace.log segment from trying to create undistricted gear +++ This bug was initially created as a clone of Bug #1064570 +++ Description of problem: I have a policy where UIDs below 10,000,000 on my hosts need to be reserved for corporate logins. So, I would like gear UIDs on my nodes to start at 10000000. This apparently cannot be done. Steps to Reproduce: 1. On my one-node system without districts, set the following values in /etc/openshift/node.conf: GEAR_MIN_UID=1000000 UID_BEGIN=1000000 # note: setting both, see bug 1051015 GEAR_MAX_UID=1005999 # range is still the same size 2. service ruby193-mcollective restart 3. Try to create a scaled php+mysql app. It fails. 4. Create a district and put the node in it. 5. Try to create a scaled php+mysql app. Results: 3. App creation fails with node execution error. There isn't much in the logs to indicate what went wrong, but I suspect it has to do with the port calculation coming up with something bogus. The platform-trace.log for this operation is attached. 5. The app creation succeeds! However, the gears have UIDs in the normal range, not in the range I want. Expected results: Both should work, both should end up with gear UIDs in the specified range (map from district UIDs to the range), and everything should route properly between gears via the external ports. Additional info: By the way, once this is working, if I already have nodes with gears in the usual range, I expect to be able to move those gears to new nodes with the higher range so that I can get rid of the old nodes. --- Additional comment from Luke Meyer on 2014-02-12 16:27:39 EST --- 1M or 10M, either way it fails. And I meant to note that at step 3, it does actually try to create the gear user with UID_BEGIN. So the node is reading the values.
Just to further confuse things, I noticed that there's a parameter /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf:DISTRICTS_FIRST_UID that apparently needs to be set to match the beginning of the node range, prior to creating a district. However, with this done gear creation still failed. I note it here to enable testing once this is fixed.
Current problems needing to be fixed: * IP addresses do not properly wrap around and the upper limit is not documented * TC does not wrap around. This should be very easy. The fix for this BZ will require tests to ensure backwards compatibility.
To test this make sure the following configs match: node.conf GEAR_MIN_UID GEAR_MAX_UID openshift-origin-msg-broker-mcollective.conf DISTRICTS_MAX_CAPACITY DISTRICTS_FIRST_UID eg, GEAR_MIN_UID=501000 GEAR_MAX_UID=506999 DISTRICTS_MAX_CAPACITY=6000 DISTRICTS_FIRST_UID=501000 Be careful to ensure the settings are correct at the time of district creation. In theory UIDs up to 2^32 should now be allowed. I have only tested in the 500k range so there is a good chance there is a point in which the selinux categories do not properly wrap around. This change allows ip address ranges and TC queuing disciplines to wrap forever.
*** Bug 1075735 has been marked as a duplicate of this bug. ***
*** Bug 1051251 has been marked as a duplicate of this bug. ***
The practical UID limit for our MCS labeling in OpenShift is currently 523,776. I think we have options to increase this if it's ever needed but given that a free range of 6000 UIDs is all that is needed this should be high enough for now. You can see the problem here: https://github.com/openshift/origin-server/blob/fa31633e6eab70783950bfc5139f74cc5c1beb4d/node/lib/openshift-origin-node/utils/selinux_context.rb#L220 The default group_size is 2 which yields 523,776 possibilities. The code is optimized for the default. Modifying the group size is possible however that triggers the logic at line #237 which a simplistic brute force way to find the correct mls label. Setting a group size of 3 will likely OOM. For now I would recommend customers keep the UID ranges below 500,000.
Verified and pass on puddle-2-1-2014-05-02 Set values as below: In node.conf GEAR_MIN_UID=523774 GEAR_MAX_UID=523777 In openshift-origin-msg-broker-mcollective.conf DISTRICTS_MAX_CAPACITY=4 DISTRICTS_FIRST_UID=523774 1. Don't enable district. Create app, scaled up app or add cartridge. the result is as expected. 1)All actions can create userid=523775. 2)All actions will fail with "Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server" when try to create user >523775. 3) The gear uid=523775 can be moved to district nodes 2. Enable district 1) Create district and set max_uid=>523775.(Valid userid are 523774, and 523775 now) Create app, scaled up app or add cartridge. catridge can be created and userid can be 523775. 2) Create one scaled app with two gears, and then add capacity one. (current avaliable UID is 523776) Scale up app,add cartridge to app, create app. all action failed with 'Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server'. That is as expected. 3) The gear uid=523775 can be moved in district. >Additional info: >By the way, once this is working, if I already have nodes with gears in the >usual range, I expect to be able to move those gears to new nodes with the >higher range so that I can get rid of the old nodes. currently, Openshift don't allow to change the uid during moving.
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/b11ca1afcc9663c16e4725c2eb178c430dda7641 Bug 1064631 - Wrap UID-based ip addresses and netclasses calculations