Bug 815406 - Some gears don't have UIDs set in mongo
Some gears don't have UIDs set in mongo
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
high Severity urgent
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
: Triaged
Depends On:
Blocks: 819074
  Show dependency treegraph
 
Reported: 2012-04-23 10:36 EDT by Thomas Wiest
Modified: 2015-05-14 21:51 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 819074 (view as bug list)
Environment:
Last Closed: 2012-05-14 13:22:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thomas Wiest 2012-04-23 10:36:01 EDT
Description of problem:
While investigating a separate issue, I noticed that some gears in mongo don't have UIDs set. 

*** Note, this is the Linux "user id" (for instance 1000) and NOT the gear's UUID guid.

These gears simply don't have the UID field set on them.

I wrote a small script to identify the affected gears and according to my script, there are 5411 gears in this state. I've provided the list of affected gears to Krishna, Dan and Mike.

*** Note, this does not seem to be affecting newly created gears, so it seems to have been caused by either the rhc-admin-moves -or- one of the release migrations.

The gears seem to be working just fine (2 of my gears are in this state and are working).


Version-Release number of selected component (if applicable):
rhc-broker-0.90.25-1.el6_2.noarch


How reproducible:
unsure


Steps to Reproduce:
1. unsure of the steps that got these gears into this situation.

  
Actual results:
gears don't have UID set in mongo


Expected results:
They should have their assigned UID set in mongo
Comment 1 Thomas Wiest 2012-04-23 10:39:48 EDT
A side affect of this bug is that if the gears without the UIDs set are moved, then they end up with UIDs on the new systems of < 1000 (still no UID set in mongo).

We currently have 37 gears in this state. I've provided a list of these gears to Dan, Mike and Krishna.
Comment 2 Dan McPherson 2012-04-23 10:41:12 EDT
It's also worth noting for those 37 gears they are taking up a district spot still.  So when they are added back to the district we need to figure out what uid they used to have.
Comment 3 Russell Harrison 2012-04-24 23:21:32 EDT
This bug is a full stop blocker for ops.  We need to get those users migrated properly as they are preventing proper management of users / groups outside of the range allotted to gears.
Comment 4 Rajat Chopra 2012-05-03 12:28:13 EDT
fixed with rev#bdef3dd9d8da028f21c5c6f621a9893114b0526b
The script is checked in at li/misc/maintenance/bin/rhc-fix-gear-uids

How to test?
 - On a devenv, or on clustered devenv with districts on/off. 
 - Just run the script with root privileges. If any gears had the 'uid' missing the script will declare them fixed.
 - On a simple devenv with districts off, gears do not have their uids assigned to them already, so this script will put them right. Running the script again will show that the fixes were alright.
 - Check in mongo, that the uids are assigned really.

Stg/Prod congruent test :
 - Create a node cluster with district on
 - create an app (not a scalable app)
 - manually change app's gear's uid and set it to nil
   quick script steps to do the above :
     require '/var/www/stickshift/broker/config/environment'
     user = CloudUser.find(<username>)
     app  = Application.find(user, <appname>)
     print app.gear.uid
     app.gear.uid = nil
     app.save
  - run the rhc-fix-gear-uids script
  - check in mongo if the gear's uid is fixed



Note: This fix just fixes the ~5.4k gears that have missing uids, not the 37 broken gears. That will be another script at release time.
Comment 5 Rajat Chopra 2012-05-03 12:35:46 EDT
Thomas found another scenario. In this scenario, the uid of the gear needs to be removed from mongo manually. So that needs to be tested as well. Pasting the email below.

Hey Guys,
    I just had a chance to read the backlog of #libra from earlier today where you guys were talking about testing the bug fix for the "missing gear UID in mongo bug" (bug 815406) and you guys said this:

(05/02/2012 03:36:11 PM) dmcphers: rchopra: so back to how does qe test this?
(05/02/2012 03:36:21 PM) dmcphers: change their mongo uids to be nil
(05/02/2012 03:36:24 PM) dmcphers: then run the script
(05/02/2012 03:36:28 PM) dmcphers: ?
(05/02/2012 03:36:44 PM) rchopra: dmcphers, for the regular ~5400 or so, this script should be alright... and test this on devenv (thats how i just tested it)
(05/02/2012 03:37:02 PM) dmcphers: rchopra: k we need to give them those instructions

So, it's important to note that it's not that the UID field is set to null, the field itself is completely absent.

Attached is the json from my PROD user.

There are 3 apps, 2 without UIDs in mongo.

[0]  deploy0116a -> no uid in mongo
[1]  jmigrate -> no uid in mongo
[2]  appwuid -> has uid in mongo


Here's a quick irb to show what I'm talking about (notice that the first 2 apps don't have the uid field / key, but the 3rd app does):
>> t["apps"][0]["name"]
=> "deploy0116a"
>> t["apps"][0]["group_instances"][0]["gears"][0].keys
=> ["group_instance_name", "node_profile", "name", "configured_components", "uuid", "server_identity"]

>> t["apps"][1]["name"]
=> "jmigrate"
>> t["apps"][1]["group_instances"][0]["gears"][0].keys
=> ["group_instance_name", "node_profile", "name", "configured_components", "uuid", "server_identity"]

>> t["apps"][2]["name"]
=> "appwuid"
>> t["apps"][2]["group_instances"][0]["gears"][0].keys
=> ["group_instance_name", "node_profile", "uid", "name", "configured_components", "uuid", "server_identity"]


So, to test, you need to actually remove the uid field from the app in mongo.

Thanks,
Thomas
Comment 6 Dan McPherson 2012-05-04 10:07:45 EDT
Hi Rajat, have both migrations been done?  If not do we need a separate bug for the 37 apps?
Comment 7 Xiaoli Tian 2012-05-08 05:44:12 EDT
Tested it from devenv_1761,

1. After creating some app in a single devenv instance, the uid value is null

2. Run rhc-fix-gear-uids, 
# ./rhc-fix-gear-uids 
Gear UID cleanup summary : 
Gear '03bdf81d2a3a401d90f262cb2f81d5f1' in app eaef426ac28147a1831afab397d6d8bb for user 'xtian+t91@redhat.com', needs its uid to be set from 'nil' to '501'...done.
Gear 'e00500a4735240bda4915a0ea20fc227' in app e00500a4735240bda4915a0ea20fc227 for user 'xtian+t91@redhat.com', needs its uid to be set from 'nil' to '502'...done.
Gear 'eaef426ac28147a1831afab397d6d8bb' in app eaef426ac28147a1831afab397d6d8bb for user 'xtian+t91@redhat.com', needs its uid to be set from 'nil' to '500'...done.

3. Query in mongo
#db.user.find({"_id":"xtian+t91@redhat.com"},{_id:1,"apps.group_instances.gears.uid":1})
{ "_id" : "xtian+t91@redhat.com", "apps" : [ 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	"uid" : "501" } ] }, 	{ 	"gears" : [ 	{ 	"uid" : "500" } ] } ] }, 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	"uid" : "502" } ] } ] } ] }

4. Try to remove the uid field from mongo to test if rhc-fix-gear-uids can fix it
PRIMARY> db.user.update({},{$unset:{"apps.group_instances.gears.uid":1}},false,true)

5. Query again in mongo

It got the same result in step 3, then it seems the uid field is not removed sucessfully.

Hi, Rajat

Is there anything wrong with my step 4 to remove the uid field ? can you please tell me how to remove the field successfully?

Thanks
Comment 8 Ravi Sankar 2012-05-08 16:21:42 EDT
Hi Xiaoli,
removing gear 'uid' field from the nested array in the mongo:

(1) PRIMARY> db.user.update({ "_id" : "rp220", "apps.name" : "app1"}, {$unset : { "apps.$.group_instances.0.gears.0.uid" : 1 }})

 The above query does this: For user 'rp220' and for given app 'app1', it removes 'uid' field for 0th gear in 0th group_instances.

(2) To remove 'uid' field from all gears and for all group_instances for the given <username> and <appname>, you can run this script on the mongo shell. 

var username = <username>;
var appname = <appname>;
var z = db.user.findOne({ "_id" : username, "apps.name" : appname});
for (var ai = 0; ai < z.apps.length; ai++) {
   if (z.apps[ai].name == appname) {
      for (var i=0; i < z.apps[ai].group_instances.length; i++) {
         for (var g=0; g < z.apps[ai].group_instances[i].gears.length; g++) {
             delete(z.apps[ai].group_instances[i].gears[g].uid);
         }
      }
  }
}
db.user.save(z);
Comment 9 Xiaoli Tian 2012-05-09 07:44:49 EDT
Thanks Ravi !

Have tested again on devenv_1762,

If the uid is null, then rhc-fix-gear-uids will add value for it.
./rhc-fix-gear-uids 
Gear UID cleanup summary : 
Gear '2985e73f18a34a8ea0dab91e01a49942' in app 2985e73f18a34a8ea0dab91e01a49942 for user 'xtian+t94@redhat.com', needs its uid to be set from 'nil' to '501'...done.
Gear 'ce2e5313142548a0b9d40a55e36287d7' in app ce2e5313142548a0b9d40a55e36287d7 for user 'xtian+t94@redhat.com', needs its uid to be set from 'nil' to '503'...done.
Gear '575ac43fc80149f8aa3d083ba8be4956' in app 575ac43fc80149f8aa3d083ba8be4956 for user 'xtian+t94@redhat.com', needs its uid to be set from 'nil' to '500'...done.
Gear 'd8f8a0fa4fad4c989815cff838f242b5' in app ce2e5313142548a0b9d40a55e36287d7 for user 'xtian+t94@redhat.com', needs its uid to be set from 'nil' to '502'...done.

After remove 3 uid fields, only 1 remaining:
Primary>>db.user.find({"_id":"xtian+t94@redhat.com"},{_id:1,"apps.group_instances.gears.uid":1})
{ "_id" : "xtian+t94@redhat.com", "apps" : [ 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	 } ] } ] }, 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	 } ] } ] }, 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	 } ] }, 	{ 	"gears" : [ 	{ 	"uid" : "502" } ] } ] } ] }


And then run 
./rhc-fix-gear-uids
All the uid field come back:
PRIMARY> db.user.find({"_id":"xtian+t94@redhat.com"},{_id:1,"apps.group_instances.gears.uid":1})
{ "_id" : "xtian+t94@redhat.com", "apps" : [ 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	"uid" : "500" } ] } ] }, 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	"uid" : "501" } ] } ] }, 	{ 	"group_instances" : [ 	{ 	"gears" : [ 	{ 	"uid" : "503" } ] }, 	{ 	"gears" : [ 	{ 	"uid" : "502" } ] } ] } ] }

Note You need to log in before you can comment on or make changes to this bug.