Bug 1043387

Summary: [fork] The app_ssh_keys array has been unshifted with meaningless hashes by the datastore migration program
Product: OpenShift Online Reporter: Jianwei Hou <jhou>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.x   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-30 00:53:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
After migration none

Description Jianwei Hou 2013-12-16 07:27:38 UTC
Created attachment 837125 [details]
After migration

Description of problem:
After migration on fork_ami_bug_1039151_1041628_967, found that in mongo, the app_ssh_keys array in applications document has been appended with some additional hashes which have only _id and _type fields.

Version-Release number of selected component (if applicable):
On fork_ami_bug_1039151_1041628_967

How reproducible:
Always

Steps to Reproduce:
1. Launch a devenv-stage ami, currently the most recent ami is devenv-stage_616
2. Prepare applications
3. Upgrade the devenv-stage ami to the same version as fork_ami_bug_1039151_1041628_967
4. Do datastore migration
rhc-admin-migrate-datastore --non-compatible --version 2.0.37.3
5. After migration, check the applications collection. Verify that each app_ssh_key has valid _id, _type, content, name and component_id

Actual results:
Take the scalable jbosseap-6 application for example:
Before migration took place:
libra_rs:PRIMARY> db.applications.findOne({"name":"jbosseap1s"},{app_ssh_keys:1})
{
	"_id" : ObjectId("52ae6900f4e9eeed250000d3"),
	"app_ssh_keys" : [
		{
			"_id" : ObjectId("52ae6959f4e9eeed25000111"),
			"_type" : "ApplicationSshKey",
			"component_id" : ObjectId("52ae6900f4e9eeed250000d4"),
			"content" : "AAAAB3NzaC1yc2EAAAABIwAAAQEAzTllYT4wwnyBbT5nyjk2Ekyy/jye6dz1yTbH3iGFAOFR7UR+jujucpu0VeZPj9qOlVIxQrUucBEWUPjRln+78csaen7JZaA4Cc8WnDuchRPZ97gXg+cCdiLFMyf6lffX9pHIIy2wJbs1tNVGaX4kVB8nZIRv9/y6CauM+SQxciU8jGeKcbX54VOSir53kJvlgzs4Ordq0kGDBusMZO7J2rJp/qh1L4suuKAmqNTYhixlSM982W07vTJ6lDam/6vWl8dfyFEfVFUZyRNBO/qY8g2vIJYbb1YoLEvw7s3lf6C1d1rlVTPLvGaYmC07HRvza4jTvnFa3v4vrWFOCGDdZQ==",
			"name" : "application-323074577224474952728576",
			"type" : "ssh-rsa"
		},
		{
			"_id" : ObjectId("52ae6959f4e9eeed25000112"),
			"_type" : "ApplicationSshKey",
			"component_id" : ObjectId("52ae6900f4e9eeed250000d3"),
			"content" : "AAAAB3NzaC1yc2EAAAABIwAAAQEAuANGwPsYI4K2y+EC8GCuLwhP03pOTE8qMU3UxgcZbNX7qzLcBFfhNhdZHHLohqnEG/OKBf8uYruzJLptle70iRCqwi8yqk29bgcspY60FhAtm3eBFw1atuYFkronTKygS33Wp5BsIbyuZlschXXwkxtmRam7N0KUVZQBCTZXPaj1XNVBYhidzdoRoIo7Oy2av2M3WY+vYDjXDLCucmyF9yLaC0+nIW3UFwsR43qQ5mRtw5vWuk2OB0KqGh6S9KvJ8Gogdwba9R2AbNAt+081JPxOa8ll12RNk7tY8ruGFIlxwpKoixpWjrXxDCyDVVMgRfAhS2iIv19luiyenvjcAQ==",
			"name" : "application-52ae6900f4e9eeed250000d3",
			"type" : "ssh-rsa"
		}
	]


After datastore migration:
{
	"_id" : ObjectId("52ae6900f4e9eeed250000d3"),
	"app_ssh_keys" : [
		{
			"_id" : ObjectId("52ae6959f4e9eeed25000111"),
			"_type" : "ApplicationSshKey",
			"component_id" : ObjectId("52ae6900f4e9eeed250000d4"),
			"content" : "AAAAB3NzaC1yc2EAAAABIwAAAQEAzTllYT4wwnyBbT5nyjk2Ekyy/jye6dz1yTbH3iGFAOFR7UR+jujucpu0VeZPj9qOlVIxQrUucBEWUPjRln+78csaen7JZaA4Cc8WnDuchRPZ97gXg+cCdiLFMyf6lffX9pHIIy2wJbs1tNVGaX4kVB8nZIRv9/y6CauM+SQxciU8jGeKcbX54VOSir53kJvlgzs4Ordq0kGDBusMZO7J2rJp/qh1L4suuKAmqNTYhixlSM982W07vTJ6lDam/6vWl8dfyFEfVFUZyRNBO/qY8g2vIJYbb1YoLEvw7s3lf6C1d1rlVTPLvGaYmC07HRvza4jTvnFa3v4vrWFOCGDdZQ==",
			"name" : "application-323074577224474952728576",
			"type" : "ssh-rsa"
		},
		{
			"_id" : ObjectId("52ae6959f4e9eeed25000112"),
			"_type" : "ApplicationSshKey",
			"component_id" : ObjectId("52ae6900f4e9eeed250000d3"),
			"content" : "AAAAB3NzaC1yc2EAAAABIwAAAQEAuANGwPsYI4K2y+EC8GCuLwhP03pOTE8qMU3UxgcZbNX7qzLcBFfhNhdZHHLohqnEG/OKBf8uYruzJLptle70iRCqwi8yqk29bgcspY60FhAtm3eBFw1atuYFkronTKygS33Wp5BsIbyuZlschXXwkxtmRam7N0KUVZQBCTZXPaj1XNVBYhidzdoRoIo7Oy2av2M3WY+vYDjXDLCucmyF9yLaC0+nIW3UFwsR43qQ5mRtw5vWuk2OB0KqGh6S9KvJ8Gogdwba9R2AbNAt+081JPxOa8ll12RNk7tY8ruGFIlxwpKoixpWjrXxDCyDVVMgRfAhS2iIv19luiyenvjcAQ==",
			"name" : "application-52ae6900f4e9eeed250000d3",
			"type" : "ssh-rsa"
		},
		{
			"_id" : ObjectId("52ae9074f4e9eee63d00001e"),
			"_type" : "ApplicationSshKey"
		}
	]
}


Noticed that the 3rd element in the array has only _id and _type attributes.

Expected results:
The meaningless elements should not be appended to the app_ssh_keys array.

Additional info:

Comment 1 Jianwei Hou 2013-12-16 08:59:31 UTC
The same problem is also happening to the cloud_users collection. The new ssh keys are added with only _id and _type, when the end user tries to add new ssh keys, an exception will be caught

"ssh_keys": [
    {
      "_id": ObjectId("52ae66c9f4e9ee42f400000a"),
      "_type": "UserSshKey",
      "content": "AAAAB3NzaC1yc2EAAAADAQABAAABAQD238fuS3rklFnfR7l5TBWQ1W+Aae6Kn25OfKismWnYUZ7MYLvl7rSMt1CY869J6M27sDojTjdgMBJrdBW9rZKGT4HIh1nxabzjKKerzsp5B4YaGpUtX7looJIujSwrAUqaPeikbq/aUbxzAi5eA/4QYW1H8Rq0JFLvrPGVgnkdEKPmH+NitgFUHB8xZUCvrDE080kyRNzHXtf7CTbhJcSHXJGhHszCH4Z1dqQa8FRoDEzzlP0Z6gd6fZ3K5XFzspdS6CNFnxRcL1+vcTJsCIBtGYb2OMUohwNFrfsw+2TU3LR3HZ0Kgr8GC7chShH4QKNKKXAp9xHg20H+fJnlTeTV",
      "name": "default",
      "type": "ssh-rsa"
    },
    {
      "_id": ObjectId("52aeb1c3f4e9eeb5a1000074"),
      "_type": "UserSshKey"
    },
    {
      "_id": ObjectId("52aeb1c3f4e9eeb5a1000075"),
      "_type": "UserSshKey"
    }
  ],

Comment 2 Abhishek Gupta 2013-12-16 23:13:40 UTC
Added a fix in --> fork_ami_bug_1039151_1041628_968

If this issue is still unresolved, please attach the mongodump _before_ migration and/or preserve your devenv so that I can take a look there itself.

Comment 3 Jianwei Hou 2013-12-17 07:04:50 UTC
Verified on fork_ami_bug_1039151_1041628_968, this issue is resolved after datastore migration