Bug 998467 - "oo-admin-chk" fails to detect storage mismatch info between applications collection and usage_records collection
"oo-admin-chk" fails to detect storage mismatch info between applications col...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Ravi Sankar
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-19 07:33 EDT by Qiushui Zhang
Modified: 2015-05-14 20:20 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-19 12:47:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Qiushui Zhang 2013-08-19 07:33:02 EDT
Description of problem:
Create storage mismatch for an app between mongo and actual used storage. Use "oo-admin-chk -l 1" to check the problem. It returns no error.

Version-Release number of selected component (if applicable):
devenv_3672

How reproducible:
always

Steps to Reproduce:
1. rhc app create php1 php-5.3
2. On instance, 
oo-admin-ctl-user -l qiuzhang@redhat.com --setmaxtrackedstorage 10
oo-admin-ctl-user -l qiuzhang@redhat.com --setmaxuntrackedstorage 6 
3. rhc cartridge storage php-5.3 -a php1 --add 10
4. On instance, mongo openshift_broker_dev
libra_rs:PRIMARY> db.applications.update({name:"php1"}, {$set: {"group_overrides.0.additional_filesystem_gb": 0}})
5. On instance, oo-admin-chk -l 1

Actual results:
It ends up successfully, with no error prompts for storage mismatch.

Expected results:
Storage usage mismatch found.

Additional info:

libra_rs:PRIMARY> db.applications.update({name:"rubyapp"}, {$set: {"group_overrides.0.additional_filesystem_gb": 0}})
libra_rs:PRIMARY> db.applications.findOne()
{
	"_id" : ObjectId("5212017b055da80661000087"),
	"analytics" : {
		"user_agent" : "rhc/1.13.1 (ruby 1.8.7; x86_64-linux) (2.3.3, ruby 1.8.7 (2011-06-30))"
	},
	"builder_id" : null,
	"canonical_name" : "rubyapp",
	"component_configure_order" : [ ],
	"component_instances" : [
		{
			"_id" : ObjectId("5212017b055da8066100009a"),
			"cartridge_vendor" : "redhat",
			"version" : "1.9",
			"component_properties" : {
				
			},
			"cartridge_name" : "ruby-1.9",
			"component_name" : "ruby-1.9",
			"group_instance_id" : ObjectId("5212017b055da80661000092")
		}
	],
	"component_start_order" : [ ],
	"component_stop_order" : [ ],
	"created_at" : ISODate("2013-08-19T11:28:59.196Z"),
	"default_gear_size" : "small",
	"domain_id" : ObjectId("5211eb60055da80661000006"),
	"domain_namespace" : "qiuzhang",
	"domain_requires" : [ ],
	"downloaded_cart_map" : {
		
	},
	"group_instances" : [
		{
			"_id" : ObjectId("5212017b055da80661000092"),
			"gears" : [
				{
					"_id" : ObjectId("5212017b055da80661000087"),
					"app_dns" : true,
					"host_singletons" : true,
					"name" : "rubyapp",
					"quarantined" : false,
					"server_identity" : "ip-10-84-222-195",
					"sparse_carts" : [ ],
					"uid" : null,
					"uuid" : "5212017b055da80abb000002"
				}
			]
		}
	],
	"group_overrides" : [
		{
			"additional_filesystem_gb" : 0,
			"components" : [
				{
					"comp" : "ruby-1.9",
					"cart" : "ruby-1.9"
				}
			],
			"max_gears" : 1,
			"min_gears" : 1
		}
	],
	"init_git_url" : null,
	"members" : [
		{
			"_id" : ObjectId("5211eb16055da80661000001"),
			"_type" : null,
			"n" : "qiuzhang@redhat.com",
			"r" : "admin",
			"f" : [
				[
					"domain",
					"admin"
				]
			],
			"e" : null
		}
	],
	"name" : "rubyapp",
	"owner_id" : ObjectId("5211eb16055da80661000001"),
	"pending_op_groups" : [ ],
	"scalable" : false,
	"updated_at" : ISODate("2013-08-19T11:30:01.201Z"),
	"uuid" : "5212017b055da80abb000002"
}
libra_rs:PRIMARY> db.usage_records.find().pretty()
{
	"_id" : ObjectId("52120180055da8066100009c"),
	"event" : "begin",
	"time" : ISODate("2013-08-19T11:29:04.673Z"),
	"created_at" : ISODate("2013-08-19T11:29:04.673Z"),
	"gear_id" : ObjectId("5212017b055da80661000087"),
	"usage_type" : "GEAR_USAGE",
	"user_id" : ObjectId("5211eb16055da80661000001"),
	"app_name" : "rubyapp",
	"gear_size" : "small",
	"updated_at" : ISODate("2013-08-19T11:29:04.680Z")
}
{
	"_id" : ObjectId("521201b9055da806610000a3"),
	"event" : "begin",
	"time" : ISODate("2013-08-19T11:30:01.464Z"),
	"created_at" : ISODate("2013-08-19T11:30:01.464Z"),
	"gear_id" : ObjectId("5212017b055da80661000087"),
	"usage_type" : "ADDTL_FS_GB",
	"user_id" : ObjectId("5211eb16055da80661000001"),
	"app_name" : "rubyapp",
	"addtl_fs_gb" : 4,
	"updated_at" : ISODate("2013-08-19T11:30:01.473Z")
}
Comment 1 Abhishek Gupta 2013-08-19 17:09:10 EDT
This is an invalid test case. The oo-admin-chk script does not check if the storage quota on the gear is the same as the one recorded in mongo for the gear. 

The oo-admin-chk script tests to check - 
 - if the additional gear storage exceeds the user capabilities
 - if there is any mismatch in usage records for storage vs the gear storage recorded in mongo


Assigning to ON_QA to update their test cases
Comment 3 Jianwei Hou 2013-08-20 07:39:09 EDT
(In reply to Abhishek Gupta from comment #1)
> This is an invalid test case. The oo-admin-chk script does not check if the
> storage quota on the gear is the same as the one recorded in mongo for the
> gear. 
> 
> The oo-admin-chk script tests to check - 
>  - if the additional gear storage exceeds the user capabilities
>  - if there is any mismatch in usage records for storage vs the gear storage
> recorded in mongo
> 
> 
> Assigning to ON_QA to update their test cases

In the description of the bug

The additional gear storage in applications collection is updated to 0
"group_overrides" : [
		{
			"additional_filesystem_gb" : 0,
.....

While in usage_records collection, the additional gear storage remains 4, and they mismatch
{
	"_id" : ObjectId("521201b9055da806610000a3"),
	"event" : "begin",
	"time" : ISODate("2013-08-19T11:30:01.464Z"),
	"created_at" : ISODate("2013-08-19T11:30:01.464Z"),
	"gear_id" : ObjectId("5212017b055da80661000087"),
	"usage_type" : "ADDTL_FS_GB",
	"user_id" : ObjectId("5211eb16055da80661000001"),
	"app_name" : "rubyapp",
	"addtl_fs_gb" : 4,
	"updated_at" : ISODate("2013-08-19T11:30:01.473Z")
}


I think this is what this bug tries to stress, oo-admin-chk at level 1 did not detect this issue.
Comment 4 Abhishek Gupta 2013-08-20 13:11:05 EDT
The script checks for gears with additional storage in applications collection but not in usage_records and viceversa. The actual value of the additional storage is not verified. 

Lowering the severity as this is a new check that needs to be added.
Comment 5 openshift-github-bot 2013-09-12 22:51:46 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/076a48678a989aa244c980e726aab2cbfd818ef4
Bug 998467 - Add addtional storage mismatch checks in oo-admin-chk
Comment 6 Qiushui Zhang 2013-09-16 06:00:38 EDT
Tested on dev_3785.

[root@ip-10-178-39-204 ~]# oo-admin-chk -l 1
Started at: 2013-09-16 05:57:52 -0400
Time to fetch mongo data: 0.035s
Total gears found in mongo: 1
Time to get all gears from nodes: 20.538s
Total gears found on the nodes: 1
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.062s
Total gears found on the nodes: 1
Total nodes that responded : 1
Check failed.
Found usage record for addtl storage with gear Id '5236d49d605104293a000001' but could not find corresponding gear with addtl storage in the application.
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 40.681s
Finished at: 2013-09-16 05:58:32 -0400

The additional storage check is added.

Mark the bug as verified.

Note You need to log in before you can comment on or make changes to this bug.