Bug 1028337 - oo-admin-chk failed with gear that has addition storage added
oo-admin-chk failed with gear that has addition storage added
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Ravi Sankar
libra bugs
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-08 04:03 EST by Jianwei Hou
Modified: 2015-05-14 20:22 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-23 22:29:04 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jianwei Hou 2013-11-08 04:03:09 EST
Description of problem:
Add additional storage to an application as user with silver plan, and execute oo-admin-chk -l 1 from broker, checked failed with addtl storage not found for corresponding usage record.

Version-Release number of selected component (if applicable):
On devenv_4003

How reproducible:
Always

Steps to Reproduce:
1. Create a user with silver plan
2. Create an application, add 3G storage to it
3. Execute 'oo-admin-chk -l 1'

Actual results:
[root@ip-10-62-121-228 ~]# oo-admin-chk -l 1
Started at: 2013-11-08 03:25:40 -0500
Time to fetch mongo data: 0.035s
Total gears found in mongo: 1
Time to get all gears from nodes: 20.71s
Total gears found on the nodes: 1
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.061s
Total gears found on the nodes: 1
Total nodes that responded : 1
Check failed.
Found addtl storage for gear Id '527c9fb7fbcf4c622d00012c' but could not find corresponding usage record.
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 42.181s
Finished at: 2013-11-08 03:26:22 -0500


Expected results:
Should ignore the setmaxuntrackedstorage to make the checking pass.

Additional info:
Comment 1 Abhishek Gupta 2013-11-08 17:36:17 EST
Fixed with --> https://github.com/openshift/origin-server/pull/4139
Comment 2 Abhishek Gupta 2013-11-08 17:50:25 EST
Updated the wrong bug, ignore comment #1
Comment 3 Ravi Sankar 2013-11-08 18:53:30 EST
Tried couple of times but unable to reproduce the issue on latest devenv.
Please verify once again and let me know exact steps for reproducing the issue.
Comment 4 Jianwei Hou 2013-11-10 21:52:46 EST
Hi, @Ravi: here is the steps to reproduce this issue on devenv_4016

1. Create a scalable php application
rhc create-app php1s php-5.3 -s
2. Upgrade current account to silver plan via restapi
3. Add 3G storage to the app
rhc cartridge-storage --add 3 -c php-5.3 -a php1s
4. On broker run oo-admin-chk
oo-admin-chk -l 1

Result:
[root@ip-10-158-73-211 ~]# oo-admin-chk -l 1
Started at: 2013-11-10 21:43:25 -0500
Time to fetch mongo data: 0.282s
Total gears found in mongo: 13
Time to get all gears from nodes: 21.454s
Total gears found on the nodes: 13
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.049s
Total gears found on the nodes: 13
Total nodes that responded : 1
Check failed.
Found addtl storage for gear Id '528038e74937b87264000186' but could not find corresponding usage record.
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 44.281s
Finished at: 2013-11-10 21:44:09 -0500

Re-check applications and usage_records collections:

libra_rs:PRIMARY> db.applications.findOne({_id:ObjectId("528038e74937b87264000186")},{name:1,members:1,group_overrides:1})
{
	"_id" : ObjectId("528038e74937b87264000186"),
	"group_overrides" : [
		{
			"components" : [
				{
					"cart" : "php-5.3",
					"comp" : "php-5.3"
				},
				{
					"cart" : "haproxy-1.4",
					"comp" : "web_proxy"
				}
			],
			"min_gears" : 1,
			"additional_filesystem_gb" : 3,
			"max_gears" : -1,
			"gear_size" : "small"
		}
	],
	"members" : [
		{
			"_type" : null,
			"n" : "jhou@redhat.com",
			"r" : "admin",
			"f" : [
				[
					"domain",
					"admin"
				]
			],
			"e" : null,
			"_id" : ObjectId("528038d84937b8e9fb000595")
		}
	],
	"name" : "php1s"
}

libra_rs:PRIMARY> db.usage_records.find({app_name:"php1s"}).pretty()
{
	"_id" : ObjectId("528038f04937b872640001a3"),
	"event" : "begin",
	"time" : ISODate("2013-11-11T01:54:56.988Z"),
	"created_at" : ISODate("2013-11-11T01:54:56.988Z"),
	"gear_id" : ObjectId("528038e74937b87264000186"),
	"usage_type" : "GEAR_USAGE",
	"user_id" : ObjectId("528038d84937b8e9fb000595"),
	"app_name" : "php1s",
	"gear_size" : "small",
	"updated_at" : ISODate("2013-11-11T01:54:56.992Z")
}
Comment 5 Jianwei Hou 2013-11-10 22:09:13 EST
Here is my account detail for this testing
{
  "api_version": 1.6,
  "data": {
    "capabilities": {
      "subaccounts": false,
      "gear_sizes": [
        "small",
        "medium",
        "large"
      ],
      "plan_upgrade_enabled": true,
      "private_ssl_certificates": true,
      "max_storage_per_gear": 5
    },
    "consumed_gears": 1,
    "created_at": "2013-11-11T01:54:32Z",
    "id": "528038d84937b8e9fb000595",
    "links": {
      "ADD_KEY": {
        "href": "https://ec2-107-20-86-143.compute-1.amazonaws.com/broker/rest/user/keys",
        "method": "POST",
        "optional_params": [

        ],
        "rel": "Add new SSH key",
        "required_params": [
          {
            "description": "Name of the key",
            "invalid_options": [

            ],
            "name": "name",
            "type": "string",
            "valid_options": [

            ]
          },
          {
            "description": "Type of Key",
            "invalid_options": [

            ],
            "name": "type",
            "type": "string",
            "valid_options": [
              "ssh-rsa",
              "ssh-dss",
              "ssh-rsa-cert-v01@openssh.com",
              "ssh-dss-cert-v01@openssh.com",
              "ssh-rsa-cert-v00@openssh.com",
              "ssh-dss-cert-v00@openssh.com",
              "krb5-principal"
            ]
          },
          {
            "description": "The key portion of an ssh key (excluding key type and comment)",
            "invalid_options": [

            ],
            "name": "content",
            "type": "string",
            "valid_options": [

            ]
          }
        ]
      },
      "LIST_KEYS": {
        "href": "https://ec2-107-20-86-143.compute-1.amazonaws.com/broker/rest/user/keys",
        "method": "GET",
        "optional_params": [

        ],
        "rel": "List SSH keys",
        "required_params": [

        ]
      }
    },
    "login": "jhou@redhat.com",
    "max_domains": 3,
    "max_gears": 16,
    "plan_id": "silver",
    "plan_state": "ACTIVE",
    "usage_account_id": "1701043"
  },
  "messages": [

  ],
  "status": "ok",
  "supported_api_versions": [
    1.0,
    1.1,
    1.2,
    1.3,
    1.4,
    1.5,
    1.6
  ],
  "type": "user",
  "version": "1.6"
}
Comment 6 Ravi Sankar 2013-11-11 15:51:25 EST
Fixed in https://github.com/openshift/origin-server/pull/4160
Comment 7 Jianwei Hou 2013-11-11 21:44:23 EST
Above PR failed to merge, will verify it when the fix is in.
Comment 8 openshift-github-bot 2013-11-12 00:14:38 EST
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/19ca083e21bd4d19e6a124ea0a0a40aa8750a8db
Bug 1028337 - Fix additional storage consistency check in oo-admin-chk
Comment 9 Jianwei Hou 2013-11-13 01:08:11 EST
Verified on devenv_4027, the problem is fixed and oo-admin-chk passed

Note You need to log in before you can comment on or make changes to this bug.