Bug 1061625

Description of problem:

I have no instances running, or stopped or suspended

---
(env)[root@osdashboard osdashboard(keystone_iwienand)]$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
---

but my quota looks like I have a vm running

---
(env)[root@osdashboard osdashboard(keystone_iwienand)]$ nova absolute-limits
+-------------------------+-------+
| Name                    | Value |
+-------------------------+-------+
| maxServerMeta           | 128   |
| maxPersonality          | 5     |
| maxImageMeta            | 128   |
| maxPersonalitySize      | 10240 |
| maxTotalRAMSize         | 51200 |
| maxSecurityGroupRules   | 20    |
| maxTotalKeypairs        | 100   |
| totalRAMUsed            | 4096  |
| maxSecurityGroups       | 10    |
| totalFloatingIpsUsed    | 0     |
| totalInstancesUsed      | 1     |
| totalSecurityGroupsUsed | 0     |
| maxTotalFloatingIps     | 10    |
| maxTotalInstances       | 10    |
| totalCoresUsed          | 2     |
| maxTotalCores           | 20    |
+-------------------------+-------+
---

(note totalInstancesUsed, totalCoresUsed) etc.

I did a little bit of digging around to try an understand where this result is coming from.  Tracing the database queries I think nova just keeps this info in quota_usages, at least it matches up

---
mysql> select resource, in_use from quota_usages where project_id ='bd6159c7f8aa4baba9a33f0b2725e6d1';
+-----------+--------+
| resource  | in_use |
+-----------+--------+
| instances |      1 |
| ram       |   4096 |
| cores     |      2 |
+-----------+--------+
3 rows in set (0.01 sec)
---

Has that table gotten out of sync?  How would I figure out the point that it became inconsistent?

While I was digging, I noticed a few other things that are maybe weird, maybe not.  I would have thought that with no vm's running, my reservations would equal out but they don't

---
mysql> select sum(delta) from reservations where project_id = 'bd6159c7f8aa4baba9a33f0b2725e6d1' and resource ='cores' ; 
+------------+
| sum(delta) |
+------------+
|        -12 |
+------------+
---

I also checked the instance table to see if anything was running, and got back the values in the text-file attachment

Despite a fair bit of googling, it's not clear to me how to debug this issue further.

Version-Release number of selected component (if applicable):

[root@host02 ~]# rpm -qa | grep openstack
openstack-nova-cert-2013.2.1-2.el6ost.noarch
openstack-neutron-2013.2.1-4el6ost.local1.noarch
openstack-nova-novncproxy-2013.2.1-2.el6ost.noarch
redhat-access-plugin-openstack-4.0.0-0.el6ost.noarch
openstack-ceilometer-central-2013.2.1-2.el6ost.noarch
openstack-heat-api-2013.2.1-4.el6ost.noarch
openstack-nova-scheduler-2013.2.1-2.el6ost.noarch
openstack-nova-common-2013.2.1-2.el6ost.noarch
openstack-nova-api-2013.2.1-2.el6ost.noarch
openstack-dashboard-2013.2.1-1.el6ost.noarch
openstack-ceilometer-common-2013.2.1-2.el6ost.noarch
openstack-keystone-2013.2.1-1.el6ost.noarch
openstack-nova-console-2013.2.1-2.el6ost.noarch
openstack-glance-2013.2.1-3.el6ost.noarch
openstack-ceilometer-api-2013.2.1-2.el6ost.noarch
openstack-heat-engine-2013.2.1-4.el6ost.noarch
openstack-nova-conductor-2013.2.1-2.el6ost.noarch
openstack-neutron-openvswitch-2013.2.1-4el6ost.local1.noarch
openstack-cinder-2013.2.1-5.el6ost.noarch
openstack-utils-2013.2-3.el6ost.noarch
openstack-dashboard-theme-2013.2.1-1.el6ost.noarch
python-django-openstack-auth-1.1.2-1.el6ost.noarch
openstack-ceilometer-collector-2013.2.1-2.el6ost.noarch
openstack-heat-common-2013.2.1-4.el6ost.noarch