Bug 1269297

Summary: Table quota_usages gets out of sync with reality
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED WONTFIX QA Contact: nlevinki <nlevinki>
Severity: high Docs Contact:
Priority: high    
Version: 5.0 (RHEL 6)CC: berrange, bhoefer, dasmith, dhill, dyocum, eglynn, kchamart, lyarwood, ndipanov, pbrady, sbauza, sferdjao, sgordon, skinjo, vromanso, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 5.0 (RHEL 6)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-02 14:34:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Hill 2015-10-06 22:19:39 UTC 
Description of problem:
Users are getting this error when they try to launch new VMs:

Error: Quota exceeded for ram: Requested 8192, but already used 244992 of 245760 ram (HTTP 413) (Request-ID: req-0beefdf0-cae5-4f76-986a-7caf1d1e0b45)

However, the actual amount of ram used by the tenant is 182GB, not 245GB.   

Version-Release number of selected component (if applicable):


How reproducible:
Intermittent

Steps to Reproduce:
1. Create/destroy VMs , get issues while creating/deleting VMs
2. Try to create a new VM
3.

Actual results:
Quota exceeded

Expected results:
Quota should always be in sync

Additional info:
Many people worked on a quota cleanup script that is attached to the case attached to this BZ .  Would it be possible to have such a cleanup script officialized or perhaps figuring out the root cause of this issue.
Comment 5 Lee Yarwood 2015-11-02 14:34:24 UTC 
Closing this bug as WONTFIX given the documented workarounds and significant effort required to rework this complex area of Nova's code base. Once plans for this work are confirmed upstream we should open fresh bugs to track this work into RHEL OSP.
Comment 8 Dan Yocum 2015-11-13 16:27:42 UTC 
Lee - Does this look like a sufficient tl;dr summary:

1. Enable periodic quota usage resyncs by Nova

# openstack-config --set /etc/nova/nova.conf DEFAULT max_age 86400
# openstack-config --set /etc/nova/nova.conf DEFAULT until_refresh 5
# mysql nova -e "update nova.quota_usages set until_refresh = 5 where deleted = 0 and until_refresh is null;"


2. Force Nova to refresh quota usage during a given users next reservation.

# mysql nova -e "update quota_usages set in_use=-1 where resource = 'instances' or resource = 'cores' or resource = 'ram' or resource = 'fixed_ips';"


3. Update the in_use values for resources in the quota_usages table.

# wget https://raw.githubusercontent.com/cernops/nova-quota-sync/master/nova-quota-sync
# python nova-quota-sync --all
Comment 9 Lee Yarwood 2015-11-14 20:40:30 UTC 
Dan, as discussed 1 and 2 looks fine and should negate the need for 3.
Comment 10 Shinobu KINJO 2015-11-16 00:23:15 UTC 
Thank you, both for working on it.
Let me double-check about #2

This query:

"update quota_usages set in_use=-1 where resource = 'instances' or resource = 'cores' or resource = 'ram' or resource = 'fixed_ips';"

should be:

"update nova.quota_usages set in_use=-1 where resource = 'instances' or resource = 'cores' or resource = 'ram' or resource = 'fixed_ips';"

What do you think?
Comment 11 Lee Yarwood 2015-11-16 09:57:42 UTC 
(In reply to Shinobu KINJO from comment #10)
> Thank you, both for working on it.
> Let me double-check about #2
> 
> This query:
> 
> "update quota_usages set in_use=-1 where resource = 'instances' or resource
> = 'cores' or resource = 'ram' or resource = 'fixed_ips';"
> 
> should be:
> 
> "update nova.quota_usages set in_use=-1 where resource = 'instances' or
> resource = 'cores' or resource = 'ram' or resource = 'fixed_ips';"
> 
> What do you think?

That isn't required if you are using `mysql nova -e $QUERY`.