Bug 1628236

Summary: duplicated entries in cinder DB quota_usages tables for the same project
Product: Red Hat OpenStack Reporter: Dariusz Wojewódzki <dwojewod>
Component: openstack-cinderAssignee: Cinder Bugs List <cinder-bugs>
Status: ON_DEV --- QA Contact: Evelina Shames <eshames>
Severity: medium Docs Contact: Kim Nylander <knylande>
Priority: medium    
Version: 10.0 (Newton)CC: abishop, aguetta, brian.rosmaita, cinder-bugs, dhill, eharney, geguileo, gfidente, jobernar, knoha, pgrist, scohen, srevivo
Target Milestone: ---Keywords: Triaged
Target Release: ---Flags: knoha: needinfo? (geguileo)
knoha: needinfo? (geguileo)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dariusz Wojewódzki 2018-09-12 13:54:17 UTC 
Description of problem:

Customer reported duplicated entries in cinder DB quota_usages tables for the same project. On RHOSP10 They noticed a 500GB usage while no volumes were present in the project. 

When creating VMs through heat templates They hit an overquota error.


Version-Release number of selected component (if applicable):

RHOSP10
openstack-cinder-9.1.4-33

$grep quota cinder.conf 
# Whether snapshots count against gigabyte quota (boolean value)
#no_snapshot_gb_quota = false
#quota_volumes = 10
#quota_snapshots = 10
#quota_consistencygroups = 10
#quota_groups = 10
#quota_gigabytes = 1000
#quota_backups = 10
#quota_backup_gigabytes = 1000
# Default driver to use for quota checks (string value)
#quota_driver = cinder.quota.DbQuotaDriver
# Enables or disables use of default quota class with default quota. (boolean
#use_default_quota_class = true




Actual results:

The specific error Customer got was:

“2018-09-11 14:47:59Z [scscf_hmain_da_cli]: CREATE_FAILED Resource CREATE failed: OverLimit: resources.SC-2_cinder_volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 250G, quota is 528G and 500G has been consumed. (HTTP 413) (Request-ID: req-d8e1acd3- ….”

They suspect there was a problem within the quota_usages:
MariaDB [cinder]> select * from quota_usages where project_id='39cce92b28ae4ea3a2b6c89e36f84843'; 
+---------------------+---------------------+------------+---------+-----+----------------------------------+----------------------------+--------+----------+---------------+
| created_at          | updated_at          | deleted_at | deleted | id  | project_id                       | resource                   | in_use | reserved | until_refresh |
+---------------------+---------------------+------------+---------+-----+----------------------------------+----------------------------+--------+----------+---------------+
| 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL       |       0 | 515 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes                  |      0 |      250 |          NULL |
| 2018-09-11 12:53:55 | 2018-09-11 15:25:17 | NULL       |       0 | 518 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes                  |     -6 |      250 |          NULL |
| 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL       |       0 | 521 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes_scaleio-totp-ssd   |      0 |        1 |          NULL |
| 2018-09-11 12:53:56 | 2018-09-11 15:25:17 | NULL       |       0 | 524 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes_scaleio-totp-ssd   |      0 |        1 |          NULL |
| 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL       |       0 | 527 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes                    |      0 |        1 |          NULL |
| 2018-09-11 12:53:56 | 2018-09-11 15:25:17 | NULL       |       0 | 530 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes                    |      0 |        1 |          NULL |
| 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL       |       0 | 533 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes_scaleio-totp-ssd |      0 |      250 |          NULL |
| 2018-09-11 12:53:56 | 2018-09-11 15:25:17 | NULL       |       0 | 536 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes_scaleio-totp-ssd |     -6 |      250 |          NULL |
+---------------------+---------------------+------------+---------+-----+----------------------------------+----------------------------+--------+----------+---------------+

Expected results:

Is it correct to have two entries for the same resource in the same project?

In which cases a specific resource in that table is marked as reserved? 
Customer noticed that for all other projects that is not the case.



Additional info:
Comment 2 Alan Bishop 2018-09-19 19:45:39 UTC 
Jon, please take a look at this one.
Comment 11 Mike Abrams 2019-06-18 14:54:50 UTC 
*** Bug 1653540 has been marked as a duplicate of this bug. ***
Comment 18 David Hill 2019-12-06 15:01:02 UTC 
After analyzing a database dump from a customer, I've come to the conclusion that the first entry we encounter in the duplicated quota_usages entry is always wrong and the second one always the one we keep on updating and using.   I wrote this little code that takes care of deleting the first entry we see for each of those entries.
~~~
for p in $(mysql -D cinder -e 'select count(resource),project_id from quota_usages  group by project_id,resource order by count(resource);' | awk '{ if ($1 > 1) print $2 }' | sort | uniq | grep -v project_id); do
  inc=0
  echo $p
  while read -r line 
  do
    if [ ! -z "$line" ]; then
      inc=$(( $inc + 1 ))
      if [ $(( $inc % 2 )) -eq 1 ]; then
        echo $inc $line
        usage_id=$( echo $line | awk '{ print $1 }'    )
        mysql -D cinder -e "delete from reservations where usage_id = $usage_id"
        mysql -D cinder -e "delete from quota_usages where id = $usage_id"
      else
        echo Skipping $line
      fi
    else
      echo "Nothing to do!"
    fi
  done <<< $( mysql -D cinder -e " select q.id from quota_usages as q where project_id like '$p' order by resource,id; " | grep -v id )
done
~~~