Description of problem: Customer reported duplicated entries in cinder DB quota_usages tables for the same project. On RHOSP10 They noticed a 500GB usage while no volumes were present in the project. When creating VMs through heat templates They hit an overquota error. Version-Release number of selected component (if applicable): RHOSP10 openstack-cinder-9.1.4-33 $grep quota cinder.conf # Whether snapshots count against gigabyte quota (boolean value) #no_snapshot_gb_quota = false #quota_volumes = 10 #quota_snapshots = 10 #quota_consistencygroups = 10 #quota_groups = 10 #quota_gigabytes = 1000 #quota_backups = 10 #quota_backup_gigabytes = 1000 # Default driver to use for quota checks (string value) #quota_driver = cinder.quota.DbQuotaDriver # Enables or disables use of default quota class with default quota. (boolean #use_default_quota_class = true Actual results: The specific error Customer got was: “2018-09-11 14:47:59Z [scscf_hmain_da_cli]: CREATE_FAILED Resource CREATE failed: OverLimit: resources.SC-2_cinder_volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 250G, quota is 528G and 500G has been consumed. (HTTP 413) (Request-ID: req-d8e1acd3- ….” They suspect there was a problem within the quota_usages: MariaDB [cinder]> select * from quota_usages where project_id='39cce92b28ae4ea3a2b6c89e36f84843'; +---------------------+---------------------+------------+---------+-----+----------------------------------+----------------------------+--------+----------+---------------+ | created_at | updated_at | deleted_at | deleted | id | project_id | resource | in_use | reserved | until_refresh | +---------------------+---------------------+------------+---------+-----+----------------------------------+----------------------------+--------+----------+---------------+ | 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL | 0 | 515 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes | 0 | 250 | NULL | | 2018-09-11 12:53:55 | 2018-09-11 15:25:17 | NULL | 0 | 518 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes | -6 | 250 | NULL | | 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL | 0 | 521 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes_scaleio-totp-ssd | 0 | 1 | NULL | | 2018-09-11 12:53:56 | 2018-09-11 15:25:17 | NULL | 0 | 524 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes_scaleio-totp-ssd | 0 | 1 | NULL | | 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL | 0 | 527 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes | 0 | 1 | NULL | | 2018-09-11 12:53:56 | 2018-09-11 15:25:17 | NULL | 0 | 530 | 39cce92b28ae4ea3a2b6c89e36f84843 | volumes | 0 | 1 | NULL | | 2018-09-11 12:53:56 | 2018-09-11 12:53:56 | NULL | 0 | 533 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes_scaleio-totp-ssd | 0 | 250 | NULL | | 2018-09-11 12:53:56 | 2018-09-11 15:25:17 | NULL | 0 | 536 | 39cce92b28ae4ea3a2b6c89e36f84843 | gigabytes_scaleio-totp-ssd | -6 | 250 | NULL | +---------------------+---------------------+------------+---------+-----+----------------------------------+----------------------------+--------+----------+---------------+ Expected results: Is it correct to have two entries for the same resource in the same project? In which cases a specific resource in that table is marked as reserved? Customer noticed that for all other projects that is not the case. Additional info:
Jon, please take a look at this one.
*** Bug 1653540 has been marked as a duplicate of this bug. ***
After analyzing a database dump from a customer, I've come to the conclusion that the first entry we encounter in the duplicated quota_usages entry is always wrong and the second one always the one we keep on updating and using. I wrote this little code that takes care of deleting the first entry we see for each of those entries. ~~~ for p in $(mysql -D cinder -e 'select count(resource),project_id from quota_usages group by project_id,resource order by count(resource);' | awk '{ if ($1 > 1) print $2 }' | sort | uniq | grep -v project_id); do inc=0 echo $p while read -r line do if [ ! -z "$line" ]; then inc=$(( $inc + 1 )) if [ $(( $inc % 2 )) -eq 1 ]; then echo $inc $line usage_id=$( echo $line | awk '{ print $1 }' ) mysql -D cinder -e "delete from reservations where usage_id = $usage_id" mysql -D cinder -e "delete from quota_usages where id = $usage_id" else echo Skipping $line fi else echo "Nothing to do!" fi done <<< $( mysql -D cinder -e " select q.id from quota_usages as q where project_id like '$p' order by resource,id; " | grep -v id ) done ~~~