Bug 1923882

Summary:	Deleting volume backup does not clear the corresponding directories when cinder-backp backend is NFS.
Product:	Red Hat OpenStack	Reporter:	Rohini Diwakar <rdiwakar>
Component:	openstack-cinder	Assignee:	Rajat Dhasmana <rdhasman>
Status:	ON_DEV ---	QA Contact:	Evelina Shames <eshames>
Severity:	medium	Docs Contact:	Andy Stillman <astillma>
Priority:	low
Version:	13.0 (Queens)	CC:	alink, eharney, jobernar, ltoscano, rdhasman
Target Milestone:	z2	Keywords:	Triaged, ZStream
Target Release:	17.1	Flags:	ifrangs: needinfo? (rdhasman)
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rohini Diwakar 2021-02-02 06:51:55 UTC

Description of problem:
Using cinder backup with NFS backend, everything works fine, except that when we delete a backup the actual "content" of the backup gets deleted but not the directory structure on the NFS share.

(overcloud) [stack@undercloud-0 ~]$ openstack volume backup list --long --all-projects -c Container -c ID
+--------------------------------------+--------------------------------------------+
| ID | Container |
+--------------------------------------+--------------------------------------------+
| 17aed1d1-33ba-466e-a51f-d6af156d120e | 17/ae/17aed1d1-33ba-466e-a51f-d6af156d120e |
+--------------------------------------+--------------------------------------------+

[backups]# ll
total 0
drwxr-xr-x. 3 root root 16 Feb 2 06:28 02
drwxr-xr-x. 3 root root 16 Feb 2 06:33 17
drwxr-xr-x. 3 root root 16 Feb 2 06:19 5a
drwxr-xr-x. 3 root root 16 Feb 2 06:20 e4
drwxr-xr-x. 3 root root 16 Feb 2 06:32 e6

The only available volume backup is 17aed1d1-33ba-466e-a51f-d6af156d120e and the rest of the backups were deleted yet their corresponding directories are still present in the backend. There is no content in the deleted volume backup directories.

Version-Release number of selected component (if applicable):
RHOSP13

How reproducible:

Steps to Reproduce:
1. Deploy RHOSP13 with cinder backend as NFS and configure cinder-backup service.
2.
3.

Actual results:
Directories for deleted volume backup are not deleted.

Expected results:
Directories for deleted volume backup should also be deleted.

Additional info:

Comment 2 Sofia Enriquez 2021-02-08 15:48:42 UTC

Greetings Rohini Diwakar,

This is the expected behaviour.

This happens because the backup service might reuse some sub-folders. Backup data and metadata will
be written to the repository at unique paths that are a function of the backup ID. That path will
be stored in the backup record service_metatdata so that it can be used to navigate to the backup
data and metadata for restore and delete operations (thanks Tzach for the detailed explanation).

Comment 5 Luigi Toscano 2021-02-08 16:55:08 UTC

Just another point in order to assess this fix: apart from the wasted inodes, does having the leftover empty directories around lead to any other issue?

Comment 7 Tzach Shefi 2021-02-10 09:29:06 UTC

Luigi, 

Even if not on this specific case, left unchecked these empty folders may potentially cause problems, 
depending on NFS server/storage system's limits, it could affect other deployments/releases too.
Reaching limits may cause: slow to no access, failure to backup/restore just when you need it most.

Agreed low priority for sure, as we need a whole lot of empty folders before things may fail.
And yep 13 EOL-ing soon means it might not be a good place to start fixing this.