Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1970037

Summary: [4.9.0] Garbage collector not cleaning up directories of removed clusters
Product: OpenShift Container Platform Reporter: Ronnie Lazar <alazar>
Component: assisted-installerAssignee: Sarah Lavie <slavie>
assisted-installer sub component: Deployment Operator QA Contact: bjacot
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: low CC: aos-bugs, mfilanov, trwest, yobshans
Version: 4.8Keywords: Triaged
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AI-Team-Hive
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1968451 Environment:
Last Closed: 2021-12-11 13:51:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1967773, 1968451    
Bug Blocks:    

Description Ronnie Lazar 2021-06-09 16:56:48 UTC 
+++ This bug was initially created as a clone of Bug #1968451 +++

+++ This bug was initially created as a clone of Bug #1967773 +++

Description of problem:

After deleting InfraEnv, AgentClusterInstall, and ClusterDeployment for a given cluster, the garbage collector is removing the ISO and discovery.ign files. However, it is leaving behind the directory that contained the discovery.ign file:

# ls -ld /mnt/pv0/d076038d-bb28-4423-a3b3-955764c2853b/
drwxr-xr-x. 2 1000640000 nobody 6 Jun  3 16:23 /mnt/pv0/d076038d-bb28-4423-a3b3-955764c2853b/

How reproducible:
100%

Steps to Reproduce:
1. Create ClusterDeployment, AgentClusterInstall, InfraEnv
2. Delete all resources

Actual results:

ISO and discovery.ign deleted but directory that contained discovery.ign still exists.


Expected results:
All files and directories related to cluster are deleted


Additional info:

Without removing these directories, the filesystem becomes very cluttered as you can see:

# ls -l /mnt/pv0
total 11748280
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 13:23 0ff5537a-3ebb-49de-871d-8c38e201b36c
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 14:23 4935b89b-6d63-444d-bcb0-9b200167c157
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 14:23 85c9a8d2-0634-443b-948d-5e81aeff0629
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 13:23 8debd9d7-815c-49b7-8bfb-5b14343f093b
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 16:23 a2de0d21-fec6-422b-b81f-fd2f947311d9
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 13:23 bf6022ed-f4cd-43d9-a64d-5a769946ad9b
drwx------. 2 1000640000 nobody         59 Jun  2 15:18 cache
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 14:23 cc7560a5-ed53-4857-a99d-ccd19052c0ba
drwxr-xr-x. 2 1000640000 nobody         27 Jun  3 14:19 cc9cd729-9b42-4b4c-853e-dc2489aaa2a3
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 16:23 d076038d-bb28-4423-a3b3-955764c2853b
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 13:23 d5c3d097-85f1-4521-98f6-d53ec4b59850
-rw-------. 1 1000640000 nobody  109111296 Jun  3 16:39 discovery-image-cc9cd729-9b42-4b4c-853e-dc2489aaa2a3.iso
-rw-------. 1 1000640000 nobody  109111296 Jun  3 16:39 discovery-image-f321b521-2941-4628-9b77-992a313df81c.iso
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 13:23 e4519e0a-3af3-4db7-b9bb-12cdca136516
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 16:23 e5a7879d-4042-498f-8601-54650bdae69a
drwxr-xr-x. 2 1000640000 nobody          6 Jun  3 14:23 e76fb548-327a-40ad-bae9-e8cc034630bc
drwxr-xr-x. 2 1000640000 nobody         27 Jun  3 14:08 f321b521-2941-4628-9b77-992a313df81c
drwxr-xr-x. 2 1000640000 nobody          6 Jun  2 16:23 f3b96b2d-c333-4881-8e6e-61d162a6bbe8
drwx------. 5 1000640000 nobody         47 Jun  2 15:24 isoutil048719477
drwx------. 5 1000640000 nobody         47 Jun  1 17:09 isoutil210420162
drwx------. 5 1000640000 nobody         47 Jun  2 15:24 isoutil470354621
-rw-------. 1 1000640000 nobody         13 Jun  2 15:24 minimal_templates_version.json
-rw-------. 1 1000640000 nobody  918552576 Jun  2 15:24 rhcos-46.82.202012051820-0.iso
-rw-------. 1 1000640000 nobody   98398208 Jun  2 15:24 rhcos-46.82.202012051820-0-minimal.iso
-rw-------. 1 1000640000 nobody  957349888 Jun  2 15:24 rhcos-47.83.202103251640-0.iso
-rw-------. 1 1000640000 nobody  105809920 Jun  2 15:24 rhcos-47.83.202103251640-0-minimal.iso
-rw-------. 1 1000640000 nobody 1032847360 Jun  1 17:08 rhcos-48.84.202105062123-0.iso
-rw-------. 1 1000640000 nobody  109111296 Jun  1 17:09 rhcos-48.84.202105062123-0-minimal.iso
Comment 2 Trey West 2021-11-05 15:20:39 UTC 
@masayag

Tested this on 2.4.0-DOWNSTREAM-2021-11-02-15-58-38. It looks like directories are not being removed after the cluster resources are deleted. This is coming from the assisted-service log:

time="2021-11-05T15:15:19Z" level=warning msg="Failed deleting cluster directory b571aa67-7dac-4787-93ec-6cefe7510306" func=github.com/openshift/assisted-service/internal/cluster.Manager.PermanentClustersDeletion file="/remote-source/assisted-service/app/internal/cluster/cluster.go:1096" error="Failed to delete file /data/b571aa67-7dac-4787-93ec-6cefe7510306: remove /data/b571aa67-7dac-4787-93ec-6cefe7510306: directory not empty" pkg=cluster-stat

The directory only contains other empty directories. This is very low priority and I don't think it needs to be addressed immediately. We can push the fix to next release if that is okay.
Comment 3 Sarah Lavie 2021-12-11 13:50:17 UTC 
solved by https://bugzilla.redhat.com/show_bug.cgi?id=1967773
Comment 4 Sarah Lavie 2021-12-11 13:51:12 UTC 

*** This bug has been marked as a duplicate of bug 1967773 ***