Description of problem: Deleting a ContentViewVersion does not remove the data from the following directories: - /var/lib/pulp/published/yum/http/repos/ - /var/lib/pulp/published/yum/https/repos/ - /var/lib/pulp/published/yum/master/ See the output below where the Version 4.0 and 5.0 are deleted. Afterwards the directories for those versions are still available: [crash] root@li-lc-1578:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml content-view info --id=13 | grep -A100 '^Versions' Versions: 1) ID: 47 Version: 3.0 Published: 2015/09/21 09:37:21 2) ID: 90 Version: 4.0 Published: 2015/09/23 12:23:47 3) ID: 91 Version: 5.0 Published: 2015/09/23 12:28:41 4) ID: 92 Version: 6.0 Published: 2015/09/23 12:30:40 Components: Activation Keys: [crash] root@li-lc-1578:~# find /var/lib/pulp/published -regextype posix-extended -regex '.*product-HOIPRODUCTS-1_1_0-ci.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0 /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0 /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0 /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0 /var/lib/pulp/published/yum/http/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0 /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0 /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0 /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0 /var/lib/pulp/published/yum/https/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-4_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-5_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-4_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-5_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci [crash] root@li-lc-1578:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml content-view version delete --organization-id=3 --content-view-id=13 --version=4 [...........................................................................................................................................................................] [100%] [crash] root@li-lc-1578:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml content-view version delete --organization-id=3 --content-view-id=13 --version=5 [...........................................................................................................................................................................] [100%] [crash] root@li-lc-1578:~# find /var/lib/pulp/published -regextype posix-extended -regex '.*product-HOIPRODUCTS-1_1_0-ci.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0 /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0 /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0 /var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0 /var/lib/pulp/published/yum/http/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0 /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0 /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0 /var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0 /var/lib/pulp/published/yum/https/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-4_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-5_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create ContentView AAAAAAAAAA 2. Publish 10x the ContentView AAAAAAAAAA 3. List ContentView Versions of AAAAAAAAAA using Hammer 4. Find all directories in Pulp that belong to the Content View AAAAAAAAAA find /var/lib/pulp/published -regextype posix-extended -regex '.*AAAAAAAAAA.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort 5. Delete ContentView Versions 3.0 - 5.0 6. Find all directories in Pulp that belong to the Content View AAAAAAAAAA find /var/lib/pulp/published -regextype posix-extended -regex '.*AAAAAAAAAA.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort Actual results: Still directories containing data of ContentViewVersions 3.0-5.0 are listed Expected results: All data of the deleted contentview versions are deleted from filesystem Additional info: Related to BZ1184442 that only implemented a fix for /var/lib/pulp/nodes
*** Bug 1251786 has been marked as a duplicate of this bug. ***
Please note that it the fix shall be generic to remove data from the filesystem for all reposititories types removed: - ContentViewVersion Repositories - Product Repositories - Lifecycle Repositories (including the default Library) - ContentView removal Next to the Packages also all directory structures created for the published repositories or content views shall be removed. E.g. after removing ContentViewA and ProductA there shall be no files or directories left that refer to either ContentViewA or ProductA. Directories that i am aware of were currently with Sat6.1.5 directory trees are left behind are: /var/lib/pulp/published/yum/https/repos/<org>/Library/<content_view_name> /var/lib/pulp/published/yum/https/repos/<org>/content_views/<content_view_name>
This appears to have been fixed upstream in 2.6.1. https://pulp.plan.io/issues/671 https://github.com/pulp/pulp_rpm/pull/648
The PR648 mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1265703#c6 does not include references that orphaned directories in the following directories are deleted: /var/lib/pulp/published/yum/master /var/lib/pulp/published/yum/master/yum_distributor Are these also being cleaned?
Regarding c#9, no, I don't think those directories are being removed. But, those two directories would only be orphaned if every single yum repository was removed from pulp. Removing them would only be of (very small) value if the user did not intended to publish another yum repository. Is that a use case we're concerned about?
I guess the use case is valid for API usage in the context of a continuous integration environment. @Peter, can you elaborate on that?
I asked for the removal of the sub directories of each repository inside the those two top directories. Those two top directories are part of the base infrastructure. E.g. the description mentions the directories for ContentView 'HOIPRODUCTS-1.1.0-ci' ContentViewVersion 6.0 /var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci /var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci When i delete this ContentViewVersion also this top directory of the ContentViewVersions shall be removed.
FailedQA. @Sat6.2.0 Beta Snap4 pulp-katello-1.0-3.el7sat.noarch pulp-puppet-plugins-2.8.0.1-2.beta.el7sat.noarch pulp-rpm-plugins-2.8.0.1-1.beta.el7sat.noarch pulp-nodes-parent-2.8.0.1-0.7.beta.el7sat.noarch pulp-server-2.8.0.1-0.7.beta.el7sat.noarch pulp-nodes-common-2.8.0.1-0.7.beta.el7sat.noarch pulp-client-1.0-1.noarch pulp-puppet-tools-2.8.0.1-2.beta.el7sat.noarch pulp-selinux-2.8.0.1-0.7.beta.el7sat.noarch pulp-docker-plugins-2.0.0.1-1.beta.el7sat.noarch Used manual reproducer in comment #0: # hammer content-view info --id 3 Warning: Option --id is deprecated. This option will be removed in Katello 2.5, Use --uuid ID: 3 Name: Custom View Label: Custom_View Composite: false Description: Content Host Count: 0 Organization: Default Organization Yum Repositories: 1) ID: 4 Name: custom repo Label: custom_repo Docker Repositories: Puppet Modules: Lifecycle Environments: 1) ID: 1 Name: Library Versions: 1) ID: 3 Version: 1.0 Published: 2016/03/22 09:54:07 2) ID: 4 Version: 2.0 Published: 2016/03/22 09:54:39 3) ID: 5 Version: 3.0 Published: 2016/03/22 10:08:49 4) ID: 6 Version: 4.0 Published: 2016/03/22 10:10:40 Components: Activation Keys: # hammer content-view remove-version --id 3 --content-view-version 4 Warning: Option --id is deprecated. This option will be removed in Katello 2.5, Use --uuid The component version has been removed # hammer content-view remove-version --id 3 --content-view-version 3 Warning: Option --id is deprecated. This option will be removed in Katello 2.5, Use --uuid The component version has been removed # find /var/lib/pulp/published -regextype posix-extended -regex '.*Custom_View.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/1.0 /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/2.0 /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/3.0 /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/4.0 /var/lib/pulp/published/yum/http/repos/Default_Organization/Library/Custom_View /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/1.0 /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/2.0 /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/3.0 /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/4.0 /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/Custom_View /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-1_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-2_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-3_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-4_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Library-Custom_View-Custom-custom_repo /var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-1_0-Custom-custom_repo /var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-2_0-Custom-custom_repo /var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-3_0-Custom-custom_repo /var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-4_0-Custom-custom_repo /var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Library-Custom_View-Custom-custom_repo >>> after deleting "Custom View" versions 3.0 and 4.0, the data are still present on disk
Not necessarily FailedQA It looks like following commands do nothing (say 'has been removed' but hasn't) # hammer content-view remove-version --id 3 --content-view-version '4.0' # hammer content-view remove-version --id 3 --content-view-version-id 6 # hammer content-view info --id 3 ... Versions: 1) ID: 3 Version: 1.0 Published: 2016/03/22 09:54:07 2) ID: 4 Version: 2.0 Published: 2016/03/22 09:54:39 3) ID: 5 Version: 3.0 Published: 2016/03/22 10:08:49 4) ID: 6 Version: 4.0 Published: 2016/03/22 10:10:40 ...
At webUI by ticking Completely remove version? I was able to completely remove CV 3.0 and 4.0 versions and then... # find /var/lib/pulp/published -regextype posix-extended -regex '.*Custom_View.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/1.0 /var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/2.0 /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/1.0 /var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/2.0 /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-1_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-2_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-3_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Custom_View-4_0-Custom-custom_repo /var/lib/pulp/published/yum/master/Default_Organization-Library-Custom_View-Custom-custom_repo /var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-1_0-Custom-custom_repo /var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-2_0-Custom-custom_repo Under /var/lib/pulp/published/yum/http{,s}/repos/ the data were deleted Only under /var/lib/pulp/published/yum/master/ the data remained Is this ^ still the problem?
This BZ is created to make sure there is no data left at all on the disk that was used only by the content view versions (or any other repository that is deleted). That includes also any directory under /var/lib/pulp/published/yum/master/ If not deleted the files will stay there forever and after many (re)publishing/promoting (composite) content views the disk will suddenly be full. I have in a my scripts a workaround to delete the directories myself, even with that workaround in place this, even with that i still have many directories: [crash] root@li-lc-1017:/var/lib/pulp/published/yum/master# ls -l | wc -l 2407 Without the workaround to delete the directories myself it will be full in a 30-60 days.
FailedQA. With regard to comment #19 switching back to assigned.
*** Bug 1325084 has been marked as a duplicate of this bug. ***
Extremely related to https://bugzilla.redhat.com/show_bug.cgi?id=1317057 these are likely dupes
*** This bug has been marked as a duplicate of bug 1317057 ***
Sorry, the duplicate bug https://bugzilla.redhat.com/show_bug.cgi?id=1317057 is not public accessible. This makes it impossible for me to follow the progress. Please make https://bugzilla.redhat.com/show_bug.cgi?id=1317057 public.
I'm tracking this exact same problem on my Satellite 6.1.9 installation and was referred to these bugzilla reports from my RedHat Support Case (CASE 01650913). Like Peter, I can't view the the other ticket (1317057) either, so I can't see if there is any progress being made or if it's just being ignored and/or put off until 6.2.0 GA. What does it take to get access to the other ticket? And if it's going to wait for 6.2.0, is there a workaround that can be used to clean up space? Maybe a script that could be run from a cronjob that would look for invalid content views and remove the content? Thank you.
Peter & Ken - I have made https://bugzilla.redhat.com/show_bug.cgi?id=1317057 public. Please let me know if you need any additional information.