Bug 1265703 - Removing ContentViewVersion does not remove published RPMs in
Removing ContentViewVersion does not remove published RPMs in
Status: CLOSED DUPLICATE of bug 1317057
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
6.1.2
Unspecified Unspecified
unspecified Severity high (vote)
: GA
: --
Assigned To: Christine Fouant
Lukas Pramuk
: Triaged
: 1251786 1325084 (view as bug list)
Depends On:
Blocks: 1122832
  Show dependency treegraph
 
Reported: 2015-09-23 09:53 EDT by Peter Vreman
Modified: 2017-04-25 12:21 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-06-16 16:17:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Peter Vreman 2015-09-23 09:53:30 EDT
Description of problem:
Deleting a ContentViewVersion does not remove the data from the following directories:
- /var/lib/pulp/published/yum/http/repos/
- /var/lib/pulp/published/yum/https/repos/
- /var/lib/pulp/published/yum/master/

See the output below where the Version 4.0 and 5.0 are deleted. Afterwards the directories for those versions are still available:


[crash] root@li-lc-1578:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml content-view info --id=13 | grep -A100 '^Versions'
Versions:
 1) ID:        47
    Version:   3.0
    Published: 2015/09/21 09:37:21
 2) ID:        90
    Version:   4.0
    Published: 2015/09/23 12:23:47
 3) ID:        91
    Version:   5.0
    Published: 2015/09/23 12:28:41
 4) ID:        92
    Version:   6.0
    Published: 2015/09/23 12:30:40
Components:

Activation Keys:

[crash] root@li-lc-1578:~# find /var/lib/pulp/published -regextype posix-extended -regex '.*product-HOIPRODUCTS-1_1_0-ci.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0
/var/lib/pulp/published/yum/http/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0
/var/lib/pulp/published/yum/https/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-4_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-5_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-4_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-5_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci

[crash] root@li-lc-1578:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml content-view version delete --organization-id=3 --content-view-id=13 --version=4
[...........................................................................................................................................................................] [100%]

[crash] root@li-lc-1578:~# hammer -c /opt/hoici/etc/sat6/hammer-hoici.yaml content-view version delete --organization-id=3 --content-view-id=13 --version=5
[...........................................................................................................................................................................] [100%]

[crash] root@li-lc-1578:~# find /var/lib/pulp/published -regextype posix-extended -regex '.*product-HOIPRODUCTS-1_1_0-ci.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0
/var/lib/pulp/published/yum/http/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0
/var/lib/pulp/published/yum/http/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/3.0
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/4.0
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/5.0
/var/lib/pulp/published/yum/https/repos/Hilti/content_views/product-HOIPRODUCTS-1_1_0-ci/6.0
/var/lib/pulp/published/yum/https/repos/Hilti/Library/product-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-4_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-5_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-Library-product-HOIPRODUCTS-1_1_0-ci-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-3_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create ContentView AAAAAAAAAA
2. Publish 10x the ContentView AAAAAAAAAA
3. List ContentView Versions of AAAAAAAAAA using Hammer
4. Find all directories in Pulp that belong to the Content View AAAAAAAAAA
find /var/lib/pulp/published -regextype posix-extended -regex '.*AAAAAAAAAA.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort
5. Delete ContentView Versions 3.0 - 5.0
6. Find all directories in Pulp that belong to the Content View AAAAAAAAAA
find /var/lib/pulp/published -regextype posix-extended -regex '.*AAAAAAAAAA.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort

Actual results:
Still directories containing data of ContentViewVersions 3.0-5.0 are listed

Expected results:
All data of the deleted contentview versions are deleted from filesystem

Additional info:
Related to BZ1184442 that only implemented a fix for /var/lib/pulp/nodes
Comment 2 Brad Buckingham 2015-10-26 12:01:27 EDT
*** Bug 1251786 has been marked as a duplicate of this bug. ***
Comment 5 Peter Vreman 2016-01-04 11:26:19 EST
Please note that it the fix shall be generic to remove data from the filesystem for all reposititories types removed:
- ContentViewVersion Repositories
- Product Repositories
- Lifecycle Repositories (including the default Library)
- ContentView removal

Next to the Packages also all directory structures created for the published repositories or content views shall be removed.

E.g. after removing ContentViewA and ProductA there shall be no files or directories left that refer to either ContentViewA or ProductA. Directories that i am aware of were currently with Sat6.1.5 directory trees are left behind are:
/var/lib/pulp/published/yum/https/repos/<org>/Library/<content_view_name>
/var/lib/pulp/published/yum/https/repos/<org>/content_views/<content_view_name>
Comment 6 Michael Hrivnak 2016-01-04 13:11:10 EST
This appears to have been fixed upstream in 2.6.1.

https://pulp.plan.io/issues/671
https://github.com/pulp/pulp_rpm/pull/648
Comment 8 Peter Vreman 2016-01-05 07:47:41 EST
The PR648 mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1265703#c6 does not include references that orphaned directories in the following directories are deleted:

/var/lib/pulp/published/yum/master
/var/lib/pulp/published/yum/master/yum_distributor

Are these also being cleaned?
Comment 11 Michael Hrivnak 2016-02-01 07:52:39 EST
Regarding c#9, no, I don't think those directories are being removed. But, those two directories would only be orphaned if every single yum repository was removed from pulp. Removing them would only be of (very small) value if the user did not intended to publish another yum repository. Is that a use case we're concerned about?
Comment 12 Harald Klein 2016-02-02 04:22:47 EST
I guess the use case is valid for API usage in the context of a continuous integration environment. @Peter, can you elaborate on that?
Comment 13 Peter Vreman 2016-02-03 09:22:28 EST
I asked for the removal of the sub directories of each repository inside the those two top directories. Those two top directories are part of the base infrastructure.

E.g. the description mentions the directories for ContentView 'HOIPRODUCTS-1.1.0-ci' ContentViewVersion 6.0

/var/lib/pulp/published/yum/master/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci
/var/lib/pulp/published/yum/master/yum_distributor/Hilti-product-HOIPRODUCTS-1_1_0-ci-6_0-HOIPRODUCTS-HOIPRODUCTS-1_1_0-ci

When i delete this ContentViewVersion also this top directory of the ContentViewVersions shall be removed.
Comment 16 Lukas Pramuk 2016-03-22 06:45:53 EDT
FailedQA.

@Sat6.2.0 Beta Snap4
pulp-katello-1.0-3.el7sat.noarch
pulp-puppet-plugins-2.8.0.1-2.beta.el7sat.noarch
pulp-rpm-plugins-2.8.0.1-1.beta.el7sat.noarch
pulp-nodes-parent-2.8.0.1-0.7.beta.el7sat.noarch
pulp-server-2.8.0.1-0.7.beta.el7sat.noarch
pulp-nodes-common-2.8.0.1-0.7.beta.el7sat.noarch
pulp-client-1.0-1.noarch
pulp-puppet-tools-2.8.0.1-2.beta.el7sat.noarch
pulp-selinux-2.8.0.1-0.7.beta.el7sat.noarch
pulp-docker-plugins-2.0.0.1-1.beta.el7sat.noarch

Used manual reproducer in comment #0:

# hammer content-view info --id 3
Warning: Option --id is deprecated. This option will be removed in Katello 2.5, Use --uuid
ID:                     3
Name:                   Custom View
Label:                  Custom_View
Composite:              false
Description:            
Content Host Count:     0
Organization:           Default Organization
Yum Repositories:       
 1) ID:    4
    Name:  custom repo
    Label: custom_repo
Docker Repositories:    

Puppet Modules:         

Lifecycle Environments: 
 1) ID:   1
    Name: Library
Versions:               
 1) ID:        3
    Version:   1.0
    Published: 2016/03/22 09:54:07
 2) ID:        4
    Version:   2.0
    Published: 2016/03/22 09:54:39
 3) ID:        5
    Version:   3.0
    Published: 2016/03/22 10:08:49
 4) ID:        6
    Version:   4.0
    Published: 2016/03/22 10:10:40
Components:             

Activation Keys:

# hammer content-view remove-version --id 3 --content-view-version 4
Warning: Option --id is deprecated. This option will be removed in Katello 2.5, Use --uuid
The component version has been removed

# hammer content-view remove-version --id 3 --content-view-version 3
Warning: Option --id is deprecated. This option will be removed in Katello 2.5, Use --uuid
The component version has been removed

# find /var/lib/pulp/published -regextype posix-extended -regex '.*Custom_View.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/1.0
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/2.0
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/3.0
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/4.0
/var/lib/pulp/published/yum/http/repos/Default_Organization/Library/Custom_View
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/1.0
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/2.0
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/3.0
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/4.0
/var/lib/pulp/published/yum/https/repos/Default_Organization/Library/Custom_View
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-1_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-2_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-3_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-4_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Library-Custom_View-Custom-custom_repo
/var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-1_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-2_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-3_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-4_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Library-Custom_View-Custom-custom_repo

>>> after deleting "Custom View" versions 3.0 and 4.0, the data are still present on disk
Comment 17 Lukas Pramuk 2016-03-22 07:20:40 EDT
Not necessarily FailedQA 

It looks like following commands do nothing (say 'has been removed' but hasn't)
# hammer content-view remove-version --id 3 --content-view-version '4.0'
# hammer content-view remove-version --id 3 --content-view-version-id 6

# hammer content-view info --id 3
...
Versions:               
 1) ID:        3
    Version:   1.0
    Published: 2016/03/22 09:54:07
 2) ID:        4
    Version:   2.0
    Published: 2016/03/22 09:54:39
 3) ID:        5
    Version:   3.0
    Published: 2016/03/22 10:08:49
 4) ID:        6
    Version:   4.0
    Published: 2016/03/22 10:10:40
...
Comment 18 Lukas Pramuk 2016-03-22 07:30:55 EDT
At webUI by ticking Completely remove version? I was able to completely remove CV 3.0 and 4.0 versions and then...

# find /var/lib/pulp/published -regextype posix-extended -regex '.*Custom_View.*' -type d | egrep -v '/(repodata|custom|[0-9]{8})' | sort
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/1.0
/var/lib/pulp/published/yum/http/repos/Default_Organization/content_views/Custom_View/2.0
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/1.0
/var/lib/pulp/published/yum/https/repos/Default_Organization/content_views/Custom_View/2.0
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-1_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-2_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-3_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Custom_View-4_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/Default_Organization-Library-Custom_View-Custom-custom_repo
/var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-1_0-Custom-custom_repo
/var/lib/pulp/published/yum/master/yum_distributor/Default_Organization-Custom_View-2_0-Custom-custom_repo


Under /var/lib/pulp/published/yum/http{,s}/repos/ the data were deleted
Only under /var/lib/pulp/published/yum/master/ the data remained

Is this ^ still the problem?
Comment 19 Peter Vreman 2016-03-22 12:51:55 EDT
This BZ is created to make sure there is no data left at all on the disk that was used only by the content view versions (or any other repository that is deleted).

That includes also any directory under /var/lib/pulp/published/yum/master/

If not deleted the files will stay there forever and after many  (re)publishing/promoting (composite) content views the disk will suddenly be full.

I have in a my scripts a workaround to delete the directories myself, even with that workaround in place this, even with that i still have many directories:

[crash] root@li-lc-1017:/var/lib/pulp/published/yum/master# ls -l | wc -l
2407

Without the workaround to delete the directories myself it will be full in a 30-60 days.
Comment 20 Lukas Pramuk 2016-03-23 07:15:38 EDT
FailedQA.

With regard to comment #19 switching back to assigned.
Comment 27 Brad Buckingham 2016-04-18 21:26:06 EDT
*** Bug 1325084 has been marked as a duplicate of this bug. ***
Comment 31 Justin Sherrill 2016-06-11 19:15:46 EDT
Extremely related to https://bugzilla.redhat.com/show_bug.cgi?id=1317057  these are likely dupes
Comment 34 Christine Fouant 2016-06-16 16:17:52 EDT

*** This bug has been marked as a duplicate of bug 1317057 ***
Comment 35 Peter Vreman 2016-06-17 03:27:43 EDT
Sorry, the duplicate bug https://bugzilla.redhat.com/show_bug.cgi?id=1317057 is not public accessible. This makes it impossible for me to follow the progress.

Please make https://bugzilla.redhat.com/show_bug.cgi?id=1317057 public.
Comment 36 Ken Long 2016-07-05 11:53:38 EDT
I'm tracking this exact same problem on my Satellite 6.1.9 installation and was referred to these bugzilla reports from my RedHat Support Case (CASE 01650913).   Like Peter, I can't view the the other ticket (1317057) either, so I can't see if there is any progress being made or if it's just being ignored and/or put off until 6.2.0 GA.    

What does it take to get access to the other ticket?  And if it's going to wait for 6.2.0, is there a workaround that can be used to clean up space?  Maybe a script that could be run from a cronjob that would look for invalid content views and remove the content?

Thank you.
Comment 37 Christine Fouant 2016-07-05 12:46:58 EDT
Peter & Ken -

I have made https://bugzilla.redhat.com/show_bug.cgi?id=1317057 public. Please let me know if you need any additional information.

Note You need to log in before you can comment on or make changes to this bug.