Bug 1175493 - pulp puppet directory import does not delete missing files
Summary: pulp puppet directory import does not delete missing files
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: Unspecified
Hardware: Unspecified
OS: Unspecified
high
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
Depends On: 1131146 1145719
Blocks: 1122832 sat6-pulp-future 1131148 1175448
TreeView+ depends on / blocked
 
Reported: 2014-12-17 20:36 UTC by Michael Hrivnak
Modified: 2019-04-01 20:26 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1131146
Environment:
Last Closed: 2016-07-27 09:12:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1501 normal SHIPPED_LIVE Red Hat Satellite 6.2 Capsule and Server 2016-07-27 12:28:58 UTC
Pulp Redmine 645 High CLOSED - CURRENTRELEASE pulp puppet directory import does not delete missing files Never

Comment 1 Dennis Kliban 2015-01-08 18:54:09 UTC
I was not able to reproduce the bug.  Here are the approaches I took:

I added the following to the puppet_importer.json

{
 "remove_missing": "True"
}

Then I ran 3 commands:

pulp-admin puppet repo create --repo-id=repo1 --description="Mirror of Puppet Forge" --display-name="Repo 1" --feed=http://forge.puppetlabs.com

pulp-admin puppet repo update --repo-id=repo1 --queries=libvirt

pulp-admin puppet repo sync run --repo-id=repo1

The three commands above created a repository with 8 modules.  Then I ran the following commands:

pulp-admin puppet repo update --repo-id=repo1 --queries=libvirttttttt
pulp-admin puppet repo sync run --repo-id=repo1

This produced an empty repository because nothing matched the query.  I didn't see any tracebacks in the logs. 

I then attempted to reproduce this bug without modifying the --querries attribute of the repo.  

pulp-admin puppet repo create --repo-id=repo1 --description="Mirror of Puppet Forge" --display-name="Repo 1" --feed=http://forge.puppetlabs.com

pulp-admin puppet repo update --repo-id=repo1 --queries=libvirt

pulp-admin puppet repo sync run --repo-id=repo1

pulp-admin puppet repo create --repo-id=repo1 --description="Mirror of repo1" --display-name="Repo 2" --feed=http://localhost/pulp/puppet/repo1

pulp-admin puppet repo sync run --repo-id=repo2

After running the above commands I had 2 repositories with 8 modules. Then I ran the following 3 commands:

pulp-admin puppet repo update --repo-id=repo1 --queries=libvirttttttt
pulp-admin puppet repo sync run --repo-id=repo1
pulp-admin puppet repo sync run --repo-id=repo2

This produced two repos that were empty.  Once again - I didn't see any tracebacks.

I also tried manually removing modules from a repository and then synced from it without seeing this bug.

Comment 2 Preethi Thomas 2015-01-30 16:37:36 UTC
verified

[root@cloud-qe-4 modules]# rpm -qa pulp-server
pulp-server-2.6.0-0.5.beta.el7.noarch
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]# rpm -qa |grep puppet
pulp-puppet-admin-extensions-2.6.0-0.5.beta.el7.noarch
pulp-puppet-tools-2.6.0-0.5.beta.el7.noarch
python-pulp-puppet-common-2.6.0-0.5.beta.el7.noarch
pulp-puppet-plugins-2.6.0-0.5.beta.el7.noarch
puppet-3.6.2-3.el7.noarch
[root@cloud-qe-4 modules]# 

Here are the steps I followed

1. Enable remove_missing in puppet_importer.json
$ sudo mkdir -p /opt/puppet/modules
$ sudo chmod -R 777 /opt/puppet
$ cd /opt/puppet
$ pulp-puppet-module-builder --url=https://github.com/puppetlabs/puppetlabs-xinetd -o ../modules

$ pulp-puppet-module-builder --working-dir=/opt/puppet --output-dir=/opt/puppet/modules --clean --url=https://github.com/puppetlabs/puppetlabs-mcollective.git -t 2.0.0

$pulp-admin puppet repo sync run --repo-id=puppet-builds

Now I removed the puppetlabs-mcollective-2.0.0.tar.gz

$ rm -rf puppetlabs-mcollective-2.0.0.tar.gz

Also removed it from the PULP_MANIFEST

Now ran pulp-admin puppet repo sync run --repo-id=puppet-builds

listing puppet repo shows that the module is removed from the repo





[root@cloud-qe-4 modules]# ls
PULP_MANIFEST  puppetlabs-inifile-1.1.3.tar.gz  puppetlabs-xinetd-1.4.0.tar.gz  reidmv-mco_profile-0.1.0.tar.gz
[root@cloud-qe-4 modules]# vi PULP_MANIFEST 
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]# rm -rf reidmv-mco_profile-0.1.0.tar.gz
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]# pulp-admin puppet repo sync run --repo-id=puppet-builds
+----------------------------------------------------------------------+
                Synchronizing Repository [puppet-builds]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[==================================================] 100%
Metadata Query: 1/1 items
... completed

Downloading new modules...
[==================================================] 100%
Module: 0/0 items
... completed


Task Succeeded



Publishing modules...
[==================================================] 100%
Module: 2/2 items
... completed

Generating repository metadata...
[-]
... completed

Publishing repository over HTTP...
... completed

Publishing repository over HTTPS...
... skipped

Task Succeeded


[root@cloud-qe-4 modules]# pulp-admin puppet repo list
+----------------------------------------------------------------------+
                          Puppet Repositories
+----------------------------------------------------------------------+

Id:                  repo1
Display Name:        Repo 1
Description:         Mirror of Puppet Forge
Content Unit Counts: 
  Puppet Module: 1835

Id:                  puppet-builds
Display Name:        puppet-builds
Description:         None
Content Unit Counts: 
  Puppet Module: 2


[root@cloud-qe-4 modules]# rpm -qa pulp-server
pulp-server-2.6.0-0.5.beta.el7.noarch
[root@cloud-qe-4 modules]# 
[root@cloud-qe-4 modules]#

Comment 3 Brian Bouterse 2015-02-28 23:16:55 UTC
Moved to https://pulp.plan.io/issues/645

Comment 4 RHEL Product and Program Management 2015-03-04 11:24:00 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 6 pulp-infra@redhat.com 2015-04-23 16:39:55 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 10 jcallaha 2016-05-20 19:41:02 UTC
Verified in Satellite 6.2 Beta Snap 12.

This functionality is now working correctly when manually managing uploaded puppet modules in a repo. Additionally, I feel it would be very beneficial to be able to filter puppet repositories in Satellite like you can with pulp-admin.

pulp-admin puppet repo update --repo-id=repo1 --queries=libvirt

Comment 12 errata-xmlrpc 2016-07-27 09:12:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501


Note You need to log in before you can comment on or make changes to this bug.