Bug 1131146 - pulp puppet directory import does not delete missing files
Summary: pulp puppet directory import does not delete missing files
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.0.3
Hardware: Unspecified
OS: Unspecified
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
Depends On: 1145719
Blocks: 1122832 sat6-pulp-future 1131148 1175448 1175493
TreeView+ depends on / blocked
Reported: 2014-08-18 14:24 UTC by Peter Vreman
Modified: 2021-04-06 18:03 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1145719 1175493 (view as bug list)
Last Closed: 2016-07-27 09:10:10 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Pulp Redmine 539 0 High CLOSED - CURRENTRELEASE pulp directory import does not delete missing files Never
Red Hat Product Errata RHBA-2016:1501 0 normal SHIPPED_LIVE Red Hat Satellite 6.2 Capsule and Server 2016-07-27 12:28:58 UTC

Description Peter Vreman 2014-08-18 14:24:29 UTC
Description of problem:
Pulp puppet directory importer remove_missing option not working

pulp: pulp.server.managers.repo.sync:ERROR: Traceback (most recent call last):
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 157, in _do_sync
pulp: pulp.server.managers.repo.sync:ERROR:     sync_report = sync_repo(transfer_repo, conduit, call_config)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 458, in wrap_f
pulp: pulp.server.managers.repo.sync:ERROR:     return f(*args, **kwargs)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/importer.py", line 75, in sync_repo
pulp: pulp.server.managers.repo.sync:ERROR:     report = self.sync_method(repo)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py", line 353, in __call__
pulp: pulp.server.managers.repo.sync:ERROR:     self._run(inventory)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py", line 138, in _run
pulp: pulp.server.managers.repo.sync:ERROR:     self._purge_unwanted_modules(inventory, imported_modules)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py", line 328, in _purge_unwa
pulp: pulp.server.managers.repo.sync:ERROR:     s = unit_key_str(unit_key)

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Enable remove_missing in puppet_importer.json
2. Create puppet directory repository
3. Sync repository
4. Delete modules from repository
5. Sync repository again

Actual results:

Expected results:
Delete modules are not available anymore

Additional info:

Comment 1 RHEL Program Management 2014-08-18 14:33:08 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Peter Vreman 2014-08-18 14:54:54 UTC
The following quick patch works, it is based on the code in forge.py.

patch -p0 -l -f << EOF
--- /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py
+++ /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py
@@ -319,10 +319,20 @@
         if not purge_option:
             # no purge requested
+        criteria = UnitAssociationCriteria(
+            type_ids=[constants.TYPE_PUPPET_MODULE], unit_fields=('name', 'version', 'author'))
+        existing_units = self.conduit.get_units(criteria=criteria)
+        existing_units_by_key = {}
+        for unit in existing_units:
+            module = Module.from_unit(unit)
+            existing_units_by_key[tuple(module.unit_key().items())] = unit
         for unit_key in inventory.unwanted_modules(imported_modules):
             if self.canceled:
-            self.conduit.remove_unit(unit_key)
+            doomed = existing_units_by_key[tuple(unit_key.items())]
+            self.conduit.remove_unit(doomed)

     def __call__(self, repository):


Comment 5 pulp-infra@redhat.com 2015-04-23 16:40:15 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 9 jcallaha 2016-05-20 19:40:53 UTC
Verified in Satellite 6.2 Beta Snap 12.

This functionality is now working correctly when manually managing uploaded puppet modules in a repo. Additionally, I feel it would be very beneficial to be able to filter puppet repositories in Satellite like you can with pulp-admin.

pulp-admin puppet repo update --repo-id=repo1 --queries=libvirt

Comment 10 Peter Vreman 2016-05-21 09:22:53 UTC
Can you explain what "manual managing" means? The ticket is explicit refers to "directory importing" also known as "Syncing". that means it shall be verified with the command:

pulp-admin puppet repo sync

Comment 12 errata-xmlrpc 2016-07-27 09:10:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.