Bug 1131146

Summary: pulp puppet directory import does not delete missing files
Product: Red Hat Satellite Reporter: Peter Vreman <peter.vreman>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.0.3CC: bbuckingham, bkearney, bmbouter, daviddavis, dkliban, ealcaniz, ggainey, ipanova, jcallaha, jyejare, mhrivnak, pcreech, rchan, sthirugn, ttereshc
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1145719 1175493 (view as bug list) Environment:
Last Closed: 2016-07-27 09:10:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1145719    
Bug Blocks: 950746, 1122832, 1131148, 1175448, 1175493    

Description Peter Vreman 2014-08-18 14:24:29 UTC
Description of problem:
Pulp puppet directory importer remove_missing option not working

pulp: pulp.server.managers.repo.sync:ERROR: Traceback (most recent call last):
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 157, in _do_sync
pulp: pulp.server.managers.repo.sync:ERROR:     sync_report = sync_repo(transfer_repo, conduit, call_config)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 458, in wrap_f
pulp: pulp.server.managers.repo.sync:ERROR:     return f(*args, **kwargs)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/importer.py", line 75, in sync_repo
pulp: pulp.server.managers.repo.sync:ERROR:     report = self.sync_method(repo)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py", line 353, in __call__
pulp: pulp.server.managers.repo.sync:ERROR:     self._run(inventory)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py", line 138, in _run
pulp: pulp.server.managers.repo.sync:ERROR:     self._purge_unwanted_modules(inventory, imported_modules)
pulp: pulp.server.managers.repo.sync:ERROR:   File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py", line 328, in _purge_unwa
nted_modules
pulp: pulp.server.managers.repo.sync:ERROR:     s = unit_key_str(unit_key)



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Enable remove_missing in puppet_importer.json
2. Create puppet directory repository
3. Sync repository
4. Delete modules from repository
5. Sync repository again

Actual results:


Expected results:
Delete modules are not available anymore


Additional info:

Comment 1 RHEL Program Management 2014-08-18 14:33:08 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Peter Vreman 2014-08-18 14:54:54 UTC
The following quick patch works, it is based on the code in forge.py.

patch -p0 -l -f << EOF
--- /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py
+++ /usr/lib/python2.6/site-packages/pulp_puppet/plugins/importers/directory.py
@@ -319,10 +319,20 @@
         if not purge_option:
             # no purge requested
             return
+
+        criteria = UnitAssociationCriteria(
+            type_ids=[constants.TYPE_PUPPET_MODULE], unit_fields=('name', 'version', 'author'))
+        existing_units = self.conduit.get_units(criteria=criteria)
+        existing_units_by_key = {}
+        for unit in existing_units:
+            module = Module.from_unit(unit)
+            existing_units_by_key[tuple(module.unit_key().items())] = unit
+
         for unit_key in inventory.unwanted_modules(imported_modules):
             if self.canceled:
                 return
-            self.conduit.remove_unit(unit_key)
+            doomed = existing_units_by_key[tuple(unit_key.items())]
+            self.conduit.remove_unit(doomed)

     def __call__(self, repository):
         """

EOF

Comment 5 pulp-infra@redhat.com 2015-04-23 16:40:15 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 9 jcallaha 2016-05-20 19:40:53 UTC
Verified in Satellite 6.2 Beta Snap 12.

This functionality is now working correctly when manually managing uploaded puppet modules in a repo. Additionally, I feel it would be very beneficial to be able to filter puppet repositories in Satellite like you can with pulp-admin.

pulp-admin puppet repo update --repo-id=repo1 --queries=libvirt

Comment 10 Peter Vreman 2016-05-21 09:22:53 UTC
Can you explain what "manual managing" means? The ticket is explicit refers to "directory importing" also known as "Syncing". that means it shall be verified with the command:

pulp-admin puppet repo sync

Comment 12 errata-xmlrpc 2016-07-27 09:10:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501