Description of problem: All my puppet repositories fail to sync with the following error: Katello::Errors::PulpError PLP0000: Update failed (not okForStorage) Below is the extract from the dynflow console and the pulp backtrace. 3: Actions::Pulp::Repository::Sync (skipped) [ 4.32s / 0.79s ] Started at: 2016-05-11 13:10:06 UTC Ended at: 2016-05-11 13:10:10 UTC Real time: 4.32s Execution time (excluding suspended state): 0.79s Input: --- pulp_id: Hilti-HOICI-HOICI-1_0-sp16-puppet task_id: source_url: remote_user: admin remote_cp_user: admin locale: en Output: --- pulp_tasks: - exception: task_type: pulp.server.managers.repo.sync.sync _href: "/pulp/api/v2/tasks/3ed321ac-f487-4397-aaab-dccb13e9b264/" task_id: 3ed321ac-f487-4397-aaab-dccb13e9b264 tags: - pulp:repository:Hilti-HOICI-HOICI-1_0-sp16-puppet - pulp:action:sync finish_time: '2016-05-11T13:10:08Z' _ns: task_status start_time: '2016-05-11T13:10:06Z' traceback: spawned_tasks: - _href: "/pulp/api/v2/tasks/e1ebaaa5-e6e2-4727-94b2-58fcab1fb5f3/" task_id: e1ebaaa5-e6e2-4727-94b2-58fcab1fb5f3 progress_report: puppet_importer: modules: error_message: execution_time: 1.1618320941925049 total_count: 2 traceback: individual_errors: [] state: success error_count: 0 error: None finished_count: 2 metadata: query_finished_count: 1 traceback: execution_time: 1.0223290920257568 query_total_count: 1 error_message: state: success error: None current_query: queue: reserved_resource_worker-3 state: finished worker_name: reserved_resource_worker-3@li-lc-1578 result: result: success importer_id: puppet_importer exception: repo_id: Hilti-HOICI-HOICI-1_0-sp16-puppet traceback: started: '2016-05-11T13:10:06Z' _ns: repo_sync_results completed: '2016-05-11T13:10:08Z' importer_type_id: puppet_importer error_message: summary: total_execution_time: 2.1841611862182617 added_count: 2 removed_count: 0 updated_count: 0 id: 57332f300586fe0f05da6beb details: finished_count: 2 total_count: 2 error_count: 0 error: _id: "$oid": 57332f2e88dd7a1822825a82 id: 57332f2e88dd7a1822825a82 - exception: task_type: pulp.server.managers.repo.publish.publish _href: "/pulp/api/v2/tasks/e1ebaaa5-e6e2-4727-94b2-58fcab1fb5f3/" task_id: e1ebaaa5-e6e2-4727-94b2-58fcab1fb5f3 tags: - pulp:repository:Hilti-HOICI-HOICI-1_0-sp16-puppet - pulp:action:publish finish_time: '2016-05-11T13:10:09Z' _ns: task_status start_time: '2016-05-11T13:10:09Z' traceback: | Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 473, in __call__ return super(Task, self).__call__(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 103, in __call__ return super(PulpTask, self).__call__(*args, **kwargs) File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 971, in publish result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config) File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 1023, in _do_publish publish_report = publish_repo(transfer_repo, conduit, call_config) File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f return f(*args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/distributor.py", line 46, in publish_repo report = publish_runner.perform_publish() File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py", line 76, in perform_publish self.progress_report.update_progress() File "/usr/lib/python2.6/site-packages/pulp_puppet/common/publish_progress.py", line 97, in update_progress self.conduit.set_progress(report) File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 618, in set_progress set__progress_report=self.progress_report) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 516, in update_one upsert=upsert, multi=False, write_concern=write_concern, **update) File "/usr/lib/python2.6/site-packages/pulp/server/db/querysets.py", line 97, in update super(CriteriaQuerySet, self).update(*args, **kwargs) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 472, in update raise OperationError(u'Update failed (%s)' % unicode(err)) DistributorConduitException: Update failed (not okForStorage) spawned_tasks: [] progress_report: Hilti-HOICI-HOICI-1_0-sp16-puppet_puppet: modules: error_message: execution_time: total_count: 2 traceback: individual_errors: state: running error_count: 0 error: None finished_count: 1 publishing: http: not-started https: not-started metadata: execution_time: state: not-started error_message: error: None traceback: queue: reserved_resource_worker-3 state: error worker_name: reserved_resource_worker-3@li-lc-1578 result: error: code: PLP0000 data: {} description: Update failed (not okForStorage) sub_errors: [] _id: "$oid": 57332f3088dd7a1822825a85 id: 57332f3088dd7a1822825a85 contents_changed: true poll_attempts: total: 5 failed: 1 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Syslog entries of such a failure: May 12 12:06:53 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:INFO: Beginning publish for repository <Hilti-HOIOS-HOIOS-6_6_0-ci-puppet> May 12 12:06:53 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:INFO: Initializing build directory for repository <Hilti-HOIOS-HOIOS-6_6_0-ci-puppet> May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:INFO: Creating symlinks for modules in repository <Hilti-HOIOS-HOIOS-6_6_0-ci-puppet> May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) Exception from server setting progress for report [Hilti-HOIOS-HOIOS-6_6_0-ci-puppet_puppet] May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) Traceback (most recent call last): May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 618, in set_progress May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) set__progress_report=self.progress_report) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 516, in update_one May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) upsert=upsert, multi=False, write_concern=write_concern, **update) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/server/db/querysets.py", line 97, in update May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) super(CriteriaQuerySet, self).update(*args, **kwargs) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 472, in update May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) raise OperationError(u'Update failed (%%s)' %% unicode(err)) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) OperationError: Update failed (not okForStorage) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: Progress value: {'modules': {'finished_count': 1, 'state': 'running', 'traceback': None, 'error_count': 1, 'error': 'None', 'execution_time': None, 'total_count': 50, 'error_message': None, 'individual_errors': {u'hilti-base_selinux-6.6.0-99-B201604291516-G4f1ffe0-hilti': [('/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py', 238, '_symlink_modules', 'os.symlink(module._storage_path, symlink_path)')]}}, 'publishing': {'http': 'not-started', 'https': 'not-started'}, 'metadata': {'execution_time': None, 'state': 'not-started', 'error_message': None, 'traceback': None, 'error': 'None'}} May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) Exception during modules step for repository <Hilti-HOIOS-HOIOS-6_6_0-ci-puppet> May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) Traceback (most recent call last): May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py", line 98, in _modules_step May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) self._symlink_modules(modules) May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py", line 243, in _symlink_modules May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) self.progress_report.update_progress() May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp_puppet/common/publish_progress.py", line 97, in update_progress May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) self.conduit.set_progress(report) May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 618, in set_progress May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) set__progress_report=self.progress_report) May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 516, in update_one May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) upsert=upsert, multi=False, write_concern=write_concern, **update) May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/server/db/querysets.py", line 97, in update May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) super(CriteriaQuerySet, self).update(*args, **kwargs) May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 472, in update May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) raise OperationError(u'Update failed (%s)' % unicode(err)) May 12 12:06:54 li-lc-1578 pulp: pulp_puppet.plugins.distributors.publish:ERROR: (12767-01824) DistributorConduitException: Update failed (not okForStorage) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) Exception from server setting progress for report [Hilti-HOIOS-HOIOS-6_6_0-ci-puppet_puppet] May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) Traceback (most recent call last): May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 618, in set_progress May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) set__progress_report=self.progress_report) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 516, in update_one May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) upsert=upsert, multi=False, write_concern=write_concern, **update) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/server/db/querysets.py", line 97, in update May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) super(CriteriaQuerySet, self).update(*args, **kwargs) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 472, in update May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) raise OperationError(u'Update failed (%%s)' %% unicode(err)) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) OperationError: Update failed (not okForStorage) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: Progress value: {'modules': {'finished_count': 1, 'state': 'failed', 'traceback': [('/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py', 98, '_modules_step', 'self._symlink_modules(modules)'), ('/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py', 243, '_symlink_modules', 'self.progress_report.update_progress()'), ('/usr/lib/python2.6/site-packages/pulp_puppet/common/publish_progress.py', 97, 'update_progress', 'self.conduit.set_progress(report)'), ('/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py', 618, 'set_progress', 'set__progress_report=self.progress_report)'), ('/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py', 516, 'update_one', 'upsert=upsert, multi=False, write_concern=write_concern, **update)'), ('/usr/lib/python2.6/site-packages/pulp/server/db/querysets.py', 97, 'update', 'super(CriteriaQuerySet, self).update(*args, **kwargs)'), ('/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py', 472, 'update', "raise OperationError(u'Update failed (%s)' % unicode(err))")], 'error_count': 1, 'error': 'Update failed (not okForStorage)', 'execution_time': 0, 'total_count': 50, 'error_message': 'Error assembling modules', 'individual_errors': {u'hilti-base_selinux-6.6.0-99-B201604291516-G4f1ffe0-hilti': [('/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py', 238, '_symlink_modules', 'os.symlink(module._storage_path, symlink_path)')]}}, 'publishing': {'http': 'not-started', 'https': 'not-started'}, 'metadata': {'execution_time': None, 'state': 'not-started', 'error_message': None, 'traceback': None, 'error': 'None'}} May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) Exception from server setting progress for report [Hilti-HOIOS-HOIOS-6_6_0-ci-puppet_puppet] May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) Traceback (most recent call last): May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", line 618, in set_progress May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) set__progress_report=self.progress_report) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 516, in update_one May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) upsert=upsert, multi=False, write_concern=write_concern, **update) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/pulp/server/db/querysets.py", line 97, in update May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) super(CriteriaQuerySet, self).update(*args, **kwargs) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) File "/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py", line 472, in update May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) raise OperationError(u'Update failed (%%s)' %% unicode(err)) May 12 12:06:54 li-lc-1578 pulp: pulp.plugins.conduits.mixins:ERROR: (12767-01824) OperationError: Update failed (not okForStorage)
This is very likely a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1331830 I'm leaving it open so that once the fix is available both issues can be verified.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
The progress report it wants to update looks like: { "Hilti-HOICI-HOICI-1_0-ci-puppet_puppet" : { "modules" : { "finished_count" : 2, "state" : "running", "traceback" : null, "error_count" : 1, "error" : "None", "execution_time" : null, "total_count" : 4, "error_message" : null, "individual_errors" : { "hilti-component_hoici-1.0.0-99-B201604060757-G7a47b2e-hilti" : [["/usr/lib/python2.6/site-packages/pulp_puppet/plugins/distributors/publish.py", 238, "_symlink_modules", "os.symlink(module._storage_path, symlink_path)"]] } }, "publishing" : { "http" : "not-started", "https" : "not-started" }, "metadata" : { "execution_time" : null, "state" : " not-started", "error_message" : null, "traceback" : null, "error" : "None" } } }
The problem dict of individual_errors where the module name is used as key: "individual_errors" : { "hilti-component_hoici-1.0.0-99-B201604060757-G7a47b2e-hilti" : [......] } In mongo key of any dict is not allowed to have dots in it.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
*** Bug 1334987 has been marked as a duplicate of this bug. ***
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
Verified in Satellite 6.2 Beta Snap 13. Syncing both puppet labs and custom puppet repos worked without issue.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501