Cloning it for testing purposes on 2.5.3 +++ This bug was initially created as a clone of Bug #1176698 +++ Description of problem: Prior to and after upgrading from 2.5.0 to 2.5.1 (and now also after applying the 2.5.2-0.0.beta packages), I've been unable to publish both feed-url and upload rpm repos. The issue appears to affect all of my repos. Version-Release number of selected component (if applicable): 2.5.0 2.5.1 2.5.2-0.0.beta How reproducible: Always Steps to Reproduce: 1. pulp-admin rpm repo publish run --repo-id=<repo name> OR pulp-admin rpm repo sync run --repo-id=<repo-name> 2. 3. Actual results: # pulp-admin rpm repo publish run --repo-id=rhel-6-x86_64-os +----------------------------------------------------------------------+ Publishing Repository [rhel-6-x86_64-os] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [====================== ] 45% 6605 of 14365 items ... failed Incorrect length of data produced Task Failed Incorrect length of data produced # pulp-admin rpm repo sync run --repo-id=rhel-6-x86_64-os +----------------------------------------------------------------------+ Synchronizing Repository [rhel-6-x86_64-os] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [\] ... completed Downloading repository content... [==================================================] 100% RPMs: 113/113 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [\] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [====================== ] 45% 6643 of 14478 items ... failed Incorrect length of data produced Task Failed Incorrect length of data produced Expected results: Successfully publishing repos for use via yum. Additional info: Current package version (although this was failing prior to upgrading to this version): yum list installed |grep pulp grinder.noarch 0.1.16-1.el6 @pulp-v2-stable m2crypto.x86_64 0.21.1.pulp-8.el6 @pulp-v2-stable mod_wsgi.x86_64 3.4-2.pulp.el6 @pulp-v2-stable pulp-admin-client.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta pulp-puppet-admin-extensions.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta pulp-puppet-plugins.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta pulp-rpm-admin-extensions.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta pulp-rpm-plugins.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta pulp-selinux.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta pulp-server.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta python-amqp.noarch 1.4.5-1.el6 @pulp-v2-stable python-anyjson.noarch 0.3.3-4.el6 @pulp-v2-stable python-billiard.x86_64 1:3.3.0.17-1.el6 @pulp-v2-stable python-celery.noarch 3.1.11-1.el6 @pulp-v2-stable python-gofer.noarch 1.3.1-1.el6 @pulp-v2-stable python-gofer-qpid.noarch 1.3.1-1.el6 @pulp-v2-stable python-isodate.noarch 0.5.0-4.pulp.el6 @pulp-v2-stable python-kombu.noarch 1:3.0.15-12.pulp.el6 @pulp-v2-stable python-nectar.noarch 1.3.1-1.el6 @pulp-v2-stable python-pulp-bindings.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta python-pulp-client-lib.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta python-pulp-common.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta python-pulp-puppet-common.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta python-pulp-rpm-common.noarch 2.5.2-0.0.beta.el6 @pulp-v2-beta python-qpid.noarch 0.26-2.el6 @pulp-v2-stable python-qpid-common.noarch 0.26-2.el6 @pulp-v2-stable python-qpid-qmf.x86_64 0.26-2.el6 @pulp-v2-stable python-requests.noarch 2.2.1-1.el6 @pulp-v2-stable python-semantic-version.noarch 2.2.0-3.el6 @pulp-v2-stable qpid-cpp-client.x86_64 0.26-9.el6 @pulp-v2-beta qpid-cpp-server.x86_64 0.26-9.el6 @pulp-v2-beta qpid-cpp-server-store.x86_64 0.26-9.el6 @pulp-v2-beta qpid-qmf.x86_64 0.26-2.el6 @pulp-v2-stable Trace from /var/log/messages: Dec 22 14:57:51 dvpuap02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[831267ce-98eb-41c3-9dad-0b0e32035ec0] Dec 22 14:57:51 dvpuap02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[831267ce-98eb-41c3-9dad-0b0e32035ec0] succeeded in 0.031 6150639992s: u'reserved_resource_worker-2.edu.dq' Dec 22 14:57:52 dvpuap02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[7003450d-2b88-45a5-b05b-f5a92cfbba67] Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [epel6] Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: Traceback (most recent call last): Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 1 26, in _do_publish Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 470, in wra p_f Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: return f(*args, **kwargs) Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor .py", line 143, in publish_repo Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: return self._publisher.publish() Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 507, in publish Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: return self.process_lifecycle() Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 456, in process_lifecycle Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: super(PluginStep, self).process_lifecycle() Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 121, in process_lifecycle Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: step.process() Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 185, in process Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self.finalize() Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 414, in finalize Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self.other_context.checksum) Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 115, in add_metadata_file_metadata Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: content = file_handle.read() Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.6/gzip.py", line 212, in read Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self._read(readsize) Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.6/gzip.py", line 284, in _read Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self._read_eof() Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.6/gzip.py", line 306, in _read_eof Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: raise IOError, "Incorrect length of data produced" Dec 22 14:57:58 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: IOError: Incorrect length of data produced Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: Task pulp.server.managers.repo.publish.publish[7003450d-2b88-45a5-b05b-f5a92cfbba67] raised unexpected: IOError('Incorrect length of data produced',) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: Traceback (most recent call last): Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: R = retval = fun(*args, **kwargs) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 315, in __call__ Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: return super(Task, self).__call__(*args, **kwargs) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: return self.run(*args, **kwargs) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 98, in publish Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: transfer_repo, conduit, call_config) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 470, in wrap_f Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: return f(*args, **kwargs) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 143, in publish_repo Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: return self._publisher.publish() Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 507, in publish Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: return self.process_lifecycle() Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 456, in process_lifecycle Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: super(PluginStep, self).process_lifecycle() Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 121, in process_lifecycle Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: step.process() Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 185, in process Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: self.finalize() Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 414, in finalize Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: self.other_context.checksum) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 115, in add_metadata_file_metadata Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: content = file_handle.read() Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.6/gzip.py", line 212, in read Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: self._read(readsize) Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.6/gzip.py", line 284, in _read Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: self._read_eof() Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.6/gzip.py", line 306, in _read_eof Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: raise IOError, "Incorrect length of data produced" Dec 22 14:57:58 dvpuap02 pulp: celery.worker.job:ERROR: IOError: Incorrect length of data produced --- Additional comment from Michael Hrivnak on 2015-01-02 09:23:39 EST --- Please also share the traceback that occurs during sync. Is this reproducible by syncing a publicly-available repository? Let us know which so we can try to reproduce it here. --- Additional comment from Michael Hrivnak on 2015-01-02 09:33:11 EST --- For reference, this is the call in pulp's publish workflow that results in an exception from python's gzip module: https://github.com/pulp/pulp_rpm/blob/pulp-rpm-2.5.2-0.0.beta/plugins/pulp_rpm/plugins/distributors/yum/metadata/repomd.py#L115 --- Additional comment from Michael Hrivnak on 2015-01-02 09:39:35 EST --- --- Additional comment from Bryce Pier on 2015-01-02 13:28:07 EST --- The publish error occurs on several of my 17 repos. I thought it was all of them as the first 4 I attempted all failed. The following are failing: https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/optional/os http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64 https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os http://mirror.us.leaseweb.net/epel/6/x86_64/ http://mirror.us.leaseweb.net/epel/5/x86_64/ These are publishing correctly: https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/rhscl/1/os https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/debug https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/optional/os https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/os https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/debug http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/ http://mirror.pnl.gov/epel/7/x86_64/ https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/scalablefilesystem/os http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64 https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os A couple of items of note: 1. The publish failures always happen on the same package for each repo (e.g. for epel6 at 75 of 19440 items, for rhel6-optional at 54 of 8104 items) 2. I have a repo called rhel-6-x86_64-os-physical which is a point in time copy of the rhel6 os repo just prior to 6.6 being released. That repo is publishing fine but the rhel6 os repo is not. Here is the traceback from the sync of the rhel6 os repo: Jan 2 11:48:06 dvpuap02 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 1 RPMs. Jan 2 11:51:06 dvpuap02 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com Jan 2 11:52:33 dvpuap02 pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {'___/primary/___': {'tot al_failed': 0, 'total_succeeded': 1}}, 'total_sources': 0} for base URL: https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os/ Jan 2 11:52:33 dvpuap02 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units. Jan 2 11:52:33 dvpuap02 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com Jan 2 11:52:34 dvpuap02 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com Jan 2 11:52:49 dvpuap02 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Sync complete. Jan 2 11:52:49 dvpuap02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[fc8b7a5f-5911-43ae-ad11-4e2661791ede] Jan 2 11:52:49 dvpuap02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[fc8b7a5f-5911-43ae-ad11-4e2661791ede] succeeded in 0.0 31124184s: u'reserved_resource_worker-0.edu.dq' Jan 2 11:52:50 dvpuap02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[f0a6ad0e-2fe6-4eaa-838c-465b91db9d37] Jan 2 11:52:50 dvpuap02 pulp: celery.worker.job:INFO: Task pulp.server.tasks.repository.sync_with_auto_publish[4b8189f1-6c87-40a1-97f2-3176bab09b76] succeeded in 1773.61550857s: <pulp.server.async.tasks.TaskResult object at 0x7fb75803eb50> Jan 2 11:52:58 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:53:09 dvpuap02 rsyslogd-2177: imuxsock lost 352 messages from pid 2344 due to rate-limiting Jan 2 11:53:09 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:53:15 dvpuap02 rsyslogd-2177: imuxsock lost 130 messages from pid 2344 due to rate-limiting Jan 2 11:53:17 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:53:21 dvpuap02 rsyslogd-2177: imuxsock lost 872 messages from pid 2344 due to rate-limiting Jan 2 11:53:22 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:53:27 dvpuap02 rsyslogd-2177: imuxsock lost 853 messages from pid 2344 due to rate-limiting Jan 2 11:53:27 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:53:33 dvpuap02 rsyslogd-2177: imuxsock lost 974 messages from pid 2344 due to rate-limiting Jan 2 11:53:42 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:53:45 dvpuap02 rsyslogd-2177: imuxsock lost 68 messages from pid 2344 due to rate-limiting Jan 2 11:54:44 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:54:51 dvpuap02 rsyslogd-2177: imuxsock lost 50 messages from pid 2344 due to rate-limiting Jan 2 11:54:52 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:54:59 dvpuap02 rsyslogd-2177: imuxsock lost 78 messages from pid 2344 due to rate-limiting Jan 2 11:55:00 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:55:05 dvpuap02 rsyslogd-2177: imuxsock lost 481 messages from pid 2344 due to rate-limiting Jan 2 11:55:08 dvpuap02 rsyslogd-2177: imuxsock begins to drop messages from pid 2344 due to rate-limiting Jan 2 11:55:14 dvpuap02 rsyslogd-2177: imuxsock lost 172 messages from pid 2344 due to rate-limiting Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [rhel-6-x86_64-os] Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: Traceback (most recent call last): Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 470, in wrap_f Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: return f(*args, **kwargs) Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 143, in publish_repo Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: return self._publisher.publish() Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 507, in publish Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: return self.process_lifecycle() Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 456, in process_lifecycle Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: super(PluginStep, self).process_lifecycle() Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 121, in process_lifecycle Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: step.process() Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 185, in process Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self.finalize() Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 414, in finalize Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self.other_context.checksum) Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 115, in add_metadata_file_metadata Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: content = file_handle.read() Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.6/gzip.py", line 212, in read Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self._read(readsize) Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.6/gzip.py", line 284, in _read Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: self._read_eof() Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.6/gzip.py", line 306, in _read_eof Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: raise IOError, "Incorrect length of data produced" Jan 2 11:55:22 dvpuap02 pulp: pulp.server.managers.repo.publish:ERROR: IOError: Incorrect length of data produced Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: Task pulp.server.managers.repo.publish.publish[f0a6ad0e-2fe6-4eaa-838c-465b91db9d37] raised unexpected: IOError('Incorrect length of data produced',) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: Traceback (most recent call last): Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: R = retval = fun(*args, **kwargs) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 315, in __call__ Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: return super(Task, self).__call__(*args, **kwargs) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: return self.run(*args, **kwargs) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 98, in publish Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: transfer_repo, conduit, call_config) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 470, in wrap_f Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: return f(*args, **kwargs) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 143, in publish_repo Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: return self._publisher.publish() Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 507, in publish Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: return self.process_lifecycle() Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 456, in process_lifecycle Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: super(PluginStep, self).process_lifecycle() Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 121, in process_lifecycle Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: step.process() Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 185, in process Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: self.finalize() Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 414, in finalize Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: self.other_context.checksum) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 115, in add_metadata_file_metadata Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: content = file_handle.read() Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.6/gzip.py", line 212, in read Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: self._read(readsize) Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.6/gzip.py", line 284, in _read Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: self._read_eof() Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.6/gzip.py", line 306, in _read_eof Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: raise IOError, "Incorrect length of data produced" Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:ERROR: IOError: Incorrect length of data produced Jan 2 11:55:22 dvpuap02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8c30c2b7-e142-4550-9914-dc1145cbd8f8] Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_release_resource[0e332bf8-5a6a-4331-b6f7-66c24b85f6a0] succeeded in 0.0150966489996s: None Jan 2 11:55:22 dvpuap02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8c30c2b7-e142-4550-9914-dc1145cbd8f8] succeeded in 0.042722529s: None --- Additional comment from Jiri Tyr on 2015-01-05 19:58:36 EST --- I think this issue it's related to the bug #1158945. The proposed patch (https://github.com/pulp/pulp_rpm/pull/605) fixes the problem but on CentOS 6.5 you have to modify the patch to use encode('utf-8') instead of encode(encoding='utf-8'). --- Additional comment from Preethi Thomas on 2015-01-06 13:37:00 EST --- I was able sync and publish rhel6 repo from cdn. Pasting the results. Please let us know if we are missing something. [root@ibm-x3250m4-02 ~]# pulp-admin rpm repo create --repo-id rhel6 --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.6/x86_64/os/ --feed-ca-cert CDN/cdn.redhat.com-chain.crt --feed-cert CDN/1391032266_4771.crt --feed-key CDN/1391032266_4771.key Successfully created repository [rhel6] [root@ibm-x3250m4-02 ~]# [root@ibm-x3250m4-02 ~]# [root@ibm-x3250m4-02 ~]# pulp-admin rpm repo sync run --repo-id rhel6 +----------------------------------------------------------------------+ Synchronizing Repository [rhel6] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [|] ... completed Downloading repository content... [==================================================] 100% RPMs: 14472/14472 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [\] ... completed Importing package groups/categories... [|] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 14472 of 14472 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 2818 of 2818 items ... completed Publishing Comps file [==================================================] 100% 212 of 212 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [|] ... completed Writing Listings File [-] ... completed Task Succeeded [root@ibm-x3250m4-02 ~]# --- Additional comment from Barnaby Court on 2015-01-07 09:44:58 EST --- https://github.com/pulp/pulp_rpm/pull/623 --- Additional comment from Chris Duryee on 2015-01-14 19:26:15 EST --- This was merged to 2.6-dev which aligns to 2.6.1. If you need this fix in 2.6.0, please create a PR on 2.6-testing, and set the bug's target release back to 2.6.0. Setting target release to 2.6.1. --- Additional comment from Irina Gulina on 2015-01-15 14:36:03 EST --- >> rpm -qa pulp-server pulp-server-2.6.0-0.4.beta.el6.noarch >> pulp-admin rpm repo create --repo-id epel6_1 --feed http://mirror.karneval.cz/pub/linux/fedora/epel/6/x86_64/ Successfully created repository [epel6_1] >> pulp-admin rpm repo sync run --repo-id epel6_1 +----------------------------------------------------------------------+ Synchronizing Repository [epel6_1] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [|] ... completed Downloading repository content... [==================================================] 100% RPMs: 11178/11178 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [\] ... completed Importing package groups/categories... [|] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [ ] 0% 1 of 11178 items ... failed encode() takes no keyword arguments Task Failed encode() takes no keyword arguments --- Additional comment from Chris Duryee on 2015-01-16 18:00:01 EST --- pulp 2.6.0 beta 5 --- Additional comment from Irina Gulina on 2015-01-16 23:14:42 EST --- after update to pulp-server-2.6.0-0.5.beta.el6.noarch >> pulp-admin rpm repo sync run --repo-id epel6_1 +----------------------------------------------------------------------+ Synchronizing Repository [epel6_1] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [\] ... completed Downloading repository content... [==================================================] 100% RPMs: 1/1 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [|] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 11178 of 11178 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 3635 of 3635 items ... completed Publishing Comps file [==================================================] 100% 211 of 211 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [|] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin rpm repo publish run --repo-id epel6_1 +----------------------------------------------------------------------+ Publishing Repository [epel6_1] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Copying files [|] ... completed Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [/] ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 3635 of 3635 items ... completed Publishing Comps file [==================================================] 100% 211 of 211 items ... completed Publishing Metadata. [\] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [/] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin rpm repo list +----------------------------------------------------------------------+ RPM Repositories +----------------------------------------------------------------------+ Id: epel6_1 Display Name: epel6_1 Description: None Content Unit Counts: Erratum: 3635 Package Category: 3 Package Group: 208 Rpm: 11178 Yum Repo Metadata File: 1 >> ls -l /var/lib/pulp/published/yum/master/yum_distributor/ total 4 drwxr-xr-x. 3 apache apache 4096 Jan 16 22:51 epel6_1 I also checked this http://mirror.us.leaseweb.net/epel/5/x86_64/ from commnet#4 >> pulp-admin rpm repo create --repo-id epel5 --feed http://mirror.us.leaseweb.net/epel/5/x86_64/ Successfully created repository [epel5] >> pulp-admin rpm repo sync run --repo-id epel5 +----------------------------------------------------------------------+ Synchronizing Repository [epel5] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [/] ... completed Downloading repository content... [==================================================] 100% RPMs: 6848/6848 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [\] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 6848 of 6848 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 1998 of 1998 items ... completed Publishing Comps file [==================================================] 100% 41 of 41 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [\] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin rpm repo publish run --repo-id epel5 +----------------------------------------------------------------------+ Publishing Repository [epel5] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Copying files [\] ... completed Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [\] ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 1998 of 1998 items ... completed Publishing Comps file [==================================================] 100% 41 of 41 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [|] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin rpm repo list +----------------------------------------------------------------------+ RPM Repositories +----------------------------------------------------------------------+ Id: epel6_1 Display Name: epel6_1 Description: None Content Unit Counts: Erratum: 3635 Package Category: 3 Package Group: 208 Rpm: 11178 Yum Repo Metadata File: 1 Id: epel5 Display Name: epel5 Description: None Content Unit Counts: Erratum: 1998 Package Category: 5 Package Group: 36 Rpm: 6848 Yum Repo Metadata File: 1 >> ls -l /var/lib/pulp/published/yum/master/yum_distributor/ total 8 drwxr-xr-x. 3 apache apache 4096 Jan 16 23:11 epel5 drwxr-xr-x. 3 apache apache 4096 Jan 16 22:51 epel6_1 --- Additional comment from Josh Baird on 2015-01-20 08:20:27 EST --- I'm having the same issue with: pulp-server-2.5.1-1.el7.noarch $ pulp-admin rpm repo publish run --repo-id=test +----------------------------------------------------------------------+ Publishing Repository [test] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [== ] 4% 1 of 25 items ... failed Incorrect length of data produced Task Failed Incorrect length of data produced Traceback: Jan 20 07:19:51 fc-d1pulp02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[5492606b-de1f-4b9f-9d85-17b422d611f7] Jan 20 07:19:51 fc-d1pulp02 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[5492606b-de1f-4b9f-9d85-17b422d611f7] succeeded in 0.0307118759956s: u'reserved_resource_worker-0.follett.com.dq' Jan 20 07:19:51 fc-d1pulp02 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[9bb85e7f-d035-478a-86b8-61c417239d9b] Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [test] Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: Traceback (most recent call last): Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 470, in wrap_f Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: return f(*args, **kwargs) Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 143, in publish_repo Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: return self._publisher.publish() Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 507, in publish Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: return self.process_lifecycle() Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 456, in process_lifecycle Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: super(PluginStep, self).process_lifecycle() Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 121, in process_lifecycle Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: step.process() Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 185, in process Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: self.finalize() Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 414, in finalize Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: self.other_context.checksum) Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 115, in add_metadata_file_metadata Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: content = file_handle.read() Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.7/gzip.py", line 254, in read Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: self._read(readsize) Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.7/gzip.py", line 325, in _read Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: self._read_eof() Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib64/python2.7/gzip.py", line 349, in _read_eof Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: raise IOError, "Incorrect length of data produced" Jan 20 07:19:52 fc-d1pulp02 pulp: pulp.server.managers.repo.publish:ERROR: IOError: Incorrect length of data produced Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: Task pulp.server.managers.repo.publish.publish[9bb85e7f-d035-478a-86b8-61c417239d9b] raised unexpected: IOError('Incorrect length of data produced',) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: Traceback (most recent call last): Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: R = retval = fun(*args, **kwargs) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 315, in __call__ Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: return super(Task, self).__call__(*args, **kwargs) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: return self.run(*args, **kwargs) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 98, in publish Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: transfer_repo, conduit, call_config) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 470, in wrap_f Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: return f(*args, **kwargs) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 143, in publish_repo Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: return self._publisher.publish() Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 507, in publish Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: return self.process_lifecycle() Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 456, in process_lifecycle Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: super(PluginStep, self).process_lifecycle() Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 121, in process_lifecycle Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: step.process() Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 185, in process Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: self.finalize() Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 414, in finalize Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: self.other_context.checksum) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/repomd.py", line 115, in add_metadata_file_metadata Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: content = file_handle.read() Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.7/gzip.py", line 254, in read Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: self._read(readsize) Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.7/gzip.py", line 325, in _read Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: self._read_eof() Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: File "/usr/lib64/python2.7/gzip.py", line 349, in _read_eof Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: raise IOError, "Incorrect length of data produced" Jan 20 07:19:52 fc-d1pulp02 pulp: celery.worker.job:ERROR: IOError: Incorrect length of data produced Is there a proposed fix for 2.5.1? --- Additional comment from Jiri Tyr on 2015-01-20 08:29:33 EST --- (In reply to Josh Baird from comment #12) > I'm having the same issue with: > > pulp-server-2.5.1-1.el7.noarch > > Is there a proposed fix for 2.5.1? Yes, apply these two patches: https://github.com/jeremycline/pulp_rpm/commit/6117c1ac703f4f759bd1aaa5da9a9ca8559495af.patch https://github.com/barnabycourt/pulp_rpm/commit/5bbb8b4be838ff4f4be9549bd1986efca5c23ce6.patch --- Additional comment from Josh Baird on 2015-01-20 09:59:31 EST --- This doesn't appear to apply cleanly to 2.5.1. --- Additional comment from Jiri Tyr on 2015-01-20 10:01:06 EST --- (In reply to Josh Baird from comment #14) > This doesn't appear to apply cleanly to 2.5.1. Then patch it manually. You only need the "if" conditions. --- Additional comment from Bryce Pier on 2015-01-20 12:17:18 EST --- I can confirm that 2.6.0-0.5.beta fixes my issue and all of my repos now publish successfully. --- Additional comment from Dhaval on 2015-01-23 06:59:08 EST --- This is still an issues with 2.5.1 after applying patch, how do i resolve this in 2.5.1..? this is bit critical.. it only does not upload some RPM..not sure what's the reason --- Additional comment from Dhaval on 2015-01-23 09:09:32 EST --- please ignore, it does work, tested it.
Moving to on-qa since its already in a beta build.
verified [root@gizmo ~]# rpm -qa pulp-server pulp-server-2.5.3-0.1.beta.el6.noarch [root@gizmo ~] [root@gizmo ~]# [root@gizmo ~]# pulp-admin rpm repo sync run --repo-id epel6_1 +----------------------------------------------------------------------+ Synchronizing Repository [epel6_1] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [|] ... completed Downloading repository content... [==================================================] 100% RPMs: 11189/11189 items Delta RPMs: 0/0 items ... completed Individual package errors encountered during sync: An unexpected error has occurred. More information can be found in the client log file ~/.pulp/admin.log. [root@gizmo ~]# pulp-admin rpm repo sync run --repo-id epel6_1 +----------------------------------------------------------------------+ Synchronizing Repository [epel6_1] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [|] ... completed Downloading repository content... [==================================================] 100% RPMs: 1/1 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [\] ... completed Task Succeeded Copying files [\] ... completed Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [-] ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 3639 of 3639 items ... completed Publishing Comps file [==================================================] 100% 211 of 211 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [/] ... completed Writing Listings File [-] ... completed Task Succeeded [root@gizmo ~]# pulp-admin rpm repo create --repo-id epel5 --feed http://mirror.us.leaseweb.net/epel/5/x86_64/ --serve-http true Successfully created repository [epel5] [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# pulp-admin rpm repo sync run --repo-id epel5 +----------------------------------------------------------------------+ Synchronizing Repository [epel5] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [\] ... completed Downloading repository content... [==================================================] 100% RPMs: 6848/6848 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [\] ... completed Importing package groups/categories... [-] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 6848 of 6848 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 1998 of 1998 items ... completed Publishing Comps file [==================================================] 100% 41 of 41 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [\] ... completed Writing Listings File [-] ... completed Task Succeeded [root@gizmo ~]# pulp-admin rpm repo create --repo-id epel5-copySuccessfully created repository [epel5-copy] [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# pulp-admin rpm repo copy Usage: pulp-admin [SUB_SECTION, ..] COMMAND Description: copies one or more content units between repositories Available Commands: all - copy all content units from one repository to another category - copy package categories from one repository to another distribution - copy distributions from one repository to another environment - copy package environment from one repository to another errata - copy errata from one repository to another group - copy package groups from one repository to another rpm - copy RPMs from one repository to another srpm - copy SRPMs from one repository to another [root@gizmo ~]# pulp-admin rpm repo copy all -f epel5 -t epel5-copy This command may be exited via ctrl+c without affecting the request. [\] Running... Copied: package_group: 36 package_category: 5 rpm: 6848 yum_repo_metadata_file: 1 erratum: 1998 [root@gizmo ~]# pulp-admin rpm repo publish run --repo-id epel5-copy +----------------------------------------------------------------------+ Publishing Repository [epel5-copy] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 6848 of 6848 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 1998 of 1998 items ... completed Publishing Comps file [==================================================] 100% 41 of 41 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [|] ... completed Writing Listings File [-] ... completed Task Succeeded [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# pulp-admin rpm repo create --repo-id upload Successfully created repository [upload] [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# pulp-admin rpm repo uploads rpm --repo-id upload --dir upload/ -v +----------------------------------------------------------------------+ Unit Upload +----------------------------------------------------------------------+ No files selected for upload [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# [root@gizmo ~]# pulp-admin rpm repo uploads rpm --repo-id upload --dir upload/ -v +----------------------------------------------------------------------+ Unit Upload +----------------------------------------------------------------------+ Extracting necessary metadata for each request... [==================================================] 100% Analyzing: gmp-4.3.1-7.el6_2.2.x86_64.rpm ... completed Files to be uploaded: kde-i18n-Hungarian-3.5.10-11.el6.noarch.rpm xorg-x11-drv-cirrus-1.5.2-2.el6.x86_64.rpm GConf2-2.28.0-6.el6.i686.rpm zd1211-firmware-1.4-4.el6.noarch.rpm festival-1.96-18.el6.x86_64.rpm texlive-texmf-errata-2007-7.1.el6.noarch.rpm krb5-libs-1.10.3-33.el6.i686.rpm libXxf86dga-1.1.4-2.1.el6.x86_64.rpm gmp-4.3.1-7.el6_2.2.x86_64.rpm Creating upload requests on the server... [==================================================] 100% Initializing: gmp-4.3.1-7.el6_2.2.x86_64.rpm ... completed Starting upload of selected units. If this process is stopped through ctrl+c, the uploads will be paused and may be resumed later using the resume command or cancelled entirely using the cancel command. Uploading: kde-i18n-Hungarian-3.5.10-11.el6.noarch.rpm [==================================================] 100% 1058680/1058680 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: xorg-x11-drv-cirrus-1.5.2-2.el6.x86_64.rpm [==================================================] 100% 34960/34960 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: GConf2-2.28.0-6.el6.i686.rpm [==================================================] 100% 984340/984340 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: zd1211-firmware-1.4-4.el6.noarch.rpm [==================================================] 100% 22664/22664 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: festival-1.96-18.el6.x86_64.rpm [==================================================] 100% 1421920/1421920 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: texlive-texmf-errata-2007-7.1.el6.noarch.rpm [==================================================] 100% 5328/5328 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: krb5-libs-1.10.3-33.el6.i686.rpm [==================================================] 100% 791164/791164 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: libXxf86dga-1.1.4-2.1.el6.x86_64.rpm [==================================================] 100% 18440/18440 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed Uploading: gmp-4.3.1-7.el6_2.2.x86_64.rpm [==================================================] 100% 211560/211560 bytes ... completed Importing into the repository... This command may be exited via ctrl+c without affecting the request. [\] Running... Task Succeeded Deleting the upload request... ... completed [root@gizmo ~]# pulp-admin rpm repo publish run --repo-id upload +----------------------------------------------------------------------+ Publishing Repository [upload] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 9 of 9 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [-] ... completed Publishing Comps file [-] ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [-] ... completed Writing Listings File [-] ... completed