Description of problem: Import of EPEL for EL 7 fails with an exception Version-Release number of selected component (if applicable): [root@pulp ~]# cat /etc/redhat-release CentOS Linux release 7.0.1406 (Core) rpm -qa | grep mongo mongodb-org-tools-2.6.6-1.x86_64 mongodb-org-mongos-2.6.6-1.x86_64 mongodb-org-shell-2.6.6-1.x86_64 mongodb-org-2.6.6-1.x86_64 python-pymongo-2.5.2-2.el7.x86_64 mongodb-org-server-2.6.6-1.x86_64 [root@pulp ~]# rpm -qa|grep pulp pulp-rpm-plugins-2.5.1-1.el7.noarch python-pulp-bindings-2.5.1-1.el7.noarch pulp-consumer-client-2.5.1-1.el7.noarch pulp-puppet-handlers-2.5.1-1.el7.noarch pulp-rpm-yumplugins-2.5.1-1.el7.noarch python-isodate-0.5.0-4.pulp.el7.noarch python-pulp-puppet-common-2.5.1-1.el7.noarch python-pulp-common-2.5.1-1.el7.noarch python-kombu-3.0.15-12.pulp.el7.noarch python-pulp-rpm-common-2.5.1-1.el7.noarch pulp-server-2.5.1-1.el7.noarch pulp-selinux-2.5.1-1.el7.noarch python-pulp-client-lib-2.5.1-1.el7.noarch pulp-admin-client-2.5.1-1.el7.noarch pulp-rpm-admin-extensions-2.5.1-1.el7.noarch python-pulp-agent-lib-2.5.1-1.el7.noarch pulp-agent-2.5.1-1.el7.noarch pulp-rpm-handlers-2.5.1-1.el7.noarch pulp-rpm-consumer-extensions-2.5.1-1.el7.noarch pulp-puppet-plugins-2.5.1-1.el7.noarch pulp-puppet-admin-extensions-2.5.1-1.el7.noarch pulp-puppet-consumer-extensions-2.5.1-1.el7.noarch How reproducible: Steps to Reproduce: 1. tail -f /var/log/messages 2. pulp-admin rpm repo create --repo-id 'EPEL_7_64' --display-name 'EPEL 7 x86_64 Repository' --feed 'https://dl.fedoraproject.org/pub/epel/7/x86_64/' --validate true --serve-https true --generate-sqlite true 3. pulp-admin rpm repo sync run --repo-id 'EPEL_7_64' Actual results: from /var/log/messages: Jan 2 12:56:01 pulp pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {'___/primary/___': {'total_failed': 0, 'total_succeeded': 6895}}, 'total_sources': 0} for base URL: https://dl.fedoraproject.org/pub/epel/7/x86_64/ Jan 2 12:56:01 pulp pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units. Jan 2 12:56:01 pulp pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org Jan 2 12:56:02 pulp pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: Content unit association failed [Unit [key={'id': 'FEDORA-EPEL-2014-3685'}] [type=erratum] [id=None]] Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: Traceback (most recent call last): Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 480, in save_unit Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: unit.id = self._update_unit(unit, pulp_unit) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 512, in _update_unit Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: return self._add_unit(unit, pulp_unit) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 534, in _add_unit Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: unit_id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/cud.py", line 35, in add_content_unit Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: collection.insert(unit_doc, safe=True) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 144, in retry Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: return method(*args, **kwargs) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 357, in insert Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: continue_on_error, self.__uuid_subtype), safe) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 167, in _with_end_request Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: return method(*args, **kwargs) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 920, in _send_message Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: rv = self.__check_response_to_last_error(response) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 863, in __check_response_to_last_error Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: raise OperationFailure(details["err"], details["code"]) Jan 2 12:56:05 pulp pulp: pulp.plugins.conduits.mixins:ERROR: OperationFailure: insertDocument :: caused by :: 17280 Btree::insert: key too large to index, failing pulp_database.units_erratum.$title_1 1408 { : "globus-common-15.26-1.el7,globus-gsi-proxy-ssl-5.7-1.el7,globus-gsi-openssl-error-3.5-1.el7,globus-openssl-module-4.6-1.el7,globus-gsi-cert-utils-9.10..." } Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: Traceback (most recent call last): Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 134, in run Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: self.get_errata(metadata_files) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 470, in get_errata Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: self.save_fileless_units(errata_file_handle, updateinfo.PACKAGE_TAG, updateinfo.process_package_element) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 541, in save_fileless_units Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: self.sync_conduit.save_unit(unit) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 480, in save_unit Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: unit.id = self._update_unit(unit, pulp_unit) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 512, in _update_unit Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: return self._add_unit(unit, pulp_unit) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 534, in _add_unit Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: unit_id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/cud.py", line 35, in add_content_unit Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: collection.insert(unit_doc, safe=True) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 144, in retry Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: return method(*args, **kwargs) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 357, in insert Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: continue_on_error, self.__uuid_subtype), safe) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 167, in _with_end_request Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: return method(*args, **kwargs) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 920, in _send_message Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: rv = self.__check_response_to_last_error(response) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 863, in __check_response_to_last_error Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: raise OperationFailure(details["err"], details["code"]) Jan 2 12:56:05 pulp pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: ImporterConduitException: insertDocument :: caused by :: 17280 Btree::insert: key too large to index, failing pulp_database.units_erratum.$title_1 1408 { : "globus-common-15.26-1.el7,globus-gsi-proxy-ssl-5.7-1.el7,globus-gsi-openssl-error-3.5-1.el7,globus-openssl-module-4.6-1.el7,globus-gsi-cert-utils-9.10..." } Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: Task pulp.server.tasks.repository.sync_with_auto_publish[c7d9c6be-424f-4074-a5dd-a031f0d90233] raised unexpected: PulpExecutionException('Importer indicated a failed response',) Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: Traceback (most recent call last): Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: R = retval = fun(*args, **kwargs) Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 315, in __call__ Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: return super(Task, self).__call__(*args, **kwargs) Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: return self.run(*args, **kwargs) Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/tasks/repository.py", line 210, in sync_with_auto_publish Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: sync_result = managers.repo_sync_manager().sync(repo_id, sync_config_override=overrides) Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 113, in sync Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: raise PulpExecutionException(_('Importer indicated a failed response')) Jan 2 12:56:05 pulp pulp: celery.worker.job:ERROR: PulpExecutionException: Importer indicated a failed response Expected results: working EPEL repository Additional info: may be related with: https://bugzilla.redhat.com/show_bug.cgi?id=1161277
Hi, I just verified your steps and was able to sync & publish epel properly. It appears that they have fixed the errata with the overly long title. Pulp has a separate bug to handle this case at https://bugzilla.redhat.com/show_bug.cgi?id=1175616
Just realized I was testing on an older version of Mongo that doesn't show this issue. You should be able to work around this issue with by opening the mongo shell and dropping the title index from the units_erratum collection: # mongo pulp_database # db.units_erratum.dropIndex("title_1") *** This bug has been marked as a duplicate of bug 1175616 ***