Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

It appears that there are too many sync result entries for mongo to sort without an index. I suspect that if we add an index to the "started" field on the "repo_sync_results" collection, this problem will go away.

A simpler work-around would be to manually remove entries from that collection. That collection periodically has old entries removed anyway, so no harm should come from removing them manually.

I'm CCing Randy Barlow, who can give more specific advice on how to proceed.

Code for reference: https://github.com/pulp/pulp/blob/2.4-release/server/pulp/server/managers/repo/sync.py#L291

0) /etc/pulp/server.conf

I tried this option however i still get the sort problem

There is not a way to directly force the reaper to run that doesn't involve opening a python shell. However, the reaper's period can be adjusted. It defaults to a period of 0.25 days (i.e., four times per day). One way to get it to run very soon is to reduce the period. For example, 0.0006944 would be about once per minute. A user could set the value very low, restart Pulp, wait for the reaper to run, and then return it to a more sane value (the default is good). If this technique were done in combination with reducing the number of days of history to keep, I believe this will solve the immediate issue.

Can we please try the fix on comment 8, and if not describe roughly the content size which is causing the issue? Thanks!

I can confirm that comment 8 didn't work for me.  I used 0.0006944 and also bumped the archived days for each section down from "60" to "2".  Still get the below:


Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024) Exception caught from plugin during publish for repo [rhel-server-rhscl-6-rpms-6Server-x86_64]
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024) Traceback (most recent call last):
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/publish.py", line 127, in _do_publish
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 495, in wrap_f
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     return f(*args, **kwargs)
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/export_distributor/distributor.py", line 136, in publish_repo
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     return self._publisher.publish()
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 558, in publish
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     return self.process_lifecycle()
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 503, in process_lifecycle
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     super(PluginStep, self).process_lifecycle()
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 127, in process_lifecycle
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     step.process()
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 204, in process
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     self._process_block()
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 669, in _process_block
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     for package_unit in package_unit_generator:
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/plugins/conduits/mixins.py", line 704, in _transfer_object_generator
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     for u in units:
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association_query.py", line 572, in _merged_units_unique_units
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     for unit in associated_units:
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/unit_association_query.py", line 499, in _units_from_chained_cursors
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     for element in cursor:
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 814, in next
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     if len(self.__data) or self._refresh():
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 763, in _refresh
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     self.__uuid_subtype))
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 720, in __send_message
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     self.__uuid_subtype)
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 100, in _unpack_response
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024)     error_object["$err"])
Dec 10 16:00:15 kat pulp: pulp.server.managers.repo.publish:ERROR: (1527-57024) OperationFailure: database error: too much data for sort() with no index.  add an index or specify a smaller limit

... but adding the index as in comment 2 seems to be working so far.  I'll continue to test and update this bug if anything errors out in the future.

Thanks,
Taylor

Unfortunately I'm still getting the errors, but on different tables now:

query pulp_database.units_rpm query

Is there an index I can add to this table too?

Unfortunately I'm still getting the errors, but on different tables now:

query pulp_database.units_rpm query

Is there an index I can add to this table too?

Found fix, Added Bug 1290952 since this issue is related but not identical.

Reset docs contact <> daobrien

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

This was fixed in pulp 2.5, so you should see this fix in Satellite 6.2. If you are still seeing this issue, please feel free to re-open this bug and provide additional data.