Hide Forgot
Created attachment 835368 [details] rhua pulp sync error Description: having updated both the system and rhui, RHEL repo sync of RHEL6 stopped working; pulp.log shows error as attached. Version: 2.1.2 -> 2.1.3 12/10 ISO & RHEL 6.4 -> RHEL 6.5 How reproducible: 1/1 Steps to reproduce: 1. deploy RHUI 2.1.2 rhua + 2 cdses 2. sync RHEL 6 server channel onto rhua 3. create custom repo with 2.1.3 12/10 packages 4. create cds cluster of cds1 and cds2 nodes 5. associate custom repo and RHEL 6 server repo with the cluster 6. sync cluster 7. create entitlement certs for both the repos 8. create client configuration packages for the repos and the cluster 9. disable all redhat content sources on RHUA & CDS nodes 10. install client config packages on RHUA & CDS nodes 11. yum update 12. reboot 13. check /var/log/pulp/pulp.log on rhua node
Created attachment 835371 [details] update log and pulp log Attaching update and pulp log
#Traceback root@rhua ~]# tail -25 /var/log/pulp/pulp.log 2013-12-11 07:20:29,265 1356:140432832939776: pulp.server.api.synchronizers:INFO: synchronizers:383 Loading comps group info from: /var/lib/pulp//repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os/repodata/877f91138d7aa17294b02f547bc340f217a821bf-comps-rhel-x86_64-server-6.xml 2013-12-11 07:20:29,425 1356:140432832939776: pulp.server.api.synchronizers:INFO: synchronizers:405 updateinfo is found in repomd.xml, it's path is /var/lib/pulp//repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os/repodata/b1e5a76d58a467e846cbe991880d2c6eac9eff9a7c592027f496d9dfdfb01463-updateinfo.xml.gz 2013-12-11 07:20:48,556 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:275 Sync returned 12046 packages, 2239 errata in 506.798788071 seconds 2013-12-11 07:20:48,849 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:286 0 old packages to process, 14 new packages to process 2013-12-11 07:20:52,404 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:301 Examining 2237 errata from repo rhel-6-server-rhui-rpms-6Server-x86_64 2013-12-11 07:20:54,994 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:305 Removing 0 old errata from repo rhel-6-server-rhui-rpms-6Server-x86_64 2013-12-11 07:20:56,486 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:314 Adding 2 new errata to repo rhel-6-server-rhui-rpms-6Server-x86_64 2013-12-11 07:20:57,283 1356:140432832939776: pulp.server.tasking.task:INFO: task:460 Task succeeded: Task 50916d73-6256-11e3-b526-22000afc058d: _sync(rhel-6-server-rhui-rpms-6Server-x86_64, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7fb94a37fcd0>, skip=SON([(u'distribution', 1)]), progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7fb94a37fe90>>) 2013-12-11 07:21:01,440 1356:140432832939776: pulp.server.tasking.task:ERROR: task:492 document too large - BSON documents are limited to 4 MB Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 490, in _complete self.complete_callback(self) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 333, in complete self.__storage.store_complete(task) File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 244, in store_complete self.history_collection.insert(history) File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 80, in retry return method(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 270, in insert check_keys, safe, kwargs), safe) File "/usr/lib64/python2.6/site-packages/pymongo/message.py", line 67, in insert bson_data = "".join([bson.BSON.encode(doc, check_keys) for doc in docs]) File "/usr/lib64/python2.6/site-packages/bson/__init__.py", line 462, in encode return cls(_dict_to_bson(document, check_keys)) InvalidDocument: document too large - BSON documents are limited to 4 MB [root@rhua ~]# rhui-manager
We need to discover why there were so many grinder errors. The issue with the BSON documents being limited to 4MB is a hard limitation of the version of MongoDB we are using in RHUI. Typically documents are much smaller than this, yet for the example shown in this BZ something abnormal happened and a large number of errors were serialized to be stored with the task, the size exceeded the 4MB limit, hence the error. From CDS1: gofer.log:2013-12-11 10:31:26,208 [INFO][worker-0] _sync_repo() @ cdslib.py:417 - CDS Sync report for [https://rhua.example.com//pulp/repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os]: 14 successes, 14 downloads, 12032 errors
We think BZ was addressed when we fixed the grinder problem in 1040739. http://download.lab.bos.redhat.com/devel/candidates/RHEL-6.4-RHUI-2.1.3-20131212.0/2.1.3/Server/x86_64/iso/
This was indeed fixed with bug #1040739 ; moving to verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1854.html