| Summary: | rhua repo sync fails after update | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Update Infrastructure for Cloud Providers | Reporter: | mkovacik | ||||||
| Component: | RHUA | Assignee: | John Matthews <jmatthew> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | mkovacik | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 2.1.2 | CC: | ipanova, jboutaud, tsanders | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 2.1.3 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2013-12-17 20:11:26 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Bug Depends On: | 1040739 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
Created attachment 835371 [details]
update log and pulp log
Attaching update and pulp log
#Traceback
root@rhua ~]# tail -25 /var/log/pulp/pulp.log
2013-12-11 07:20:29,265 1356:140432832939776: pulp.server.api.synchronizers:INFO: synchronizers:383 Loading comps group info from: /var/lib/pulp//repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os/repodata/877f91138d7aa17294b02f547bc340f217a821bf-comps-rhel-x86_64-server-6.xml
2013-12-11 07:20:29,425 1356:140432832939776: pulp.server.api.synchronizers:INFO: synchronizers:405 updateinfo is found in repomd.xml, it's path is /var/lib/pulp//repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os/repodata/b1e5a76d58a467e846cbe991880d2c6eac9eff9a7c592027f496d9dfdfb01463-updateinfo.xml.gz
2013-12-11 07:20:48,556 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:275 Sync returned 12046 packages, 2239 errata in 506.798788071 seconds
2013-12-11 07:20:48,849 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:286 0 old packages to process, 14 new packages to process
2013-12-11 07:20:52,404 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:301 Examining 2237 errata from repo rhel-6-server-rhui-rpms-6Server-x86_64
2013-12-11 07:20:54,994 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:305 Removing 0 old errata from repo rhel-6-server-rhui-rpms-6Server-x86_64
2013-12-11 07:20:56,486 1356:140432832939776: pulp.server.api.repo_sync:INFO: repo_sync:314 Adding 2 new errata to repo rhel-6-server-rhui-rpms-6Server-x86_64
2013-12-11 07:20:57,283 1356:140432832939776: pulp.server.tasking.task:INFO: task:460 Task succeeded: Task 50916d73-6256-11e3-b526-22000afc058d: _sync(rhel-6-server-rhui-rpms-6Server-x86_64, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7fb94a37fcd0>, skip=SON([(u'distribution', 1)]), progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7fb94a37fe90>>)
2013-12-11 07:21:01,440 1356:140432832939776: pulp.server.tasking.task:ERROR: task:492 document too large - BSON documents are limited to 4 MB
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 490, in _complete
self.complete_callback(self)
File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/queue.py", line 333, in complete
self.__storage.store_complete(task)
File "/usr/lib/python2.6/site-packages/pulp/server/tasking/taskqueue/storage.py", line 244, in store_complete
self.history_collection.insert(history)
File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 80, in retry
return method(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 270, in insert
check_keys, safe, kwargs), safe)
File "/usr/lib64/python2.6/site-packages/pymongo/message.py", line 67, in insert
bson_data = "".join([bson.BSON.encode(doc, check_keys) for doc in docs])
File "/usr/lib64/python2.6/site-packages/bson/__init__.py", line 462, in encode
return cls(_dict_to_bson(document, check_keys))
InvalidDocument: document too large - BSON documents are limited to 4 MB
[root@rhua ~]# rhui-manager
We need to discover why there were so many grinder errors. The issue with the BSON documents being limited to 4MB is a hard limitation of the version of MongoDB we are using in RHUI. Typically documents are much smaller than this, yet for the example shown in this BZ something abnormal happened and a large number of errors were serialized to be stored with the task, the size exceeded the 4MB limit, hence the error. From CDS1: gofer.log:2013-12-11 10:31:26,208 [INFO][worker-0] _sync_repo() @ cdslib.py:417 - CDS Sync report for [https://rhua.example.com//pulp/repos/content/dist/rhel/rhui/server/6/6Server/x86_64/os]: 14 successes, 14 downloads, 12032 errors We think BZ was addressed when we fixed the grinder problem in 1040739. http://download.lab.bos.redhat.com/devel/candidates/RHEL-6.4-RHUI-2.1.3-20131212.0/2.1.3/Server/x86_64/iso/ This was indeed fixed with bug #1040739 ; moving to verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1854.html |
Created attachment 835368 [details] rhua pulp sync error Description: having updated both the system and rhui, RHEL repo sync of RHEL6 stopped working; pulp.log shows error as attached. Version: 2.1.2 -> 2.1.3 12/10 ISO & RHEL 6.4 -> RHEL 6.5 How reproducible: 1/1 Steps to reproduce: 1. deploy RHUI 2.1.2 rhua + 2 cdses 2. sync RHEL 6 server channel onto rhua 3. create custom repo with 2.1.3 12/10 packages 4. create cds cluster of cds1 and cds2 nodes 5. associate custom repo and RHEL 6 server repo with the cluster 6. sync cluster 7. create entitlement certs for both the repos 8. create client configuration packages for the repos and the cluster 9. disable all redhat content sources on RHUA & CDS nodes 10. install client config packages on RHUA & CDS nodes 11. yum update 12. reboot 13. check /var/log/pulp/pulp.log on rhua node