Refiled upstream from the RHUI project: 708279 Repo sync was running for rhel6 updates and parallely I started a sync of new repo rhel6 release. Sync was running for both repos for few minutes and then I got a traceback in pulp.log. 2011-05-26 19:22:33,735 [ERROR][Thread-71] _create_repo() @ util.py:368 - createrepo on /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os failed 2011-05-26 19:22:36,971 [INFO][Thread-71] run() @ task.py:371 - Task id:a55ab257-8781-11e0-b17f-525400ff61a5, method_name:_sync Exception: sh: line 1: 28480 Killed createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os 2011-05-26 19:22:39,294 [ERROR][Thread-71] failed() @ task.py:412 - Task id:a55ab257-8781-11e0-b17f-525400ff61a5, method_name:_sync: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 355, in run result = self.callable(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo.py", line 1574, in _sync threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 116, in sync progress_callback, max_speed, threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 470, in sync pulp.server.util.create_repo(store_path, groups=groups_xml_path, checksum_type=repo['checksum_type']) File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 385, in create_repo _create_repo(dir, groups=groups, checksum_type=checksum_type) File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 369, in _create_repo raise CreateRepoError(out) CreateRepoError: sh: line 1: 28480 Killed createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os Version-Release number of selected component (if applicable): rhui 2.0.25 pulp 0.181 How reproducible: Always Steps to Reproduce: 1. Added a new repo of rhel6 (release) while sync is running for rhel6 (updates) 2. 3. Actual results: Got traceback as mentioned above. Expected results: There shouldn't be any error in logs; and sync should finish properly. Additional info: Scheduled the sync for newly added repo (rhel6 releases) ======================================================== -= Repository Synchronization Status =- Last Refreshed: 18:57:29 (updated every 50 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64) 05-26-2011 19:46 Never scheduled Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 05-26-2011 16:49 Never running Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-i386) 05-26-2011 19:46 05-26-2011 18:47 finished Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-x86_64) 05-26-2011 19:46 05-26-2011 18:47 finished Connected: dhcp193-79.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) => dr Sync is running for both repos: ===================================== ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Repository Synchronization Status =- Last Refreshed: 20:04:48 (updated every 50 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64) 05-26-2011 19:46 Never running Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 05-26-2011 19:49 Never running Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-i386) 05-26-2011 20:46 05-26-2011 19:48 finished Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-x86_64) 05-26-2011 20:46 05-26-2011 19:48 finished Connected: dhcp193-79.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) => dr when the error message appeared in pulp.log; sync status changed to "scheduled" ============================================================================== ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Repository Synchronization Status =- Last Refreshed: 20:06:37 (updated every 50 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64) 05-26-2011 20:46 Never scheduled Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 05-26-2011 20:49 Never scheduled Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-i386) 05-26-2011 20:46 05-26-2011 19:48 scheduled Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-x86_64) 05-26-2011 20:46 05-26-2011 19:48 scheduled Connected: dhcp193-79.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) =>
Reassigning to Prad.
Based on my initial investigation its createrepo that's exiting. I can reproduce this outside of pulp as well.. $ sudo createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/repodata/1a7fc54d30d0d44222742c63069ab0126afef9f160931cc15e564dbe6414f268-comps-rhel6-Server.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os Spawning worker 0 with 3391 pkgs Worker 0: Worker 0: Traceback (most recent call last): Worker 0: iso-8859-1 encoding on Ville Skytt� <ville.skytta> - 2.8.2-2 Worker 0: File "/usr/share/createrepo/worker.py", line 101, in <module> Worker exited with non-zero value: 1. Fatal. Worker exited with non-zero value: 1. Fatal. You have new mail in /var/spool/mail/pkilambi In another window: $ sudo createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os Spawning worker 0 with 5043 pkgs Cannot access/write repodata files: [Errno 12] Cannot allocate memory The out of memory error is essentially kills the createrepo process and results in CreateRepoError: sh: line 1: 28480 Killed createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os
Got same CreateRepoError with new build ( pulp 0.194 and rhui-tools 2.0.32) 2011-06-23 13:55:48,800 3970:140208043001616: pulp.server.util:ERROR: util:366 createrepo on /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os failed 2011-06-23 13:55:52,036 3970:140208043001616: pulp.server.tasking.task:ERROR: task:380 Task failed: Task b55df6f8-9cef-11e0-a722-525400734a2e: _sync(rhel-server-6-updates-6Server-x86_64, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f84b45c93d0>, skip_dict={}, max_speed=None, threads=None) Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 329, in run result = self.callable(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 224, in _sync progress_callback, synchronizer, max_speed, threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 302, in fetch_content progress_callback, max_speed, threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 448, in sync pulp.server.util.create_repo(store_path, groups=groups_xml_path, checksum_type=repo['checksum_type']) File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 383, in create_repo _create_repo(dir, groups=groups, checksum_type=checksum_type) File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 367, in _create_repo raise CreateRepoError(out) CreateRepoError: sh: line 1: 5127 Killed createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os Not sure if this is the issue with rhel6 upadates (x86_64) repo, because earlier I got this against same repo.
build: 0.198
verified [root@preethi ~]# rpm -q pulp pulp-0.0.223-1.fc14.noarch [root@preethi ~]# 1. created and synced f14-updates repo with --preserve_metadata 2. created an f14 repo with --preserve_metadata 3. sync f14-updates again while its syncing syncing f14 [root@preethi ~]# pulp-admin -u admin -p admin repo create --id=f14 --feed=http://download.devel.redhat.com/released/F-14/GOLD/Fedora/x86_64/os --preserve_metadata Successfully created repository [ f14 ] [root@preethi ~]# pulp-admin repo sync --id=f14 -FSync for repository f14 started Sync: Finished 2773/2773 new items downloaded 0/2773 existing items processed Item Details: Tree_Files: 7/7 Rpms: 2766/2766 [root@preethi ~]# pulp-admin repo sync --id=f14-updates -FSync for repository f14-updates started Sync: Finished 9214/17069 new items downloaded 7855/17069 existing items processed Item Details: Rpms: 9214/9214 Delta_Rpms: 7855/7855 [root@preethi ~]# pulp-admin repo sync --id=f14-updates -F Sync for repository f14-updates started Sync: Finished 0/17069 new items downloaded 17069/17069 existing items processed Item Details: Rpms: 9214/9214 Delta_Rpms: 7855/7855
Closing with Community Release 15 pulp-0.0.223-4.