Bug 708279

Here is the full traceback, I started the sync last night, and it was failed with same traceback.



2011-05-27 01:05:54,629 [WARNING][Thread-315] schedule() @ task.py:289 - Task 907e36c5-87a5-11e0-96c1-525400ff61a5: RepoApi._sync(rhel-server-6-updates-6Server-x86_64, synchronizer=<pulp.server.api.repo_sync.YumSynchronizer object at 0x7f16c85f3150>) missed 1 scheduled runs
2011-05-27 01:23:28,271 [INFO][Thread-333] cds_sync() @ cds.py:419 - Synchronizing CDS [dhcp193-65.pnq.redhat.com]
2011-05-27 01:23:29,176 [INFO][Thread-333] sync() @ dispatcher.py:158 - sync using timeout=(10, 7200)
2011-05-27 01:23:30,691 [WARNING][Thread-333] schedule() @ task.py:289 - Task 908de985-87a5-11e0-9728-525400ff61a5: CdsApi.cds_sync(dhcp193-65.pnq.redhat.com, ) missed 1 scheduled runs
2011-05-27 01:46:31,726 [INFO][Thread-251] sync() @ repo_sync.py:458 - Running createrepo, this may take a few minutes to complete.
2011-05-27 01:46:32,488 [INFO][Thread-251] create_repo() @ util.py:377 - metadata found; taking backup.
2011-05-27 01:46:33,078 [INFO][Thread-251] _create_repo() @ util.py:364 - started repo metadata update
2011-05-27 01:46:59,270 [INFO][Thread-334] _sync() @ repo.py:1564 - Sync of rhui-srpms-1.2-5Server-x86_64 starting, skip_dict = {}
2011-05-27 01:46:59,606 [INFO][Thread-334] sync() @ repo_sync.py:414 - cacert = </etc/pki/content/rhui-srpms-1.2-5Server-x86_64/feed-rhui-srpms-1.2-5Server-x86_64.ca>, cert = </etc/pki/content/rhui-srpms-1.2-5Server-x86_64/feed-rhui-srpms-1.2-5Server-x86_64.cert>, key = </etc/pki/content/rhui-srpms-1.2-5Server-x86_64/feed-rhui-srpms-1.2-5Server-x86_64.key>
2011-05-27 01:46:59,643 [INFO][Thread-334] sync() @ repo_sync.py:438 - Limiting download speed to 0 KB/sec per thread. [15] threads will be used
2011-05-27 01:47:51,749 [INFO][Thread-335] _sync() @ repo.py:1564 - Sync of rhui-srpms-1.2-5Server-i386 starting, skip_dict = {}
2011-05-27 01:47:51,750 [INFO][Thread-335] sync() @ repo_sync.py:414 - cacert = </etc/pki/content/rhui-srpms-1.2-5Server-i386/feed-rhui-srpms-1.2-5Server-i386.ca>, cert = </etc/pki/content/rhui-srpms-1.2-5Server-i386/feed-rhui-srpms-1.2-5Server-i386.cert>, key = </etc/pki/content/rhui-srpms-1.2-5Server-i386/feed-rhui-srpms-1.2-5Server-i386.key>
2011-05-27 01:47:51,750 [INFO][Thread-335] sync() @ repo_sync.py:438 - Limiting download speed to 0 KB/sec per thread. [15] threads will be used
2011-05-27 01:48:26,001 [INFO][Thread-334] sync() @ repo_sync.py:458 - Running createrepo, this may take a few minutes to complete.
2011-05-27 01:48:26,340 [INFO][Thread-334] create_repo() @ util.py:377 - metadata found; taking backup.
2011-05-27 01:48:26,553 [INFO][Thread-334] _create_repo() @ util.py:357 - Checking what metadata types are available: ['filelists', 'other', 'primary']
2011-05-27 01:48:26,588 [INFO][Thread-334] _create_repo() @ util.py:364 - started repo metadata update
2011-05-27 01:48:31,742 [INFO][Thread-335] sync() @ repo_sync.py:458 - Running createrepo, this may take a few minutes to complete.
2011-05-27 01:48:31,743 [INFO][Thread-335] create_repo() @ util.py:377 - metadata found; taking backup.
2011-05-27 01:48:31,770 [INFO][Thread-335] _create_repo() @ util.py:357 - Checking what metadata types are available: ['filelists', 'other', 'primary']
2011-05-27 01:48:31,790 [INFO][Thread-335] _create_repo() @ util.py:364 - started repo metadata update
2011-05-27 01:48:54,739 [INFO][Thread-335] _create_repo() @ util.py:370 - [createrepo --database --checksum sha256 --update /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/i386/rhui/1.2/source/SRPMS ] on /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/i386/rhui/1.2/source/SRPMS finished
2011-05-27 01:48:54,772 [INFO][Thread-334] _create_repo() @ util.py:370 - [createrepo --database --checksum sha256 --update /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/x86_64/rhui/1.2/source/SRPMS ] on /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/x86_64/rhui/1.2/source/SRPMS finished
2011-05-27 01:48:54,954 [INFO][Thread-335] sync() @ repo_sync.py:472 - Createrepo finished in 23.2118370533 seconds
2011-05-27 01:48:55,005 [INFO][Thread-335] sync() @ repo_sync.py:474 - YumSynchronizer reported 13 successes, 0 downloads, 0 errors
2011-05-27 01:48:55,229 [INFO][Thread-334] sync() @ repo_sync.py:472 - Createrepo finished in 29.2278420925 seconds
2011-05-27 01:48:55,229 [INFO][Thread-334] sync() @ repo_sync.py:474 - YumSynchronizer reported 13 successes, 0 downloads, 0 errors
2011-05-27 01:49:16,709 [INFO][Thread-335] _process_repo_images() @ repo_sync.py:231 - No image files to import to repo..
2011-05-27 01:49:16,857 [INFO][Thread-335] add_packages_from_dir() @ repo_sync.py:224 - Skipping errata imports from sync process
2011-05-27 01:49:16,857 [INFO][Thread-335] _sync() @ repo.py:1577 - Sync returned 13 packages, 0 errata in 85.1071350574 seconds
2011-05-27 01:49:17,068 [INFO][Thread-334] _process_repo_images() @ repo_sync.py:231 - No image files to import to repo..
2011-05-27 01:49:17,152 [INFO][Thread-334] add_packages_from_dir() @ repo_sync.py:224 - Skipping errata imports from sync process
2011-05-27 01:49:17,470 [INFO][Thread-334] _sync() @ repo.py:1577 - Sync returned 13 packages, 0 errata in 137.864296913 seconds
2011-05-27 01:49:18,467 [INFO][Thread-335] _sync() @ repo.py:1588 - 0 old packages to process, 0 new packages to process
2011-05-27 01:49:19,181 [INFO][Thread-334] _sync() @ repo.py:1588 - 0 old packages to process, 0 new packages to process
2011-05-27 01:49:20,742 [INFO][Thread-335] _sync() @ repo.py:1602 - Examining 0 errata from repo rhui-srpms-1.2-5Server-i386
2011-05-27 01:49:20,755 [INFO][Thread-334] _sync() @ repo.py:1602 - Examining 0 errata from repo rhui-srpms-1.2-5Server-x86_64
2011-05-27 01:49:20,856 [INFO][Thread-335] _sync() @ repo.py:1606 - Removing 0 old errata from repo rhui-srpms-1.2-5Server-i386
2011-05-27 01:49:20,881 [INFO][Thread-334] _sync() @ repo.py:1606 - Removing 0 old errata from repo rhui-srpms-1.2-5Server-x86_64
2011-05-27 01:49:21,693 [INFO][Thread-335] _sync() @ repo.py:1615 - Adding 0 new errata to repo rhui-srpms-1.2-5Server-i386
2011-05-27 01:49:21,862 [INFO][Thread-334] _sync() @ repo.py:1615 - Adding 0 new errata to repo rhui-srpms-1.2-5Server-x86_64
2011-05-27 01:49:22,652 [WARNING][Thread-335] schedule() @ task.py:289 - Task 907df10a-87a5-11e0-96c0-525400ff61a5: RepoApi._sync(rhui-srpms-1.2-5Server-i386, synchronizer=<pulp.server.api.repo_sync.YumSynchronizer object at 0x7f16c85f3150>) missed 1 scheduled runs
2011-05-27 01:49:22,654 [WARNING][Thread-334] schedule() @ task.py:289 - Task 907baee8-87a5-11e0-96b1-525400ff61a5: RepoApi._sync(rhui-srpms-1.2-5Server-x86_64, synchronizer=<pulp.server.api.repo_sync.YumSynchronizer object at 0x7f16c85f3150>) missed 1 scheduled runs
2011-05-27 01:49:30,692 [INFO][Thread-366] _sync() @ repo.py:1564 - Sync of rhel-server-6-updates-6Server-x86_64 starting, skip_dict = {}
2011-05-27 01:49:30,692 [INFO][Thread-366] sync() @ repo_sync.py:414 - cacert = </etc/pki/content/rhel-server-6-updates-6Server-x86_64/feed-rhel-server-6-updates-6Server-x86_64.ca>, cert = </etc/pki/content/rhel-server-6-updates-6Server-x86_64/feed-rhel-server-6-updates-6Server-x86_64.cert>, key = </etc/pki/content/rhel-server-6-updates-6Server-x86_64/feed-rhel-server-6-updates-6Server-x86_64.key>
2011-05-27 01:49:30,717 [INFO][Thread-366] sync() @ repo_sync.py:438 - Limiting download speed to 0 KB/sec per thread. [15] threads will be used
2011-05-27 02:01:58,960 [INFO][Thread-366] sync() @ repo_sync.py:458 - Running createrepo, this may take a few minutes to complete.
2011-05-27 02:01:59,117 [INFO][Thread-366] create_repo() @ util.py:377 - metadata found; taking backup.
2011-05-27 02:02:01,986 [INFO][Thread-366] _create_repo() @ util.py:364 - started repo metadata update
2011-05-27 02:02:34,084 [ERROR][Thread-366] _create_repo() @ util.py:368 - createrepo on /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os failed
2011-05-27 02:02:36,595 [INFO][Thread-366] run() @ task.py:371 - Task id:907e36c5-87a5-11e0-96c1-525400ff61a5, method_name:_sync Exception: sh: line 1: 30102 Killed                  createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os
2011-05-27 02:02:38,107 [ERROR][Thread-366] failed() @ task.py:412 - Task id:907e36c5-87a5-11e0-96c1-525400ff61a5, method_name:_sync:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 355, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo.py", line 1574, in _sync
    threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 116, in sync
    progress_callback, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 470, in sync
    pulp.server.util.create_repo(store_path, groups=groups_xml_path, checksum_type=repo['checksum_type'])
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 385, in create_repo
    _create_repo(dir, groups=groups, checksum_type=checksum_type)
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 369, in _create_repo
    raise CreateRepoError(out)
CreateRepoError: sh: line 1: 30102 Killed                  createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os

added to rhui-20 tracker

Refiled upstream: 713776

Got same CreateRepoError with new build ( pulp 0.194 and rhui-tools 2.0.32)

2011-06-23 13:55:48,800 3970:140208043001616: pulp.server.util:ERROR: util:366 createrepo on /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os failed
2011-06-23 13:55:52,036 3970:140208043001616: pulp.server.tasking.task:ERROR: task:380 Task failed: Task b55df6f8-9cef-11e0-a722-525400734a2e: _sync(rhel-server-6-updates-6Server-x86_64, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f84b45c93d0>, skip_dict={}, max_speed=None, threads=None)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 329, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 224, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 302, in fetch_content
    progress_callback, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 448, in sync
    pulp.server.util.create_repo(store_path, groups=groups_xml_path, checksum_type=repo['checksum_type'])
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 383, in create_repo
    _create_repo(dir, groups=groups, checksum_type=checksum_type)
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 367, in _create_repo
    raise CreateRepoError(out)
CreateRepoError: sh: line 1:  5127 Killed                  createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os


Not sure if this is the issue with rhel6 upadates (x86_64) repo, because earlier I got this against same repo.

commit 5a5752205cc5ad36ac3c81b97c350e7d73bca37b
Author: Pradeep Kilambi <pkilambi>
Date:   Thu Jun 23 08:29:02 2011 -0400

    Adding a preserve metadata as an option at repo creation time. More info about feature

src/pulp/client/api/repository.py
src/pulp/client/core/repo.py
src/pulp/server/api/repo.py
src/pulp/server/api/repo_sync.py
src/pulp/server/api/synchronizers.py
src/pulp/server/db/migrate/versions/nineteen.py
src/pulp/server/db/model/resource.py
src/pulp/server/db/version.py
src/pulp/server/webservices/controllers/repositories.py

commit b4b19ae6dfee075f98b2733b661dd803724bb1df
Author: Jay Dobies <jason.dobies>
Date:   Thu Jun 23 09:17:18 2011 -0400

    708279 - Don't use createrepo at all in the RHUI world for Red Hat
    repos.

rhui-2.0/tools/src/rhui/tools/pulp_api.py

Fixed in:

Pulp: 0.197
RHUI: 2.0.33

No createrepo errors observed with the builds

[root@ip-10-170-226-4 pulp]# rpm -qav | grep -ie pulp -ie rh-rhui-tools
pulp-client-0.0.197-1.el6.noarch
pulp-common-0.0.197-1.el6.noarch
rh-rhui-tools-2.0.33-1.el6.noarch
pulp-0.0.197-1.el6.noarch



------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Repository Management =-

   l   list repositories currently managed by the RHUI
   i   display detailed information on a repository
   a   add a new Red Hat content repository
   c   create a new custom repository
   d   delete a repository from the RHUI
   u   upload content to a custom repository
   p   list packages in a repository

                         Connected: ip-10-170-226-4.us-west-1.compute.internal
------------------------------------------------------------------------------
rhui (repo) => l

Custom Repositories
  Qpid

Red Hat Repositories
  Red Hat Update Infrastructure 1.2 (RPMs) (5Server-x86_64)
  Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64)
  Red Hat Enterprise Linux Server (RPMs) (5Server-i386)
  Red Hat Enterprise Linux Server (RPMs) (5Server-x86_64)


------------------------------------------------------------------------------
rhui (repo) => logout

[root@ip-10-170-226-4 pulp]# cat pulp.log | grep -i createrepo
2011-06-27 09:54:05,917 2247:139755685672720: pulp.server.util:INFO: util:368 [createrepo --database --checksum sha256 --update /var/lib/pulp//repos/baseos/x86_64/qpid/os ] on /var/lib/pulp//repos/baseos/x86_64/qpid/os finished
2011-06-27 12:09:57,912 2247:139755675182864: pulp.server.util:INFO: util:368 [createrepo --database --checksum sha256 --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os ] on /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os finished
2011-06-27 12:09:59,245 2247:139755675182864: pulp.server.util:INFO: util:368 [createrepo --database --checksum sha256 --update /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/x86_64/rhui/1.2/os ] on /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/x86_64/rhui/1.2/os finished
2011-06-27 13:52:48,045 2247:139755675182864: pulp.server.util:INFO: util:368 [createrepo --database --checksum sha256 --update /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/i386/os ] on /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/i386/os finished
2011-06-27 13:52:49,277 2247:139755675182864: pulp.server.util:INFO: util:368 [createrepo --database --checksum sha256 --update /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/x86_64/os ] on /var/lib/pulp//repos/content/dist/rhel/rhui/server/5Server/x86_64/os finished



[root@ip-10-170-226-4 pulp]# cat pulp.log | grep -i CreateRepoError
[root@ip-10-170-226-4 pulp]# cat pulp.log | grep  CreateRepoError

moving to release pending

closing out, product released