Bug 713776
| Summary: | CreateRepoError: when sync was in "running" state for two repos | ||
|---|---|---|---|
| Product: | [Retired] Pulp | Reporter: | Jay Dobies <jason.dobies> |
| Component: | z_other | Assignee: | Pradeep Kilambi <pkilambi> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | unspecified | CC: | sghai, skarmark, tsanders |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | Sprint 25 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-08-16 12:06:51 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 647488, 688298 | ||
Reassigning to Prad. Based on my initial investigation its createrepo that's exiting. I can reproduce this outside of pulp as well.. $ sudo createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/repodata/1a7fc54d30d0d44222742c63069ab0126afef9f160931cc15e564dbe6414f268-comps-rhel6-Server.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os Spawning worker 0 with 3391 pkgs Worker 0: Worker 0: Traceback (most recent call last): Worker 0: iso-8859-1 encoding on Ville Skytt� <ville.skytta> - 2.8.2-2 Worker 0: File "/usr/share/createrepo/worker.py", line 101, in <module> Worker exited with non-zero value: 1. Fatal. Worker exited with non-zero value: 1. Fatal. You have new mail in /var/spool/mail/pkilambi In another window: $ sudo createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os Spawning worker 0 with 5043 pkgs Cannot access/write repodata files: [Errno 12] Cannot allocate memory The out of memory error is essentially kills the createrepo process and results in CreateRepoError: sh: line 1: 28480 Killed createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os Got same CreateRepoError with new build ( pulp 0.194 and rhui-tools 2.0.32)
2011-06-23 13:55:48,800 3970:140208043001616: pulp.server.util:ERROR: util:366
createrepo on
/var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os
failed
2011-06-23 13:55:52,036 3970:140208043001616: pulp.server.tasking.task:ERROR:
task:380 Task failed: Task b55df6f8-9cef-11e0-a722-525400734a2e:
_sync(rhel-server-6-updates-6Server-x86_64,
synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at
0x7f84b45c93d0>, skip_dict={}, max_speed=None, threads=None)
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line
329, in run
result = self.callable(*self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line
224, in _sync
progress_callback, synchronizer, max_speed, threads)
File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line
302, in fetch_content
progress_callback, max_speed, threads)
File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py",
line 448, in sync
pulp.server.util.create_repo(store_path, groups=groups_xml_path,
checksum_type=repo['checksum_type'])
File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 383, in
create_repo
_create_repo(dir, groups=groups, checksum_type=checksum_type)
File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 367, in
_create_repo
raise CreateRepoError(out)
CreateRepoError: sh: line 1: 5127 Killed createrepo
--database --checksum sha256 -g
/var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml
--update
/var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os
Not sure if this is the issue with rhel6 upadates (x86_64) repo, because
earlier I got this against same repo.
build: 0.198 verified [root@preethi ~]# rpm -q pulp pulp-0.0.223-1.fc14.noarch [root@preethi ~]# 1. created and synced f14-updates repo with --preserve_metadata 2. created an f14 repo with --preserve_metadata 3. sync f14-updates again while its syncing syncing f14 [root@preethi ~]# pulp-admin -u admin -p admin repo create --id=f14 --feed=http://download.devel.redhat.com/released/F-14/GOLD/Fedora/x86_64/os --preserve_metadata Successfully created repository [ f14 ] [root@preethi ~]# pulp-admin repo sync --id=f14 -FSync for repository f14 started Sync: Finished 2773/2773 new items downloaded 0/2773 existing items processed Item Details: Tree_Files: 7/7 Rpms: 2766/2766 [root@preethi ~]# pulp-admin repo sync --id=f14-updates -FSync for repository f14-updates started Sync: Finished 9214/17069 new items downloaded 7855/17069 existing items processed Item Details: Rpms: 9214/9214 Delta_Rpms: 7855/7855 [root@preethi ~]# pulp-admin repo sync --id=f14-updates -F Sync for repository f14-updates started Sync: Finished 0/17069 new items downloaded 17069/17069 existing items processed Item Details: Rpms: 9214/9214 Delta_Rpms: 7855/7855 Closing with Community Release 15 pulp-0.0.223-4. Closing with Community Release 15 pulp-0.0.223-4. |
Refiled upstream from the RHUI project: 708279 Repo sync was running for rhel6 updates and parallely I started a sync of new repo rhel6 release. Sync was running for both repos for few minutes and then I got a traceback in pulp.log. 2011-05-26 19:22:33,735 [ERROR][Thread-71] _create_repo() @ util.py:368 - createrepo on /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os failed 2011-05-26 19:22:36,971 [INFO][Thread-71] run() @ task.py:371 - Task id:a55ab257-8781-11e0-b17f-525400ff61a5, method_name:_sync Exception: sh: line 1: 28480 Killed createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os 2011-05-26 19:22:39,294 [ERROR][Thread-71] failed() @ task.py:412 - Task id:a55ab257-8781-11e0-b17f-525400ff61a5, method_name:_sync: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 355, in run result = self.callable(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo.py", line 1574, in _sync threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 116, in sync progress_callback, max_speed, threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 470, in sync pulp.server.util.create_repo(store_path, groups=groups_xml_path, checksum_type=repo['checksum_type']) File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 385, in create_repo _create_repo(dir, groups=groups, checksum_type=checksum_type) File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 369, in _create_repo raise CreateRepoError(out) CreateRepoError: sh: line 1: 28480 Killed createrepo --database --checksum sha256 -g /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/repodata/comps-rhel-x86_64-server-6.xml --update /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os Version-Release number of selected component (if applicable): rhui 2.0.25 pulp 0.181 How reproducible: Always Steps to Reproduce: 1. Added a new repo of rhel6 (release) while sync is running for rhel6 (updates) 2. 3. Actual results: Got traceback as mentioned above. Expected results: There shouldn't be any error in logs; and sync should finish properly. Additional info: Scheduled the sync for newly added repo (rhel6 releases) ======================================================== -= Repository Synchronization Status =- Last Refreshed: 18:57:29 (updated every 50 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64) 05-26-2011 19:46 Never scheduled Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 05-26-2011 16:49 Never running Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-i386) 05-26-2011 19:46 05-26-2011 18:47 finished Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-x86_64) 05-26-2011 19:46 05-26-2011 18:47 finished Connected: dhcp193-79.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) => dr Sync is running for both repos: ===================================== ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Repository Synchronization Status =- Last Refreshed: 20:04:48 (updated every 50 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64) 05-26-2011 19:46 Never running Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 05-26-2011 19:49 Never running Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-i386) 05-26-2011 20:46 05-26-2011 19:48 finished Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-x86_64) 05-26-2011 20:46 05-26-2011 19:48 finished Connected: dhcp193-79.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) => dr when the error message appeared in pulp.log; sync status changed to "scheduled" ============================================================================== ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Repository Synchronization Status =- Last Refreshed: 20:06:37 (updated every 50 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 6 Releases (RPMs) (6Server-x86_64) 05-26-2011 20:46 Never scheduled Red Hat Enterprise Linux Server 6 Updates (RPMs) (6Server-x86_64) 05-26-2011 20:49 Never scheduled Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-i386) 05-26-2011 20:46 05-26-2011 19:48 scheduled Red Hat Update Infrastructure 1.2 (SRPMS) (5Server-x86_64) 05-26-2011 20:46 05-26-2011 19:48 scheduled Connected: dhcp193-79.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) =>