Bug 863187
Summary: | failed to sync: ('Package [%s] does not exist', u'b017e5e0-6d3e-4a9b-b3bb-53f55fc3e209') | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Satellite | Reporter: | James Laska <jlaska> | ||||||||||||
Component: | Content Management | Assignee: | Katello Bug Bin <katello-bugs> | ||||||||||||
Status: | CLOSED ERRATA | QA Contact: | Corey Welton <cwelton> | ||||||||||||
Severity: | high | Docs Contact: | |||||||||||||
Priority: | unspecified | ||||||||||||||
Version: | 6.0.0 | CC: | asettle, dmacpher, inecas, jturner, mmccune, omaciel | ||||||||||||
Target Milestone: | Unspecified | Keywords: | Triaged | ||||||||||||
Target Release: | Unused | ||||||||||||||
Hardware: | Unspecified | ||||||||||||||
OS: | Unspecified | ||||||||||||||
Whiteboard: | |||||||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||
Doc Text: |
Attempts to sync a RHEL-5.8-i386 repository failed when syncing simultaneous repositories from Red Hat CDN. This fix adds a retry mechanism.
|
Story Points: | --- | ||||||||||||
Clone Of: | |||||||||||||||
: | 865003 (view as bug list) | Environment: | |||||||||||||
Last Closed: | 2012-12-04 19:56:52 UTC | Type: | Bug | ||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||
Documentation: | --- | CRM: | |||||||||||||
Verified Versions: | Category: | --- | |||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
Embargoed: | |||||||||||||||
Bug Depends On: | |||||||||||||||
Bug Blocks: | 865003 | ||||||||||||||
Attachments: |
|
Description
James Laska
2012-10-04 15:57:07 UTC
Created attachment 623430 [details] katello-debug-20121008080416.tar.gz Reproduced using CloudForms snapshot#3 > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [##################################################] 100.0% > > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'8975f49d-eb41-47db-853c-829bde3499dc') > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) > Exiting on user command Full test output available at http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/8281114//TESTOUT.log Once again, inspecting the list of repositories seems to show the faulty repo is RHEL-5.8-i386. > # katello -u admin -p admin repo list --environment Library > ------------------------------------------------------------------------------------------------------------------------------------------------------------ > Repo List For Org redhat Environment Library > > Id Name Label Package Count Last Sync > ----------------------------------------------------------------------------------------------------------------------------------------------------------- > 967 Red Hat Enterprise Linux 5 Server RPMs i386 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8 10830 2012/10/05 12:50:07 > 965 Red Hat Enterprise Linux 5 Server RPMs i386 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server 0 never > 969 Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8 14092 2012/10/05 14:54:04 > 966 Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server 14092 2012/10/05 14:55:13 > 1053 Red Hat Enterprise Linux 6 Server RPMs i386 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2 5599 2012/10/05 16:01:56 > 1055 Red Hat Enterprise Linux 6 Server RPMs i386 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3 6569 2012/10/05 16:03:26 > 1056 Red Hat Enterprise Linux 6 Server RPMs i386 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server 6569 2012/10/05 16:29:48 > 1050 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2 7281 2012/10/05 16:59:34 > 1052 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3 8560 2012/10/05 17:01:10 > 1051 Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server 8560 2012/10/05 17:20:40 Escalating as this is impacting RHEL5 content sync from cdn.redhat.com. I'm unable to workaround this issue. Re-syncing also fails. I re-installed katello and enabled+synced *only* RHEL 5.8 and 5Server (i386 and x86_64). I was unable to reproduce the problem. will look into this Created attachment 624841 [details] katello-debug-20121010083059.tar.gz (snap#4) Reproduced on snap#4 * katello-1.1.12-14.el6cf * pulp-1.1.12-1.el6cf > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [##################################################] 100.0% > > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'198b5298-2a39-447f-89de-e1973c4e0637') > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) > Exiting on user command Full test output available at ... http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/8372110//TESTOUT.log The failed system is available for remote debugging, please ping on irc (nick:jlaska) for the details. (In reply to comment #6) > > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'198b5298-2a39-447f-89de-e1973c4e0637') Looking into mongo for the affected package ... > > db.packages.find({'id': '198b5298-2a39-447f-89de-e1973c4e0637'}) > { "_id" : "198b5298-2a39-447f-89de-e1973c4e0637", "license" : "distributable", "vendor" : "Red Hat, Inc.", "name" : "libtiff", "repo_defined" : true, "checksum" : { "sha" : "7325f8ad2aeb889f5b878cc403c6b7fa13be9851" }, "description" : "The libtiff package contains a library of functions for manipulating\nTIFF (Tagged Image File Format) image format files. TIFF is a widely\nused file format for bitmapped images. TIFF files usually end in the\n.tif extension and they are often quite large.\n\nThe libtiff package should be installed if you need to manipulate TIFF\nformat image files.", "filename" : "libtiff-3.8.2-7.el5.i386.rpm", "download_url" : "https://qeblade41.rhq.lab.eng.bos.redhat.com//pulp/repos/redhat/Library/content/dist/rhel/server/5/5Server/x86_64/os/libtiff-3.8.2-7.el5.i386.rpm", "buildhost" : "ls20-bc1-14.build.redhat.com", "repoids" : [ > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server", > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8", > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server" > ], "epoch" : "0", "version" : "3.8.2", "arch" : "i386", "provides" : [ "libtiffxx.so.3", "libtiff.so.3", "libtiff" ], "_ns" : "packages", "release" : "7.el5", "group" : "System Environment/Libraries", "requires" : [ > "rtld(GNU_HASH)", ... snip ... > "/sbin/ldconfig" > ], "id" : "198b5298-2a39-447f-89de-e1973c4e0637", "size" : 319548 } With this latest failure, the affected package is provided by 3 repositories. All 5.8 (or 5Server), i386 and x86_64. The only workaround I'm aware of with this issue is to either ... 1) Sync repositories individually, not the entire 'Red Hat' provider Or 2) When syncing the provider, only enable only a few repositories at a time Both seem like arbitrary workarounds and not something I'd expect to see documented in our release notes. They're certainly not difficult workarounds, just seems unrealistic to expect customers to do this. I could be wrong, please correct if needed. Created attachment 625908 [details] katello-debug-20121012070106.tar.gz Reproduced overnight. Pretty easy to reproduce, it just requires *real* data, and a lot of it. For example, I enabled the following CDN $releasevers repos: * 5Server * 5.8 * 6Server * 6.3 * 6.2 Then sync the 'Red Hat' provider: > # katello -u admin -p admin provider synchronize --name "Red Hat" ... wait 6 hours ... > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d') > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) == Additional information == [ /var/log/pulp/pulp.log ] > 2012-10-11 19:56:21,396 29437:140707692017408: pulp.server.api.synchronizers:ERROR: synchronizers:574 Package "<pulp.server.util.Package instance at 0x7ff908f9c3f8>", import failed > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 571, in import_package > self.package_api.update(newpkg["id"], delta) > File "/usr/lib/python2.6/site-packages/pulp/server/api/package.py", line 71, in update > raise PulpException('Package [%s] does not exist', id) > PulpException: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d') > 2012-10-11 19:56:21,543 29437:140707692017408: pulp.server.tasking.task:ERROR: task:472 Task failed: Task 140e4a7a-13ef-11e2-9f90-e83935b34928: _sync(redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7ff910965050>, skip={}, max_speed=None, threads=4, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7ff93004af90>>) > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 418, in run > result = self.callable(*self.args, **self.kwargs) > File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 283, in _sync > progress_callback, synchronizer, max_speed, threads) > File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 373, in fetch_content > added_packages = synchronizer.process_packages_from_source(repo_dir, repo_id, skip_dict, progress_callback) > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 215, in process_packages_from_source > added_packages = self.add_packages_from_dir(dir, repo_id, skip_dict) > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 256, in add_packages_from_dir > package = self.import_package(package, repo, repo_defined=True) > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 571, in import_package > self.package_api.update(newpkg["id"], delta) > File "/usr/lib/python2.6/site-packages/pulp/server/api/package.py", line 71, in update > raise PulpException('Package [%s] does not exist', id) > PulpException: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d') [ mongo matching package id ] > # mongo pulp_database > > db.packages.find({'id': 'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d'}) > { "_id" : "a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d", "license" : "Dual GPL/BSD/CPL", "vendor" : "Red Hat, Inc.", "name" : "dapl-utils", "repo_defined" : true, "checksum" : { "sha" : "7d25739721c473d431b1a0191a14e072391dbf96" }, "id" : "a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d", "description" : "Useful test suites to validate the dapl library API's and operation.", "filename" : "dapl-utils-2.0.13-4.el5.i386.rpm", "download_url" : "https://qeblade41.rhq.lab.eng.bos.redhat.com//pulp/repos/redhat/Library/content/dist/rhel/server/5/5.8/i386/os/dapl-utils-2.0.13-4.el5.i386.rpm", "buildhost" : "hs20-bc2-4.build.redhat.com", "repoids" : [ > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8" > ], "version" : "2.0.13", "arch" : "i386", "provides" : [ "dapl-utils" ], "epoch" : "0", "release" : "4.el5", "group" : "System Environment/Libraries", "requires" : [ > "rtld(GNU_HASH)", > "libpthread.so.0(GLIBC_2.3.2)", > "libpthread.so.0(GLIBC_2.1)", > "libpthread.so.0(GLIBC_2.0)", > "libpthread.so.0", > "libibverbs.so.1", > "libdl.so.2", > "libdat2.so.2(DAT_2.0)", > "libdat2.so.2", > "libc.so.6(GLIBC_2.4)", > "libc.so.6(GLIBC_2.3.4)", > "libc.so.6(GLIBC_2.1)", > "libc.so.6(GLIBC_2.0)", > "libc.so.6", > "dapl" > ], "_ns" : "packages", "size" : 93479 } [ katello unsynced repo ] > # katello -u admin -p admin repo list --environment Library > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Repo List For Org redhat Environment Library > > Id Name Label Package Count Last Sync > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > 1000 Red Hat Enterprise Linux 5 Server RPMs i386 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8 10841 2012/10/11 20:00:50 > 995 Red Hat Enterprise Linux 5 Server RPMs i386 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server 0 never > 997 Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8 14110 2012/10/11 21:45:29 > 994 Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server 14110 2012/10/11 21:46:24 > 1071 Red Hat Enterprise Linux 6 Server RPMs i386 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2 5599 2012/10/11 22:39:47 > 1067 Red Hat Enterprise Linux 6 Server RPMs i386 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3 6577 2012/10/11 22:41:08 > 1070 Red Hat Enterprise Linux 6 Server RPMs i386 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server 6577 2012/10/11 23:06:35 > 1073 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2 7281 2012/10/11 23:37:51 > 1069 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3 8573 2012/10/11 23:40:10 > 1074 Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server 8573 2012/10/11 23:59:35 > Testing a patch from prad (http://pkilambi.fedorapeople.org/sync.patch). Results available at https://beaker.engineering.redhat.com/jobs/315608 (In reply to comment #10) > Testing a patch from prad (http://pkilambi.fedorapeople.org/sync.patch). > Results available at https://beaker.engineering.redhat.com/jobs/315608 Test completed successfully. In /var/log/pulp/pulp.log, I noticed 4 retry instances, by way of the new log.info() call in pulp/server/api/synchronizers.py::import_package() > # grep "Retrying" /var/log/pulp/pulp.log * > /var/log/pulp/pulp.log:2012-10-12 18:27:13,026 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 > /var/log/pulp/pulp.log:2012-10-12 18:27:25,159 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 > /var/log/pulp/pulp.log:2012-10-12 18:27:35,289 6891:140032658614016: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 > /var/log/pulp/pulp.log:2012-10-12 18:27:44,834 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 I might suggest altering the log.info() to include the package name as well? > /usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py > log.info("Retrying '%s' ['%s'] ... %s" % (package, repo, num_retries)) From the bug Prad cloned into the Pulp product ... (In reply to comment #1) > commit b7e12df0d0e767f1f20ec8cb24e475aafbcbcc9c My reproducer for this issue is underway in beaker (https://beaker.engineering.redhat.com/jobs/320405). I'll update with status when the test completes. Testing with pulp-1.1.14-1.el6cf, I was unable to reproduce the original problem using the same reproducer https://beaker.engineering.redhat.com/jobs/320405 Created attachment 632038 [details]
katello-debug-20121023073026.tar.gz
katello-debug logs from a fixed system.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-1543.html |