Created attachment 621732 [details] katello-debug-20121004115319.tar.gz Description of problem: Failure while attempting to a RHEL-5.8-i386 repository from https://cdn.redhat.com. The provider has several RHEL repositories enabled, and it seems that the pulp sync completes. However, katello-cli seems to think the sync failed as it returns a nono-zero (65) exit code. > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) > # katello -u admin -p admin provider synchronize --name "Red Hat" > Progress: [##################################################] 100.0% > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'b017e5e0-6d3e-4a9b-b3bb-53f55fc3e209') > # katello -u admin -p admin repo list --environment Library > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Repo List For Org redhat Environment Library > > Id Name Label Package Count Last Sync > --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > 653 Red Hat Enterprise Linux 5 Server RPMs i386 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8 0 never > 648 Red Hat Enterprise Linux 5 Server RPMs i386 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server 10821 2012/10/04 08:42:43 > 651 Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8 14083 2012/10/03 15:41:29 > 649 Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server 14083 2012/10/04 09:18:09 > 734 Red Hat Enterprise Linux 6 Server RPMs i386 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2 5599 2012/10/04 09:19:20 > 733 Red Hat Enterprise Linux 6 Server RPMs i386 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3 6552 2012/10/04 09:36:39 > 739 Red Hat Enterprise Linux 6 Server RPMs i386 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server 6552 2012/10/04 09:36:35 > 732 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2 7281 2012/10/04 10:00:17 > 737 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3 8538 2012/10/04 10:01:52 > 736 Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server 8538 2012/10/04 10:25:00 Version-Release number of selected component (if applicable): * candlepin-0.7.8-1.el6cf.src.rpm * katello-1.1.12-12.el6cf.src.rpm * katello-candlepin-cert-key-pair-1.0-1.src.rpm * katello-certs-tools-1.1.8-1.el6cf.src.rpm * katello-cli-1.1.8-6.el6cf.src.rpm * katello-configure-1.1.9-6.el6cf.src.rpm * katello-qpid-broker-key-pair-1.0-1.src.rpm * katello-qpid-client-key-pair-1.0-1.src.rpm * katello-selinux-1.1.1-1.el6cf.src.rpm * pulp-1.1.12-1.el6cf.src.rpm How reproducible: * First time I've observed this Steps to Reproduce: 1. Install CFSE from the snap#3 puddle 2. Configure Katello > # katello-configure 3. Import a Red hat manifest > # curl -o /tmp/manifest.zip http://file.rdu.redhat.com/jlaska/manifests_access.zip > # katello -u admin -p admin provider import_manifest --name "Red Hat" --file /tmp/manifest.zip 4. Enable RHEL repositories > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6Server" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6.3" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6.2" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5Server" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5.8" > # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8" 5. Initiate a sync of the 'Red Hat' provider > # katello -u admin -p admin provider synchronize --name "Red Hat"' Actual results: > # katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) # Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'b017e5e0-6d3e-4a9b-b3bb-53f55fc3e209') :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) When I attempt to re-sync the same 'Red Hat' provider ... I get the following result: # katello -u admin -p admin provider synchronize --name "Red Hat" Provider [ Red Hat ] failed to sync: E11000 duplicate key error index: pulp_database.packages.$_id_ dup key: { : "1b6a6e98-a19e-4bf5-a2c1-314ccec8bcdc" } # echo "EXIT_CODE=$?" EXIT_CODE=65 Expected results: The sync should complete successfully Additional info: * See attached katello-debug.tgz * I'm not aware of a workaround, but at this time it appears I'm unable to sync RHEL-5.8-i386. I'll retest to determine whether this failure is consistent (vs intermittent)
Created attachment 623430 [details] katello-debug-20121008080416.tar.gz Reproduced using CloudForms snapshot#3 > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [##################################################] 100.0% > > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'8975f49d-eb41-47db-853c-829bde3499dc') > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) > Exiting on user command Full test output available at http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/8281114//TESTOUT.log Once again, inspecting the list of repositories seems to show the faulty repo is RHEL-5.8-i386. > # katello -u admin -p admin repo list --environment Library > ------------------------------------------------------------------------------------------------------------------------------------------------------------ > Repo List For Org redhat Environment Library > > Id Name Label Package Count Last Sync > ----------------------------------------------------------------------------------------------------------------------------------------------------------- > 967 Red Hat Enterprise Linux 5 Server RPMs i386 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8 10830 2012/10/05 12:50:07 > 965 Red Hat Enterprise Linux 5 Server RPMs i386 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server 0 never > 969 Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8 14092 2012/10/05 14:54:04 > 966 Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server 14092 2012/10/05 14:55:13 > 1053 Red Hat Enterprise Linux 6 Server RPMs i386 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2 5599 2012/10/05 16:01:56 > 1055 Red Hat Enterprise Linux 6 Server RPMs i386 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3 6569 2012/10/05 16:03:26 > 1056 Red Hat Enterprise Linux 6 Server RPMs i386 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server 6569 2012/10/05 16:29:48 > 1050 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2 7281 2012/10/05 16:59:34 > 1052 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3 8560 2012/10/05 17:01:10 > 1051 Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server 8560 2012/10/05 17:20:40
Escalating as this is impacting RHEL5 content sync from cdn.redhat.com. I'm unable to workaround this issue. Re-syncing also fails.
I re-installed katello and enabled+synced *only* RHEL 5.8 and 5Server (i386 and x86_64). I was unable to reproduce the problem.
will look into this
Created attachment 624841 [details] katello-debug-20121010083059.tar.gz (snap#4) Reproduced on snap#4 * katello-1.1.12-14.el6cf * pulp-1.1.12-1.el6cf > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [############################################# ] 90.0% > Progress: [##################################################] 100.0% > > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'198b5298-2a39-447f-89de-e1973c4e0637') > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) > Exiting on user command Full test output available at ... http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/8372110//TESTOUT.log The failed system is available for remote debugging, please ping on irc (nick:jlaska) for the details.
(In reply to comment #6) > > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'198b5298-2a39-447f-89de-e1973c4e0637') Looking into mongo for the affected package ... > > db.packages.find({'id': '198b5298-2a39-447f-89de-e1973c4e0637'}) > { "_id" : "198b5298-2a39-447f-89de-e1973c4e0637", "license" : "distributable", "vendor" : "Red Hat, Inc.", "name" : "libtiff", "repo_defined" : true, "checksum" : { "sha" : "7325f8ad2aeb889f5b878cc403c6b7fa13be9851" }, "description" : "The libtiff package contains a library of functions for manipulating\nTIFF (Tagged Image File Format) image format files. TIFF is a widely\nused file format for bitmapped images. TIFF files usually end in the\n.tif extension and they are often quite large.\n\nThe libtiff package should be installed if you need to manipulate TIFF\nformat image files.", "filename" : "libtiff-3.8.2-7.el5.i386.rpm", "download_url" : "https://qeblade41.rhq.lab.eng.bos.redhat.com//pulp/repos/redhat/Library/content/dist/rhel/server/5/5Server/x86_64/os/libtiff-3.8.2-7.el5.i386.rpm", "buildhost" : "ls20-bc1-14.build.redhat.com", "repoids" : [ > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server", > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8", > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server" > ], "epoch" : "0", "version" : "3.8.2", "arch" : "i386", "provides" : [ "libtiffxx.so.3", "libtiff.so.3", "libtiff" ], "_ns" : "packages", "release" : "7.el5", "group" : "System Environment/Libraries", "requires" : [ > "rtld(GNU_HASH)", ... snip ... > "/sbin/ldconfig" > ], "id" : "198b5298-2a39-447f-89de-e1973c4e0637", "size" : 319548 } With this latest failure, the affected package is provided by 3 repositories. All 5.8 (or 5Server), i386 and x86_64.
The only workaround I'm aware of with this issue is to either ... 1) Sync repositories individually, not the entire 'Red Hat' provider Or 2) When syncing the provider, only enable only a few repositories at a time Both seem like arbitrary workarounds and not something I'd expect to see documented in our release notes. They're certainly not difficult workarounds, just seems unrealistic to expect customers to do this. I could be wrong, please correct if needed.
Created attachment 625908 [details] katello-debug-20121012070106.tar.gz Reproduced overnight. Pretty easy to reproduce, it just requires *real* data, and a lot of it. For example, I enabled the following CDN $releasevers repos: * 5Server * 5.8 * 6Server * 6.3 * 6.2 Then sync the 'Red Hat' provider: > # katello -u admin -p admin provider synchronize --name "Red Hat" ... wait 6 hours ... > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d') > :: [ FAIL ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65) == Additional information == [ /var/log/pulp/pulp.log ] > 2012-10-11 19:56:21,396 29437:140707692017408: pulp.server.api.synchronizers:ERROR: synchronizers:574 Package "<pulp.server.util.Package instance at 0x7ff908f9c3f8>", import failed > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 571, in import_package > self.package_api.update(newpkg["id"], delta) > File "/usr/lib/python2.6/site-packages/pulp/server/api/package.py", line 71, in update > raise PulpException('Package [%s] does not exist', id) > PulpException: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d') > 2012-10-11 19:56:21,543 29437:140707692017408: pulp.server.tasking.task:ERROR: task:472 Task failed: Task 140e4a7a-13ef-11e2-9f90-e83935b34928: _sync(redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7ff910965050>, skip={}, max_speed=None, threads=4, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7ff93004af90>>) > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 418, in run > result = self.callable(*self.args, **self.kwargs) > File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 283, in _sync > progress_callback, synchronizer, max_speed, threads) > File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 373, in fetch_content > added_packages = synchronizer.process_packages_from_source(repo_dir, repo_id, skip_dict, progress_callback) > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 215, in process_packages_from_source > added_packages = self.add_packages_from_dir(dir, repo_id, skip_dict) > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 256, in add_packages_from_dir > package = self.import_package(package, repo, repo_defined=True) > File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 571, in import_package > self.package_api.update(newpkg["id"], delta) > File "/usr/lib/python2.6/site-packages/pulp/server/api/package.py", line 71, in update > raise PulpException('Package [%s] does not exist', id) > PulpException: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d') [ mongo matching package id ] > # mongo pulp_database > > db.packages.find({'id': 'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d'}) > { "_id" : "a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d", "license" : "Dual GPL/BSD/CPL", "vendor" : "Red Hat, Inc.", "name" : "dapl-utils", "repo_defined" : true, "checksum" : { "sha" : "7d25739721c473d431b1a0191a14e072391dbf96" }, "id" : "a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d", "description" : "Useful test suites to validate the dapl library API's and operation.", "filename" : "dapl-utils-2.0.13-4.el5.i386.rpm", "download_url" : "https://qeblade41.rhq.lab.eng.bos.redhat.com//pulp/repos/redhat/Library/content/dist/rhel/server/5/5.8/i386/os/dapl-utils-2.0.13-4.el5.i386.rpm", "buildhost" : "hs20-bc2-4.build.redhat.com", "repoids" : [ > "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8" > ], "version" : "2.0.13", "arch" : "i386", "provides" : [ "dapl-utils" ], "epoch" : "0", "release" : "4.el5", "group" : "System Environment/Libraries", "requires" : [ > "rtld(GNU_HASH)", > "libpthread.so.0(GLIBC_2.3.2)", > "libpthread.so.0(GLIBC_2.1)", > "libpthread.so.0(GLIBC_2.0)", > "libpthread.so.0", > "libibverbs.so.1", > "libdl.so.2", > "libdat2.so.2(DAT_2.0)", > "libdat2.so.2", > "libc.so.6(GLIBC_2.4)", > "libc.so.6(GLIBC_2.3.4)", > "libc.so.6(GLIBC_2.1)", > "libc.so.6(GLIBC_2.0)", > "libc.so.6", > "dapl" > ], "_ns" : "packages", "size" : 93479 } [ katello unsynced repo ] > # katello -u admin -p admin repo list --environment Library > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Repo List For Org redhat Environment Library > > Id Name Label Package Count Last Sync > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > 1000 Red Hat Enterprise Linux 5 Server RPMs i386 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8 10841 2012/10/11 20:00:50 > 995 Red Hat Enterprise Linux 5 Server RPMs i386 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server 0 never > 997 Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8 Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8 14110 2012/10/11 21:45:29 > 994 Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server 14110 2012/10/11 21:46:24 > 1071 Red Hat Enterprise Linux 6 Server RPMs i386 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2 5599 2012/10/11 22:39:47 > 1067 Red Hat Enterprise Linux 6 Server RPMs i386 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3 6577 2012/10/11 22:41:08 > 1070 Red Hat Enterprise Linux 6 Server RPMs i386 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server 6577 2012/10/11 23:06:35 > 1073 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2 7281 2012/10/11 23:37:51 > 1069 Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3 Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3 8573 2012/10/11 23:40:10 > 1074 Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server 8573 2012/10/11 23:59:35 >
Testing a patch from prad (http://pkilambi.fedorapeople.org/sync.patch). Results available at https://beaker.engineering.redhat.com/jobs/315608
(In reply to comment #10) > Testing a patch from prad (http://pkilambi.fedorapeople.org/sync.patch). > Results available at https://beaker.engineering.redhat.com/jobs/315608 Test completed successfully. In /var/log/pulp/pulp.log, I noticed 4 retry instances, by way of the new log.info() call in pulp/server/api/synchronizers.py::import_package() > # grep "Retrying" /var/log/pulp/pulp.log * > /var/log/pulp/pulp.log:2012-10-12 18:27:13,026 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 > /var/log/pulp/pulp.log:2012-10-12 18:27:25,159 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 > /var/log/pulp/pulp.log:2012-10-12 18:27:35,289 6891:140032658614016: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 > /var/log/pulp/pulp.log:2012-10-12 18:27:44,834 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5 I might suggest altering the log.info() to include the package name as well? > /usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py > log.info("Retrying '%s' ['%s'] ... %s" % (package, repo, num_retries))
From the bug Prad cloned into the Pulp product ... (In reply to comment #1) > commit b7e12df0d0e767f1f20ec8cb24e475aafbcbcc9c
My reproducer for this issue is underway in beaker (https://beaker.engineering.redhat.com/jobs/320405). I'll update with status when the test completes.
Testing with pulp-1.1.14-1.el6cf, I was unable to reproduce the original problem using the same reproducer https://beaker.engineering.redhat.com/jobs/320405
Created attachment 632038 [details] katello-debug-20121023073026.tar.gz katello-debug logs from a fixed system.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-1543.html