Bug 863187 - failed to sync: ('Package [%s] does not exist', u'b017e5e0-6d3e-4a9b-b3bb-53f55fc3e209')
Summary: failed to sync: ('Package [%s] does not exist', u'b017e5e0-6d3e-4a9b-b3bb-53f...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Katello Bug Bin
QA Contact: Corey Welton
URL:
Whiteboard:
Depends On:
Blocks: 865003
TreeView+ depends on / blocked
 
Reported: 2012-10-04 15:57 UTC by James Laska
Modified: 2019-09-25 20:45 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Attempts to sync a RHEL-5.8-i386 repository failed when syncing simultaneous repositories from Red Hat CDN. This fix adds a retry mechanism.
Clone Of:
: 865003 (view as bug list)
Environment:
Last Closed: 2012-12-04 19:56:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
katello-debug-20121004115319.tar.gz (8.68 MB, application/x-gzip)
2012-10-04 15:57 UTC, James Laska
no flags Details
katello-debug-20121008080416.tar.gz (8.42 MB, application/octet-stream)
2012-10-08 12:05 UTC, James Laska
no flags Details
katello-debug-20121010083059.tar.gz (snap#4) (6.95 MB, application/octet-stream)
2012-10-10 12:32 UTC, James Laska
no flags Details
katello-debug-20121012070106.tar.gz (5.75 MB, application/x-gzip)
2012-10-12 11:18 UTC, James Laska
no flags Details
katello-debug-20121023073026.tar.gz (6.43 MB, application/octet-stream)
2012-10-23 11:32 UTC, James Laska
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:1543 0 normal SHIPPED_LIVE Important: CloudForms System Engine 1.1 update 2012-12-05 00:39:57 UTC

Description James Laska 2012-10-04 15:57:07 UTC
Created attachment 621732 [details]
katello-debug-20121004115319.tar.gz

Description of problem:

Failure while attempting to a RHEL-5.8-i386 repository from https://cdn.redhat.com.  The provider has several RHEL repositories enabled, and it seems that the pulp sync completes.  However, katello-cli seems to think the sync failed as it returns a nono-zero (65) exit code.

> :: [   FAIL   ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65)
> # katello -u admin -p admin provider synchronize --name "Red Hat"
> Progress: [##################################################] 100.0%
> Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'b017e5e0-6d3e-4a9b-b3bb-53f55fc3e209')

> # katello -u admin -p admin repo list --environment Library
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                                                                                 Repo List For Org redhat Environment Library
> 
>  Id    Name                                                    Label                                                   Package Count   Last Sync            
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>  653   Red Hat Enterprise Linux 5 Server RPMs i386 5.8         Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8         0               never                
>  648   Red Hat Enterprise Linux 5 Server RPMs i386 5Server     Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server     10821           2012/10/04 08:42:43  
>  651   Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8       Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8       14083           2012/10/03 15:41:29  
>  649   Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server   Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server   14083           2012/10/04 09:18:09  
>  734   Red Hat Enterprise Linux 6 Server RPMs i386 6.2         Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2         5599            2012/10/04 09:19:20  
>  733   Red Hat Enterprise Linux 6 Server RPMs i386 6.3         Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3         6552            2012/10/04 09:36:39  
>  739   Red Hat Enterprise Linux 6 Server RPMs i386 6Server     Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server     6552            2012/10/04 09:36:35  
>  732   Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2       Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2       7281            2012/10/04 10:00:17  
>  737   Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3       Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3       8538            2012/10/04 10:01:52  
>  736   Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server   Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server   8538            2012/10/04 10:25:00  

Version-Release number of selected component (if applicable):
 * candlepin-0.7.8-1.el6cf.src.rpm
 * katello-1.1.12-12.el6cf.src.rpm
 * katello-candlepin-cert-key-pair-1.0-1.src.rpm
 * katello-certs-tools-1.1.8-1.el6cf.src.rpm
 * katello-cli-1.1.8-6.el6cf.src.rpm
 * katello-configure-1.1.9-6.el6cf.src.rpm
 * katello-qpid-broker-key-pair-1.0-1.src.rpm
 * katello-qpid-client-key-pair-1.0-1.src.rpm
 * katello-selinux-1.1.1-1.el6cf.src.rpm
 * pulp-1.1.12-1.el6cf.src.rpm

How reproducible:
 * First time I've observed this

Steps to Reproduce:
1. Install CFSE from the snap#3 puddle
2. Configure Katello
> # katello-configure

3. Import a Red hat manifest
> # curl -o /tmp/manifest.zip http://file.rdu.redhat.com/jlaska/manifests_access.zip
> # katello -u admin -p admin provider import_manifest --name "Red Hat" --file /tmp/manifest.zip

4. Enable RHEL repositories
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6Server"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6.3"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs i386 6.2"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5Server"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs i386 5.8"
> # katello -u admin -p admin repo enable --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8"

5. Initiate a sync of the 'Red Hat' provider
> # katello -u admin -p admin provider synchronize --name "Red Hat"'

Actual results:

> # katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65)
# Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'b017e5e0-6d3e-4a9b-b3bb-53f55fc3e209')
:: [   FAIL   ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65)

When I attempt to re-sync the same 'Red Hat' provider ... I get the following result:

# katello -u admin -p admin provider synchronize --name "Red Hat"
Provider [ Red Hat ] failed to sync: E11000 duplicate key error index: pulp_database.packages.$_id_  dup key: { : "1b6a6e98-a19e-4bf5-a2c1-314ccec8bcdc" }

# echo "EXIT_CODE=$?"
EXIT_CODE=65

Expected results:

The sync should complete successfully

Additional info:

 * See attached katello-debug.tgz
 * I'm not aware of a workaround, but at this time it appears I'm unable to sync RHEL-5.8-i386.  I'll retest to determine whether this failure is consistent (vs intermittent)

Comment 2 James Laska 2012-10-08 12:05:25 UTC
Created attachment 623430 [details]
katello-debug-20121008080416.tar.gz

Reproduced using CloudForms snapshot#3

> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [##################################################] 100.0%
>                                                                       
> Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'8975f49d-eb41-47db-853c-829bde3499dc')
> :: [   FAIL   ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65)
> Exiting on user command

Full test output available at http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/8281114//TESTOUT.log

Once again, inspecting the list of repositories seems to show the faulty repo is RHEL-5.8-i386.

> # katello -u admin -p admin repo list --environment Library
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>                                                                                 Repo List For Org redhat Environment Library
> 
>  Id     Name                                                    Label                                                   Package Count   Last Sync          
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>  967    Red Hat Enterprise Linux 5 Server RPMs i386 5.8         Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8         10830           2012/10/05 12:50:07  
>  965    Red Hat Enterprise Linux 5 Server RPMs i386 5Server     Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server     0               never                
>  969    Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8       Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8       14092           2012/10/05 14:54:04  
>  966    Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server   Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server   14092           2012/10/05 14:55:13  
>  1053   Red Hat Enterprise Linux 6 Server RPMs i386 6.2         Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2         5599            2012/10/05 16:01:56  
>  1055   Red Hat Enterprise Linux 6 Server RPMs i386 6.3         Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3         6569            2012/10/05 16:03:26  
>  1056   Red Hat Enterprise Linux 6 Server RPMs i386 6Server     Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server     6569            2012/10/05 16:29:48  
>  1050   Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2       Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2       7281            2012/10/05 16:59:34  
>  1052   Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3       Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3       8560            2012/10/05 17:01:10  
>  1051   Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server   Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server   8560            2012/10/05 17:20:40

Comment 3 James Laska 2012-10-08 13:15:01 UTC
Escalating as this is impacting RHEL5 content sync from cdn.redhat.com.  I'm unable to workaround this issue.  Re-syncing also fails.

Comment 4 James Laska 2012-10-09 11:48:33 UTC
I re-installed katello and enabled+synced *only* RHEL 5.8 and 5Server (i386 and x86_64).  I was unable to reproduce the problem.

Comment 5 Mike McCune 2012-10-09 21:48:31 UTC
will look into this

Comment 6 James Laska 2012-10-10 12:32:21 UTC
Created attachment 624841 [details]
katello-debug-20121010083059.tar.gz (snap#4)

Reproduced on snap#4 
 * katello-1.1.12-14.el6cf
 * pulp-1.1.12-1.el6cf

> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [#############################################     ] 90.0%
> Progress: [##################################################] 100.0%
>                                                                       
> Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'198b5298-2a39-447f-89de-e1973c4e0637')
> :: [   FAIL   ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65)
> Exiting on user command

Full test output available at ... http://lab-02.rhts.eng.bos.redhat.com/beaker/logs/tasks/8372110//TESTOUT.log

The failed system is available for remote debugging, please ping on irc (nick:jlaska) for the details.

Comment 7 James Laska 2012-10-10 12:42:48 UTC
(In reply to comment #6)
> > Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'198b5298-2a39-447f-89de-e1973c4e0637')

Looking into mongo for the affected package ...

> > db.packages.find({'id': '198b5298-2a39-447f-89de-e1973c4e0637'})
> { "_id" : "198b5298-2a39-447f-89de-e1973c4e0637", "license" : "distributable", "vendor" : "Red Hat, Inc.", "name" : "libtiff", "repo_defined" : true, "checksum" : { "sha" : "7325f8ad2aeb889f5b878cc403c6b7fa13be9851" }, "description" : "The libtiff package contains a library of functions for manipulating\nTIFF (Tagged Image File Format) image format files.  TIFF is a widely\nused file format for bitmapped images.  TIFF files usually end in the\n.tif extension and they are often quite large.\n\nThe libtiff package should be installed if you need to manipulate TIFF\nformat image files.", "filename" : "libtiff-3.8.2-7.el5.i386.rpm", "download_url" : "https://qeblade41.rhq.lab.eng.bos.redhat.com//pulp/repos/redhat/Library/content/dist/rhel/server/5/5Server/x86_64/os/libtiff-3.8.2-7.el5.i386.rpm", "buildhost" : "ls20-bc1-14.build.redhat.com", "repoids" : [
>         "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server",
>         "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8",
>         "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server"
> ], "epoch" : "0", "version" : "3.8.2", "arch" : "i386", "provides" : [ "libtiffxx.so.3", "libtiff.so.3", "libtiff" ], "_ns" : "packages", "release" : "7.el5", "group" : "System Environment/Libraries", "requires" : [
>         "rtld(GNU_HASH)",
 ... snip ...
>         "/sbin/ldconfig"
> ], "id" : "198b5298-2a39-447f-89de-e1973c4e0637", "size" : 319548 }

With this latest failure, the affected package is provided by 3 repositories.  All 5.8 (or 5Server), i386 and x86_64.

Comment 8 James Laska 2012-10-10 16:35:06 UTC
The only workaround I'm aware of with this issue is to either ...

 1) Sync repositories individually, not the entire 'Red Hat' provider

Or 

 2) When syncing the provider, only enable only a few repositories at a time

Both seem like arbitrary workarounds and not something I'd expect to see documented in our release notes.  They're certainly not difficult workarounds, just seems unrealistic to expect customers to do this.  I could be wrong, please correct if needed.

Comment 9 James Laska 2012-10-12 11:18:47 UTC
Created attachment 625908 [details]
katello-debug-20121012070106.tar.gz

Reproduced overnight.  Pretty easy to reproduce, it just requires *real* data, and a lot of it.  For example, I enabled the following CDN $releasevers repos:
 * 5Server
 * 5.8
 * 6Server
 * 6.3
 * 6.2

Then sync the 'Red Hat' provider:
> # katello -u admin -p admin provider synchronize --name "Red Hat"

... wait 6 hours ...

> Provider [ Red Hat ] failed to sync: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d')
> :: [   FAIL   ] :: Running 'katello -u admin -p admin provider synchronize --name "Red Hat"' (Expected 0, got 65)

== Additional information ==

[ /var/log/pulp/pulp.log ]
> 2012-10-11 19:56:21,396 29437:140707692017408: pulp.server.api.synchronizers:ERROR: synchronizers:574 Package "<pulp.server.util.Package instance at 0x7ff908f9c3f8>", import failed
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 571, in import_package
>     self.package_api.update(newpkg["id"], delta)
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/package.py", line 71, in update
>     raise PulpException('Package [%s] does not exist', id)
> PulpException: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d')
> 2012-10-11 19:56:21,543 29437:140707692017408: pulp.server.tasking.task:ERROR: task:472 Task failed: Task 140e4a7a-13ef-11e2-9f90-e83935b34928: _sync(redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7ff910965050>, skip={}, max_speed=None, threads=4, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7ff93004af90>>)
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 418, in run
>     result = self.callable(*self.args, **self.kwargs)
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 283, in _sync
>     progress_callback, synchronizer, max_speed, threads)
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 373, in fetch_content
>     added_packages = synchronizer.process_packages_from_source(repo_dir, repo_id, skip_dict, progress_callback)
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 215, in process_packages_from_source
>     added_packages = self.add_packages_from_dir(dir, repo_id, skip_dict)
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 256, in add_packages_from_dir
>     package = self.import_package(package, repo, repo_defined=True)
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 571, in import_package
>     self.package_api.update(newpkg["id"], delta)
>   File "/usr/lib/python2.6/site-packages/pulp/server/api/package.py", line 71, in update
>     raise PulpException('Package [%s] does not exist', id)
> PulpException: ('Package [%s] does not exist', u'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d')

[ mongo matching package id ]
> # mongo pulp_database
> > db.packages.find({'id': 'a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d'})
> { "_id" : "a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d", "license" : "Dual GPL/BSD/CPL", "vendor" : "Red Hat, Inc.", "name" : "dapl-utils", "repo_defined" : true, "checksum" : { "sha" : "7d25739721c473d431b1a0191a14e072391dbf96" }, "id" : "a49ece8e-9f7d-4ae9-8ead-a6b5fe7d1f5d", "description" : "Useful test suites to validate the dapl library API's and operation.", "filename" : "dapl-utils-2.0.13-4.el5.i386.rpm", "download_url" : "https://qeblade41.rhq.lab.eng.bos.redhat.com//pulp/repos/redhat/Library/content/dist/rhel/server/5/5.8/i386/os/dapl-utils-2.0.13-4.el5.i386.rpm", "buildhost" : "hs20-bc2-4.build.redhat.com", "repoids" : [
>         "redhat-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8"
> ], "version" : "2.0.13", "arch" : "i386", "provides" : [ "dapl-utils" ], "epoch" : "0", "release" : "4.el5", "group" : "System Environment/Libraries", "requires" : [
>         "rtld(GNU_HASH)",
>         "libpthread.so.0(GLIBC_2.3.2)",
>         "libpthread.so.0(GLIBC_2.1)",
>         "libpthread.so.0(GLIBC_2.0)",
>         "libpthread.so.0",
>         "libibverbs.so.1",
>         "libdl.so.2",
>         "libdat2.so.2(DAT_2.0)",
>         "libdat2.so.2",
>         "libc.so.6(GLIBC_2.4)",
>         "libc.so.6(GLIBC_2.3.4)",
>         "libc.so.6(GLIBC_2.1)",
>         "libc.so.6(GLIBC_2.0)",
>         "libc.so.6",
>         "dapl"
> ], "_ns" : "packages", "size" : 93479 }


[ katello unsynced repo ]
> # katello -u admin -p admin repo list --environment Library
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                                                                    Repo List For Org redhat Environment Library
> 
>  Id     Name                                                    Label                                                   Package Count   Last Sync            
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>  1000   Red Hat Enterprise Linux 5 Server RPMs i386 5.8         Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5_8         10841           2012/10/11 20:00:50  
>  995    Red Hat Enterprise Linux 5 Server RPMs i386 5Server     Red_Hat_Enterprise_Linux_5_Server_RPMs_i386_5Server     0               never                
>  997    Red Hat Enterprise Linux 5 Server RPMs x86_64 5.8       Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5_8       14110           2012/10/11 21:45:29  
>  994    Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server   Red_Hat_Enterprise_Linux_5_Server_RPMs_x86_64_5Server   14110           2012/10/11 21:46:24  
>  1071   Red Hat Enterprise Linux 6 Server RPMs i386 6.2         Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_2         5599            2012/10/11 22:39:47  
>  1067   Red Hat Enterprise Linux 6 Server RPMs i386 6.3         Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6_3         6577            2012/10/11 22:41:08  
>  1070   Red Hat Enterprise Linux 6 Server RPMs i386 6Server     Red_Hat_Enterprise_Linux_6_Server_RPMs_i386_6Server     6577            2012/10/11 23:06:35  
>  1073   Red Hat Enterprise Linux 6 Server RPMs x86_64 6.2       Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_2       7281            2012/10/11 23:37:51  
>  1069   Red Hat Enterprise Linux 6 Server RPMs x86_64 6.3       Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6_3       8573            2012/10/11 23:40:10  
>  1074   Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server   Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server   8573            2012/10/11 23:59:35  
>

Comment 10 James Laska 2012-10-12 21:04:30 UTC
Testing a patch from prad (http://pkilambi.fedorapeople.org/sync.patch).  Results available at https://beaker.engineering.redhat.com/jobs/315608

Comment 11 James Laska 2012-10-15 12:09:17 UTC
(In reply to comment #10)
> Testing a patch from prad (http://pkilambi.fedorapeople.org/sync.patch). 
> Results available at https://beaker.engineering.redhat.com/jobs/315608

Test completed successfully.  In /var/log/pulp/pulp.log, I noticed 4 retry instances, by way of the new log.info() call in pulp/server/api/synchronizers.py::import_package()

> # grep "Retrying" /var/log/pulp/pulp.log *
> /var/log/pulp/pulp.log:2012-10-12 18:27:13,026 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5
> /var/log/pulp/pulp.log:2012-10-12 18:27:25,159 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5
> /var/log/pulp/pulp.log:2012-10-12 18:27:35,289 6891:140032658614016: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5
> /var/log/pulp/pulp.log:2012-10-12 18:27:44,834 6891:140032442627840: pulp.server.api.synchronizers:INFO: synchronizers:575 Retrying ...5

I might suggest altering the log.info() to include the package name as well?

> /usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py
>  log.info("Retrying '%s' ['%s'] ... %s" % (package, repo, num_retries))

Comment 12 James Laska 2012-10-16 12:18:12 UTC
From the bug Prad cloned into the Pulp product ...

(In reply to comment #1)
> commit b7e12df0d0e767f1f20ec8cb24e475aafbcbcc9c

Comment 15 James Laska 2012-10-22 20:09:01 UTC
My reproducer for this issue is underway in beaker (https://beaker.engineering.redhat.com/jobs/320405).  I'll update with status when the test completes.

Comment 16 James Laska 2012-10-23 11:30:13 UTC
Testing with pulp-1.1.14-1.el6cf, I was unable to reproduce the original problem using the same reproducer

https://beaker.engineering.redhat.com/jobs/320405

Comment 17 James Laska 2012-10-23 11:32:51 UTC
Created attachment 632038 [details]
katello-debug-20121023073026.tar.gz

katello-debug logs from a fixed system.

Comment 19 errata-xmlrpc 2012-12-04 19:56:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1543.html


Note You need to log in before you can comment on or make changes to this bug.