Bug 808183 - parallel syncing 2 repos with same feed syncs the distribution to only one repo
Summary: parallel syncing 2 repos with same feed syncs the distribution to only one repo
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: user-experience
Version: 1.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Pradeep Kilambi
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-03-29 18:36 UTC by Preethi Thomas
Modified: 2013-09-09 16:34 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-05-25 14:16:19 UTC
Embargoed:


Attachments (Terms of Use)

Description Preethi Thomas 2012-03-29 18:36:47 UTC
Description of problem:
If you sync 2 repos both from the same feed that has distribution, it only syncs the distribution to 1 repo.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. create repo with a feed which has a distribution
2. create a second repo with same feed as above (use relativepath)
3. sync the first repo
4. while the first repo is syncing run sync on the 2nd repo
  
Actual results:
I tried a 2 f16 repos on which the sync said completed but only one repo had the distribution

I tried it a second time with an f15 repo

[root@pulp-v1-server ~]# pulp-admin repo sync --id f15-new -F
Sync for repository f15-new started
Sync: Finished
1112/2941 new items downloaded
1829/2941 existing items processed

Item Details: 
Tree Files: 6/6
RPMs: 2935/2935

[root@pulp-v1-server ~]# pulp-admin repo sync --id f15 -F
Sync for repository f15 started
Sync: Error

Item Details: 
Tree Files: 6/6
RPMs: 2935/2935
error:  DuplicateKeyError: E11000 duplicate key error index: pulp_database.distribution.$_id_  dup key: { : "ks-Fedora-Fedora-15-x86_64" }




Expected results:


Additional info:

From the log

2012-03-29 14:17:13,645 15656:139732480665344: pulp.server.tasking.task:ERROR: task:468 Task failed: Task 6b6d6dae-79c7-11e1-96f1-52540067ae07: _sync(f15, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f15fc0b0050>, skip={}, max_speed=None, threads=None, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f15fc0b0d90>>)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 414, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 278, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 368, in fetch_content
    synchronizer.add_distribution_from_dir(repo_dir, repo_id, skip_dict)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 345, in add_distribution_from_dir
    self._process_repo_images(dir, repo)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 467, in _process_repo_images
    arch=treeinfo['arch'], repoids=[repo['id']])
  File "/usr/lib/python2.6/site-packages/pulp/server/api/distribution.py", line 59, in create
    self.collection.insert(d, safe=True)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 80, in retry
    return method(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 270, in insert
    check_keys, safe, kwargs), safe)
  File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 642, in _send_message
    return self.__check_response_to_last_error(response)
  File "/usr/lib64/python2.6/site-packages/pymongo/connection.py", line 611, in __check_response_to_last_error
    raise DuplicateKeyError(error["err"])
DuplicateKeyError: E11000 duplicate key error index: pulp_database.distribution.$_id_  dup key: { : "ks-Fedora-Fedora-15-x86_64" }

Comment 1 Pradeep Kilambi 2012-04-13 19:14:06 UTC
fixed! commit eaae98fe1b3c79f4fc48be160d3af4cf8b0505d0



$ pulp-admin repo create --id=RHEL6_2 --feed=http://cdn.rcm-qa.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/ --preserve_metadata
Successfully created repository [ RHEL6_2 ]

$ pulp-admin repo create --id=RHEL6_2_ --feed=http://cdn.rcm-qa.redhat.com/content/dist/rhel/server/6/6.2/x86_64/os/ --preserve_metadata --relativepath=duplicate_el6
Successfully created repository [ RHEL6_2_ ]

$ pulp-admin repo sync -F --id=RHEL6_2
Sync for repository RHEL6_2 started
Sync: Finished
3915/7105 new items downloaded
3190/7105 existing items processed

Item Details: 
Tree Files: 6/6
RPMs: 7099/7099

$ pulp-admin repo sync -F --id=RHEL6_2_
Sync for repository RHEL6_2_3 started
Sync: Finished
3190/7105 new items downloaded
3915/7105 existing items processed

Item Details: 
Tree Files: 6/6
RPMs: 7099/7099

Comment 2 Jeff Ortel 2012-04-19 15:44:39 UTC
build: 1.1.3

Comment 3 Preethi Thomas 2012-04-20 17:55:50 UTC
[root@pulp-v1-1-server ~]# rpm -q pulp
pulp-1.1.3-1.fc16.noarch

[root@pulp-v1-1-server ~]# pulp-admin repo sync --id=f16 -FSync for repository f16 started
Sync: Finished
1773/3113 new items downloaded
1340/3113 existing items processed

Item Details: 
Tree Files: 6/6
RPMs: 3107/3107

[root@pulp-v1-1-server ~]# pulp-admin repo sync --id=f16-new -FSync for repository f16-new started
Sync: Finished
1340/3113 new items downloaded
1773/3113 existing items processed

Item Details: 
Tree Files: 6/6
RPMs: 3107/3107


[root@pulp-v1-1-server ~]# pulp-admin distribution list
+------------------------------------------+
       List of Available Distributions
+------------------------------------------+

Id                    	ks-Fedora-Fedora-16-x86_64
Description           	ks-Fedora-Fedora-16-x86_64
Family                	Fedora                   
Variant               	Fedora                   
Version               	16                       
Arch                  	x86_64                   
URL                   	http://pulp-v1-1-server.usersys.redhat.com/pulp/ks/f16-new/
 			http://pulp-v1-1-server.usersys.redhat.com/pulp/ks/released/F-16/GOLD/Fedora/x86_64/os/
Timestamp             	2011-11-02T23:10:12      


[root@pulp-v1-1-server ~]#

Comment 4 Preethi Thomas 2012-05-25 14:16:19 UTC
Pulp v1.1 Release


Note You need to log in before you can comment on or make changes to this bug.