Bug 797195 - Syncing multiple repos concurrently could cause missing symlinks
Summary: Syncing multiple repos concurrently could cause missing symlinks
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: user-experience
Version: 1.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: John Matthews
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks: 790595
TreeView+ depends on / blocked
 
Reported: 2012-02-24 14:22 UTC by Pradeep Kilambi
Modified: 2013-09-09 16:33 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2012-05-25 14:15:23 UTC
Embargoed:


Attachments (Terms of Use)

Description Pradeep Kilambi 2012-02-24 14:22:50 UTC
Description of problem:

Adding Ivan's comments from bug 795819

Ivan Necas 2012-02-24 05:58:49 EST

I can confirm this error, but I'm not sure it's related directly to the
description of the original bug. When reproducing this one, I've seen the
packages not being positioned correctly, but I haven't seen  the 'No such file
or directory' error. Also on the machine with the original bug, there were no
packages in incorrect place.

The original bug appears in packages synchronized from remote repo.The strange
thing is that for some packages synchronized from remote repo the link is not
created. However it seems the package should be there (when cloning, it shouts
that the package is missing). So at the end, most (but not all) links are
created.

Another observation:

Pulp says:

 2012-02-21 17:58:16,785 11788:140544395171584:
pulp.server.api.synchronizers:INFO       : synchronizers:783 YumSynchronizer
reported 5632 successes, 4729 downloads, 1 er       rors

But in grinder there are only 4729 lines with line like:

grinder.log:2012-02-21 17:55:35,435 11788:140544374191872:
grinder.BaseFetch:INFO: activeobject:160 Create a link in repo directory for
the package at
/var/lib/pulp//repos/Seattle/Library/content/dist/rhel/server/6/6.1/x86_64/os//Packages/zlib-1.2.3-25.el6.i686.rpm
to
../../../../../../../../../../../../packages/zlib/1.2.3/25.el6/i686/36a/zlib-1.2.3-25.el6.i686.rpm

So it seems like the link was not created for packages, that were already
downloaded for another repo (e.g. 6.2)

Althought there are also lines in grinder.log like:

grinder.log:2012-02-21 17:54:09,017 11788:140544374191872:
grinder.BaseFetch:INFO: activeobject:160 Symlink missing in repo directory.
Creating link
/var/lib/pulp//repos/Seattle/Library/content/dist/rhel/server/6/6.1/x86_64/os//Packages/xulrunner-1.9.2.20-2.el6_1.x86_64.rpm
to
../../../../../../../../../../../../packages/xulrunner/1.9.2.20/2.el6_1/x86_64/dd4/xulrunner-1.9.2.20-2.el6_1.x86_64.rpm

There are only 177 for the repo. So we still missing links for over 700
packages

I suspect this lines starting grinder/BaseFetch.py:193 to cause this troubles:

        # Acquire a write lock so no other process duplicates the effort
        grinder_write_locker = GrinderLock(filePath + '.lock')
        existing_lock_pid = grinder_write_locker.readlock()
        new_pid = os.getpid()
        if existing_lock_pid and int(existing_lock_pid) != new_pid and
grinder_write_locker.isvalid(existing_lock_pid):
            # If there is an existing write pid
            # and if the pid is not same as the current pid
            # and pid is valid there is another process alive
            # and handling this, exit here.
            LOG.debug("another process is already handling this path [%s] and
is alive; no need to process this again " % filePath)
            return (BaseFetch.STATUS_NOOP,None)

Bare in mind that synchronization of 6.1 and 6.2 repos runs concurrently, there
might be situations, that the package is being downloaded for another repo. In
this case it's skipped and the symlink is no longer created.

Comment 1 John Matthews 2012-02-27 21:38:25 UTC
Added a new status of REQUEUE, which allows us to add this item back to the item queue so the symlink can be created after the package download has completed.

http://git.fedorahosted.org/git/?p=grinder.git;a=commitdiff;h=ba659f8ab36398f55f6fd17ecaa64b1aedc4858d

Will be in Grinder 0.139

Comment 2 Jeff Ortel 2012-02-29 23:16:15 UTC
build: 0.270

Comment 3 Preethi Thomas 2012-03-06 20:49:42 UTC
verified

[root@preethi-el6-pulp ~]# rpm -q pulp
pulp-0.0.271-1.el6.noarch


[root@preethi-el6-pulp ~]# ls -l /var/lib/pulp/repos/released/RHEL-6/6.1/Server/x86_64/os/Packages/
Display all 3480 possibilities? (y or n)
[root@preethi-el6-pulp ~]# ls -l /var/lib/pulp/repos/released/RHEL-6/6.2/Server/x86_64/os/Packages/
6.2/
[root@preethi-el6-pulp ~]# ls -l /var/lib/pulp/repos/released/RHEL-6/6.2/Server/x86_64/os/Packages/
Display all 3529 possibilities? (y or n)
[root@preethi-el6-pulp ~]# pulp-admin repo list
+------------------------------------------+
       List of Available Repositories
+------------------------------------------+

Id                 	rhel6-2                  
Name               	rhel6-2                  
Repo URL           	https://preethi-el6-pulp.usersys.redhat.com/pulp/repos/released/RHEL-6/6.2/Server/x86_64/os/Server/
Feed URL           	http://download.devel.redhat.com/released/RHEL-6/6.2/Server/x86_64/os/Server/
Feed Type          	remote                   
Content Type       	yum                      
Feed Certs         	CA:No   Cert:No
Consumer Certs     	CA:No   Cert:No
Architecture       	noarch                   
Sync Schedule      	2012-02-29T16:20:00-05:00/PT2M
Packages           	3529                     
Files              	0                        
Distributions      	None                     
Publish            	True                     
Clones             	[]                       
Groups             	None                     
Filters            	[]                       
Notes              	{}                       
Preserve Metadata  	False                    
Checksum Type      	sha256                   


Id                 	rhel5-8                  
Name               	rhel5-8                  
Repo URL           	https://preethi-el6-pulp.usersys.redhat.com/pulp/repos/released/RHEL-5-Client/U8/x86_64/os/Client/
Feed URL           	http://download.devel.redhat.com/released/RHEL-5-Client/U8/x86_64/os/Client/
Feed Type          	remote                   
Content Type       	yum                      
Feed Certs         	CA:No   Cert:No
Consumer Certs     	CA:No   Cert:No
Architecture       	noarch                   
Sync Schedule      	2012-02-29T16:20:00-05:00/PT2M
Packages           	2141                     
Files              	0                        
Distributions      	None                     
Publish            	True                     
Clones             	[]                       
Groups             	None                     
Filters            	[]                       
Notes              	{}                       
Preserve Metadata  	False                    
Checksum Type      	sha                      


Id                 	rhel6-1                  
Name               	rhel6-1                  
Repo URL           	https://preethi-el6-pulp.usersys.redhat.com/pulp/repos/released/RHEL-6/6.1/Server/x86_64/os/Server/
Feed URL           	http://download.devel.redhat.com/released/RHEL-6/6.1/Server/x86_64/os/Server/
Feed Type          	remote                   
Content Type       	yum                      
Feed Certs         	CA:No   Cert:No
Consumer Certs     	CA:No   Cert:No
Architecture       	noarch                   
Sync Schedule      	2012-02-29T16:20:00-05:00/PT2M
Packages           	3480                     
Files              	0                        
Distributions      	None                     
Publish            	True                     
Clones             	[]                       
Groups             	None                     
Filters            	[]                       
Notes              	{}                       
Preserve Metadata  	False                    
Checksum Type      	sha256

Comment 4 Jeff Weiss 2012-03-21 17:11:41 UTC
I verified the update fixed this problem on katello also.

Comment 5 Preethi Thomas 2012-05-25 14:15:23 UTC
Pulp v1.1 Release


Note You need to log in before you can comment on or make changes to this bug.