Bug 797195

Summary:	Syncing multiple repos concurrently could cause missing symlinks
Product:	[Retired] Pulp	Reporter:	Pradeep Kilambi <pkilambi>
Component:	user-experience	Assignee:	John Matthews <jmatthew>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	high	Docs Contact:
Priority:	high
Version:	1.1.0	CC:	inecas, pkilambi
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-05-25 14:15:23 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	790595

Description Pradeep Kilambi 2012-02-24 14:22:50 UTC

Description of problem:

Adding Ivan's comments from bug 795819

Ivan Necas 2012-02-24 05:58:49 EST

I can confirm this error, but I'm not sure it's related directly to the
description of the original bug. When reproducing this one, I've seen the
packages not being positioned correctly, but I haven't seen  the 'No such file
or directory' error. Also on the machine with the original bug, there were no
packages in incorrect place.

The original bug appears in packages synchronized from remote repo.The strange
thing is that for some packages synchronized from remote repo the link is not
created. However it seems the package should be there (when cloning, it shouts
that the package is missing). So at the end, most (but not all) links are
created.

Another observation:

Pulp says:

 2012-02-21 17:58:16,785 11788:140544395171584:
pulp.server.api.synchronizers:INFO       : synchronizers:783 YumSynchronizer
reported 5632 successes, 4729 downloads, 1 er       rors

But in grinder there are only 4729 lines with line like:

grinder.log:2012-02-21 17:55:35,435 11788:140544374191872:
grinder.BaseFetch:INFO: activeobject:160 Create a link in repo directory for
the package at
/var/lib/pulp//repos/Seattle/Library/content/dist/rhel/server/6/6.1/x86_64/os//Packages/zlib-1.2.3-25.el6.i686.rpm
to
../../../../../../../../../../../../packages/zlib/1.2.3/25.el6/i686/36a/zlib-1.2.3-25.el6.i686.rpm

So it seems like the link was not created for packages, that were already
downloaded for another repo (e.g. 6.2)

Althought there are also lines in grinder.log like:

grinder.log:2012-02-21 17:54:09,017 11788:140544374191872:
grinder.BaseFetch:INFO: activeobject:160 Symlink missing in repo directory.
Creating link
/var/lib/pulp//repos/Seattle/Library/content/dist/rhel/server/6/6.1/x86_64/os//Packages/xulrunner-1.9.2.20-2.el6_1.x86_64.rpm
to
../../../../../../../../../../../../packages/xulrunner/1.9.2.20/2.el6_1/x86_64/dd4/xulrunner-1.9.2.20-2.el6_1.x86_64.rpm

There are only 177 for the repo. So we still missing links for over 700
packages

I suspect this lines starting grinder/BaseFetch.py:193 to cause this troubles:

        # Acquire a write lock so no other process duplicates the effort
        grinder_write_locker = GrinderLock(filePath + '.lock')
        existing_lock_pid = grinder_write_locker.readlock()
        new_pid = os.getpid()
        if existing_lock_pid and int(existing_lock_pid) != new_pid and
grinder_write_locker.isvalid(existing_lock_pid):
            # If there is an existing write pid
            # and if the pid is not same as the current pid
            # and pid is valid there is another process alive
            # and handling this, exit here.
            LOG.debug("another process is already handling this path [%s] and
is alive; no need to process this again " % filePath)
            return (BaseFetch.STATUS_NOOP,None)

Bare in mind that synchronization of 6.1 and 6.2 repos runs concurrently, there
might be situations, that the package is being downloaded for another repo. In
this case it's skipped and the symlink is no longer created.

Comment 1 John Matthews 2012-02-27 21:38:25 UTC

Added a new status of REQUEUE, which allows us to add this item back to the item queue so the symlink can be created after the package download has completed.

http://git.fedorahosted.org/git/?p=grinder.git;a=commitdiff;h=ba659f8ab36398f55f6fd17ecaa64b1aedc4858d

Will be in Grinder 0.139

Comment 2 Jeff Ortel 2012-02-29 23:16:15 UTC

build: 0.270

Comment 3 Preethi Thomas 2012-03-06 20:49:42 UTC

verified

[root@preethi-el6-pulp ~]# rpm -q pulp
pulp-0.0.271-1.el6.noarch


[root@preethi-el6-pulp ~]# ls -l /var/lib/pulp/repos/released/RHEL-6/6.1/Server/x86_64/os/Packages/
Display all 3480 possibilities? (y or n)
[root@preethi-el6-pulp ~]# ls -l /var/lib/pulp/repos/released/RHEL-6/6.2/Server/x86_64/os/Packages/
6.2/
[root@preethi-el6-pulp ~]# ls -l /var/lib/pulp/repos/released/RHEL-6/6.2/Server/x86_64/os/Packages/
Display all 3529 possibilities? (y or n)
[root@preethi-el6-pulp ~]# pulp-admin repo list
+------------------------------------------+
       List of Available Repositories
+------------------------------------------+

Id                 	rhel6-2                  
Name               	rhel6-2                  
Repo URL           	https://preethi-el6-pulp.usersys.redhat.com/pulp/repos/released/RHEL-6/6.2/Server/x86_64/os/Server/
Feed URL           	http://download.devel.redhat.com/released/RHEL-6/6.2/Server/x86_64/os/Server/
Feed Type          	remote                   
Content Type       	yum                      
Feed Certs         	CA:No   Cert:No
Consumer Certs     	CA:No   Cert:No
Architecture       	noarch                   
Sync Schedule      	2012-02-29T16:20:00-05:00/PT2M
Packages           	3529                     
Files              	0                        
Distributions      	None                     
Publish            	True                     
Clones             	[]                       
Groups             	None                     
Filters            	[]                       
Notes              	{}                       
Preserve Metadata  	False                    
Checksum Type      	sha256                   


Id                 	rhel5-8                  
Name               	rhel5-8                  
Repo URL           	https://preethi-el6-pulp.usersys.redhat.com/pulp/repos/released/RHEL-5-Client/U8/x86_64/os/Client/
Feed URL           	http://download.devel.redhat.com/released/RHEL-5-Client/U8/x86_64/os/Client/
Feed Type          	remote                   
Content Type       	yum                      
Feed Certs         	CA:No   Cert:No
Consumer Certs     	CA:No   Cert:No
Architecture       	noarch                   
Sync Schedule      	2012-02-29T16:20:00-05:00/PT2M
Packages           	2141                     
Files              	0                        
Distributions      	None                     
Publish            	True                     
Clones             	[]                       
Groups             	None                     
Filters            	[]                       
Notes              	{}                       
Preserve Metadata  	False                    
Checksum Type      	sha                      


Id                 	rhel6-1                  
Name               	rhel6-1                  
Repo URL           	https://preethi-el6-pulp.usersys.redhat.com/pulp/repos/released/RHEL-6/6.1/Server/x86_64/os/Server/
Feed URL           	http://download.devel.redhat.com/released/RHEL-6/6.1/Server/x86_64/os/Server/
Feed Type          	remote                   
Content Type       	yum                      
Feed Certs         	CA:No   Cert:No
Consumer Certs     	CA:No   Cert:No
Architecture       	noarch                   
Sync Schedule      	2012-02-29T16:20:00-05:00/PT2M
Packages           	3480                     
Files              	0                        
Distributions      	None                     
Publish            	True                     
Clones             	[]                       
Groups             	None                     
Filters            	[]                       
Notes              	{}                       
Preserve Metadata  	False                    
Checksum Type      	sha256

Comment 4 Jeff Weiss 2012-03-21 17:11:41 UTC

I verified the update fixed this problem on katello also.

Comment 5 Preethi Thomas 2012-05-25 14:15:23 UTC

Pulp v1.1 Release