Bug 682992

Summary:	parallel uploads still failing
Product:	[Retired] Pulp	Reporter:	Daniel Mach <dmach>
Component:	z_other	Assignee:	Jeff Ortel <jortel>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	high	Docs Contact:
Priority:	high
Version:	unspecified	CC:	dgao, tsanders
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-08-16 13:59:03 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	563609, 641987

Description Daniel Mach 2011-03-08 09:03:22 UTC

run following command at least in 2 parallel processes
$ pulp-admin content upload --repoid=<repo> <rpm>

(I used input duplication in Konsole to start the commands at the exactly same time)

Result: at least one process fails


I believe it's because Pulp can't recover from re-uploading the same file, this should pass when started over and over again:
upload_id = upload_api.upload(pkg_path, checksum, chunksize)
uploaded = upload_api.import_content(pkginfo, upload_id)

import_content fails when a package is already imported

Comment 1 Daniel Mach 2011-03-08 14:34:59 UTC

Consoles get the same input -> commands are executed at the same time.


CONSOLE #1
----------

$ pulp-admin content upload --repoid=<repo> test_data/rpm1/testpkg-1-1.el5.noarch.rpm --nosig
* Starting Package Upload operation. See /var/log/pulp/client.log for more verbose output

* Performing Package Uploads to Pulp server

* Performing Repo Associations 

* Content Upload complete.


CONSOLE #2
----------
$ pulp-admin content upload --repoid=<repo> test_data/rpm1/testpkg-1-1.el5.noarch.rpm --nosig
* Starting Package Upload operation. See /var/log/pulp/client.log for more verbose output

* Performing Package Uploads to Pulp server

* Performing Repo Associations 
error: operation failed: 

$ echo $?
244

Comment 2 Daniel Mach 2011-03-08 14:38:13 UTC

The log says:
IOError: [Errno 2] No such file or directory: '/var/lib/pulp//repos/<repo>/.repodata/other.xml.gz'

Looks like 2 createrepo commands are running at the same time and one of them fails...

Comment 3 Todd Sanders 2011-03-08 17:43:38 UTC

For this sprint, we are going to tighten up the checking/functionality to support simultaneous uploads of the same content.  However, we do recognize a short-coming when it comes to "locking" across our model.  So at this point, simultaneous operations on repositories is going to lead to potential issues.  We will prioritize this in our next sprint, but will not be available for 3/15.

So in the above scenario, I'd recommend refraining from simultaneous repository operations.  Meaning, we should be able to handle:

CONSOLE #1:
$ pulp-admin content upload --repoid=<repoA> test_data/rpm1/testpkg-1-1.el5.noarch.rpm --nosig

CONSOLE #2:
$ pulp-admin content upload --repoid=<repoB> test_data/rpm1/testpkg-1-1.el5.noarch.rpm --nosig

-Todd

Comment 4 Jeff Ortel 2011-03-08 22:59:14 UTC

Eliminated a race condition (related to package import) in package upload.  I was able to produce the race condition and verify the fix by running:

pulp-admin content upload python-urlgrabber-3.9.1-6.fc13.noarch.rpm -r foo1 & \
pulp-admin content upload python-urlgrabber-3.9.1-6.fc13.noarch.rpm -r foo2 &

* Performing Package Uploads to Pulp server
* Starting Package Upload operation. See /var/log/pulp/client.log for more verbose output

* Performing Package Uploads to Pulp server

* Performing Repo Associations 

* Performing Repo Associations 

* Content Upload complete.

* Content Upload complete.

[1]-  Done                    pulp-admin content upload python-urlgrabber-3.9.1-6.fc13.noarch.rpm -r foo1
[2]+  Done                    pulp-admin content upload python-urlgrabber-3.9.1-6.fc13.noarch.rpm -r foo2

This test is probably not as close to parallel as dmach's test using Konsole but should be pretty close.

Comment 5 Jeff Ortel 2011-03-09 02:01:05 UTC

build: 0.146

Comment 6 Preethi Thomas 2011-03-10 21:38:26 UTC

[root@preethi ~]# pulp-admin content upload  /root/upload/gofer-0.20-1.fc14.noarch.rpm -r upload1 --nosig -v & pulp-admin content upload /root/upload/gofer-0.20-1.fc14.noarch.rpm -r upload --nosig -v
[1] 21700
* Starting Package Upload

* Performing Package Uploads to Pulp server
* Starting Package Upload

* Performing Package Uploads to Pulp server
Successfully uploaded [gofer-0.20-1.fc14.noarch.rpm] to server

* Performing Repo Associations 
Successfully uploaded [gofer-0.20-1.fc14.noarch.rpm] to server

* Performing Repo Associations 
Package association Complete for Repo [upload]: 
 Packages: 
gofer-0.20-1.fc14.noarch.rpm 
 
 Files: 
None

* Content Upload complete.
[root@preethi ~]# Package association Complete for Repo [upload1]: 
 Packages: 
gofer-0.20-1.fc14.noarch.rpm 
 
 Files: 
None

* Content Upload complete.

[1]+  Done                    pulp-admin content upload /root/upload/gofer-0.20-1.fc14.noarch.rpm -r upload1 --nosig -v
[root@preethi ~]#  echo $?
0
[root@preethi ~]#

Comment 7 Preethi Thomas 2011-08-16 13:59:03 UTC

Closing with Community Release 15

pulp-0.0.223-4.