Bug 664026

Summary:	[RFE] upload files by chunks
Product:	[Retired] Pulp	Reporter:	Daniel Mach <dmach>
Component:	z_other	Assignee:	Jeff Ortel <jortel>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	unspecified	CC:	jason.dobies, jortel, skarmark, tsanders
Target Milestone:	---	Keywords:	FutureFeature, Triaged
Target Release:	Sprint 21
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-08-15 15:14:34 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	578256, 647488, 673012

Description Daniel Mach 2010-12-17 18:49:05 UTC

Files of any sizes have to be supported.
AFAIK pulp is now limited by POST request size.

Uploading by chunks should also help when an upload fails.
Retransmitting a single chunk is definitely cheaper than re-uploading the whole file.

Comment 1 Daniel Mach 2011-02-15 13:38:58 UTC

Uploading big files (600M) fails with a memory error:

Traceback (most recent call last):
  File "./pulp-admin", line 147, in ?
    admin.main()
  File "/var/www/django/pulp/src/pulp/client/cli/base.py", line 98, in main
    command.main(args[1:])
  File "/var/www/django/pulp/src/pulp/client/core/base.py", line 118, in main
    action.main(args[1:])
  File "/var/www/django/pulp/src/pulp/client/core/base.py", line 231, in main
    self.run()
  File "/var/www/django/pulp/src/pulp/client/core/package.py", line 293, in run
    pkgstream = base64.b64encode(open(frpm).read())
  File "/usr/lib/python2.4/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
MemoryError

Comment 2 Jeff Ortel 2011-02-22 14:36:22 UTC

Build: 0.140

Comment 3 Daniel Mach 2011-02-22 17:38:15 UTC

Would it be feasible not to use base64?
That would boost upload speed by 30% at least.

Comment 4 Jeff Ortel 2011-02-22 22:41:07 UTC

Yes.  Done.
Good suggestion.

Comment 5 Preethi Thomas 2011-03-07 20:25:43 UTC

verified
[root@preethi ~]# rpm -q pulp
pulp-0.0.144-1.fc14.noarch

[root@preethi ~]# pulp-admin content upload -r upload2 /root/iso/Fedora-14-x86_64-Live-Desktop.iso  --help
Usage: pulp-admin <options> content upload <options>

Options:
  -h, --help            show this help message and exit
  --dir=DIR             process content from this directory
  -r REPOIDS, --repoid=REPOIDS
                        Optional repoid, to associate the uploaded content
  --nosig               pushes unsigned content(rpms)
  --chunksize=CHUNK     chunk size to use for uploads. Default:10485760
  -v, --verbose         verbose output.

[root@preethi ~]# pulp-admin content upload -r upload2 /root/iso/Fedora-14-x86_64-Live-Desktop.iso  --chunksize=20485760 -v
* Starting Package Upload

* Performing Package Uploads to Pulp server
Package [Fedora-14-x86_64-Live-Desktop.iso] already exists on the server with checksum [8f81741e938924f68965b5dceac7013d72fe875725b53f61bebd36e740f79b99]

* Performing Repo Associations 
Package association Complete for Repo [upload2]: 
 Packages: 
None 
 
 Files: 
Fedora-14-x86_64-Live-Desktop.iso

* Content Upload complete.
[root@preethi ~]#

Comment 6 Preethi Thomas 2011-08-15 15:14:34 UTC

Closing with current community release

pulp-0.0.223