Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 664026 - [RFE] upload files by chunks
Summary: [RFE] upload files by chunks
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: z_other
Version: unspecified
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: Sprint 21
Assignee: Jeff Ortel
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks: 578256 verified-to-close 673012
TreeView+ depends on / blocked
 
Reported: 2010-12-17 18:49 UTC by Daniel Mach
Modified: 2011-08-15 15:14 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-08-15 15:14:34 UTC


Attachments (Terms of Use)

Description Daniel Mach 2010-12-17 18:49:05 UTC
Files of any sizes have to be supported.
AFAIK pulp is now limited by POST request size.

Uploading by chunks should also help when an upload fails.
Retransmitting a single chunk is definitely cheaper than re-uploading the whole file.

Comment 1 Daniel Mach 2011-02-15 13:38:58 UTC
Uploading big files (600M) fails with a memory error:

Traceback (most recent call last):
  File "./pulp-admin", line 147, in ?
    admin.main()
  File "/var/www/django/pulp/src/pulp/client/cli/base.py", line 98, in main
    command.main(args[1:])
  File "/var/www/django/pulp/src/pulp/client/core/base.py", line 118, in main
    action.main(args[1:])
  File "/var/www/django/pulp/src/pulp/client/core/base.py", line 231, in main
    self.run()
  File "/var/www/django/pulp/src/pulp/client/core/package.py", line 293, in run
    pkgstream = base64.b64encode(open(frpm).read())
  File "/usr/lib/python2.4/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
MemoryError

Comment 2 Jeff Ortel 2011-02-22 14:36:22 UTC
Build: 0.140

Comment 3 Daniel Mach 2011-02-22 17:38:15 UTC
Would it be feasible not to use base64?
That would boost upload speed by 30% at least.

Comment 4 Jeff Ortel 2011-02-22 22:41:07 UTC
Yes.  Done.
Good suggestion.

Comment 5 Preethi Thomas 2011-03-07 20:25:43 UTC
verified
[root@preethi ~]# rpm -q pulp
pulp-0.0.144-1.fc14.noarch

[root@preethi ~]# pulp-admin content upload -r upload2 /root/iso/Fedora-14-x86_64-Live-Desktop.iso  --help
Usage: pulp-admin <options> content upload <options>

Options:
  -h, --help            show this help message and exit
  --dir=DIR             process content from this directory
  -r REPOIDS, --repoid=REPOIDS
                        Optional repoid, to associate the uploaded content
  --nosig               pushes unsigned content(rpms)
  --chunksize=CHUNK     chunk size to use for uploads. Default:10485760
  -v, --verbose         verbose output.

[root@preethi ~]# pulp-admin content upload -r upload2 /root/iso/Fedora-14-x86_64-Live-Desktop.iso  --chunksize=20485760 -v
* Starting Package Upload

* Performing Package Uploads to Pulp server
Package [Fedora-14-x86_64-Live-Desktop.iso] already exists on the server with checksum [8f81741e938924f68965b5dceac7013d72fe875725b53f61bebd36e740f79b99]

* Performing Repo Associations 
Package association Complete for Repo [upload2]: 
 Packages: 
None 
 
 Files: 
Fedora-14-x86_64-Live-Desktop.iso

* Content Upload complete.
[root@preethi ~]#

Comment 6 Preethi Thomas 2011-08-15 15:14:34 UTC
Closing with current community release

pulp-0.0.223


Note You need to log in before you can comment on or make changes to this bug.