Bug 983968 - redhat-support-tool: file splitting for uploading
redhat-support-tool: file splitting for uploading
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: redhat-support-tool (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Keith Robertson
Filip Holec
:
Depends On:
Blocks: 982704 1036771 1036772
  Show dependency treegraph
 
Reported: 2013-07-12 08:15 EDT by Florian Weimer
Modified: 2016-07-03 21:36 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1036771 1036772 (view as bug list)
Environment:
Last Closed: 2014-06-13 07:21:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Florian Weimer 2013-07-12 08:15:30 EDT
It should not be necessary to split large files into smaller files on disk.  If you create just one 100 MB slice in memory (or on disk), upload that, then proceed to the next chunk, you won't have to ask for a temporary directory with sufficient storage.

Alternatively, you could enhance Connection.doUpload() in redhat-support-lib-python-0.9.4_master/src/redhat_support_lib/web/connection.py so that it takes a size, a name, and a file-like object (or a generator for the file content), instead of a path on disk.  This way, you could directly upload pieces of the original file, without the need for keeping even one 100 MB slice in memory.
Comment 5 Mark Huth 2014-04-02 19:33:10 EDT
Hmm, there were *lots* of code changes in this BZ.  I would prefer it was formally tested rather than just a code review.  I will try to come up some reproducers for testing (but first I gotta remember how I was testing it coz it was a long time ago now ...)

-- Mark
Comment 6 Mark Huth 2014-04-22 02:16:06 EDT
Testing that split files aren't stored on disk for uploading.
-----------------------------------------------------------------------------------------

# rpm -qa | grep redhat-support
redhat-support-tool-0.9.6-0.el6.noarch
redhat-support-lib-python-0.9.6-0.el6.noarch

-----------------------------------------------------------------------------------------

Splitting network.cap into 20mb chunks and uploading to the ticket ...

# redhat-support-tool 
...
Command (? for help): addattachment -c 01079464 /home/mhuth/tickets/network.cap -s 20
Compressing /home/mhuth/tickets/network.cap for upload ... completed successfully.
Please provide a description or enter to accept default (or 'q' to exit): testing bz983968
Uploading network.cap.gz to the case ... completed successfully.

-----------------------------------------------------------------------------------------

network.cap is 221mb in size ...

# ll -h /home/mhuth/tickets/network.cap 
-rw-rw-r--. 1 mhuth mhuth 221M Feb 18 14:58 /home/mhuth/tickets/network.cap


It is compressed (down to ~45mb) before uploading and the compressed file is stored on disk ...

[root@mhuth-laptop tmp]# ll -tr /tmp
total 324
...
drwx------. 2 root    root     4096 Apr 22 16:05 tmpvnmUtV
[root@mhuth-laptop tmp]# ll -tr /tmp/tmpvnmUtV/
total 45452
-rw-r--r--. 1 root root 46541526 Apr 22 16:05 network.cap.gz

... but none of the 20mb file chunks are stored on disk.

-----------------------------------------------------------------------------------------

The file as it appears in the ticket:

 	network.cap.gz.002	Huth, Mark (4/22/2014 4:07 PM)	https://api.access.redhat.com/rs/cases/01079464...	4,598,486
	network.cap.gz.001	Huth, Mark (4/22/2014 4:07 PM)	https://api.access.redhat.com/rs/cases/01079464...	20,971,520
	network.cap.gz.000	Huth, Mark (4/22/2014 4:06 PM)	https://api.access.redhat.com/rs/cases/01079464...	20,971,520

... and the comment that is created for the split file ...

[RHST] The following split files were uploaded to the case:

    network.cap.gz.000
    network.cap.gz.001
    network.cap.gz.002
Comment 7 Mark Huth 2014-04-24 02:12:21 EDT
Tested a number of the other new options introduced in this BZ, eg -s, -f and -z:

<rhst>
Command (? for help): help addattachment

Usage: addattachment -c CASENUMBER [options] /path/to/file

Use the 'addattachment' command to add an attachment to a case.
Options:
  -h, --help            show this help message and exit
  -c CASENUMBER, --casenumber=CASENUMBER
                        The case number from which the comment should be
                        added. (required)
  -d DESCRIPTION, --description=DESCRIPTION
                        A description for the attachment. The Red Hat Support
                        Tool will generate a default description for the
                        attachment if none is provided that contains the name
                        of the file and the RPM package to which it belongs if
                        available. (optional)
  -x, --no-split        Do not attempt to split uploaded files, upload may
                        fail as a result if an alternative destination is not
                        available.
  -s, --split           The uploaded attachment file will be intentionally
                        split.  An optional size parameter (in MB) can be
                        supplied and the attachment will be split into 'size'
                        (MB) chunks.  Default/Maximum chunk size: 1024 (MB)
  -f, --use-ftp         Upload via FTP to dropbox.redhat.com instead of the
                        Red Hat Customer Portal.
  -z, --no-compress     If the attachment file is uncompressed, don't compress
                        it for upload.
</rhst>

They all seemed to work ok in my testing.
Comment 8 Ludek Smid 2014-06-13 07:21:15 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.