Bug 1002041

Summary:	spacewalk-abrt: large file upload fails
Product:	Red Hat Satellite 5	Reporter:	Matej Kollar <mkollar>
Component:	Server	Assignee:	Milan Zázrivec <mzazrivec>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Lukas Pramuk <lpramuk>
Severity:	medium	Docs Contact:
Priority:	high
Version:	560	CC:	cperry, lpramuk, mzazrivec, tkasparek
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	spacewalk-abrt-2.0.1-4	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-10-01 19:59:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	924171, 924199, 982642

Description Matej Kollar 2013-08-28 11:28:44 UTC

Description of problem:

  When crash directory contains large files,

  > spacewalk-abrt --sync

  ends either with "Internal Server Error" or "Network error: Unexpected EOF".
  This is issue for abrt-addon-vmcore crash-directories


Version-Release number of selected component:

  spacewalk-backend-2.0.3-10.el6sat.noarch

How reproducible: almost always/deterministic
sometime few (observed max 2) vmcore crashes are synced.

Steps to Reproduce:

  1. sleep 100 &
  2. kill -s SEGV $!
  3. abrt-cli list # determine crash-direcotry
  4. dd if=/dev/urandom of=$crashdir/reason bs=1M count=100  # pump up some file
  5. spacewalk-abrt --sync

Actual results:
  Upload starts, few files are uploaded, then either
  "Internal Server Error" or "Network error: Unexpected EOF" error is produced.

Expected results:
  Files are uploaded, if limit to file-size is set, it should be evaluated sooner than
  file is fully uploaded. No crash...

Additional info:
  First observed while reproducing
  https://bugzilla.redhat.com/show_bug.cgi?id=982642

Comment 3 Milan Zázrivec 2013-09-03 11:16:06 UTC

The problem here is that whenever the client tries to upload a big file
to the Spacewalk / Satellite server, it takes a long time for the server
to actually process the request (in case of a 100MB file it took ~ 30 minutes),
so by the time the server is ready to respond, the client connection times out
and the whole upload on the client side finishes with the following traceback:

Traceback (most recent call last):
  File "/usr/bin/spacewalk-abrt", line 86, in <module>
    cli.run()
  File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run
    sys.exit(self.main() or 0)
  File "/usr/bin/spacewalk-abrt", line 72, in main
    spacewalk_abrt.abrt.report(self.options.report)
  File "/usr/share/rhn/spacewalk_abrt/abrt.py", line 117, in report
    server.abrt.upload_crash_file(systemid, basename, crash_file_data)
  File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__
    return rpcServer.doCall(method, *args, **kwargs)
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 204, in doCall
    ret = method(*args, **kwargs)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1
    ret = self._request(methodname, params)
  File "/usr/lib/python2.7/site-packages/rhn/rpclib.py", line 384, in _request
    self._handler, request, verbose=self._verbose)
  File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 171, in request
    headers, fd = req.send_http(host, handler)
  File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 725, in send_http
    response = self._connection.getresponse()
  File "/usr/lib/python2.7/site-packages/rhn/connections.py", line 94, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/site-packages/rhn/SSL.py", line 287, in readline
    self._poll(select.POLLIN, 'readline')
  File "/usr/lib/python2.7/site-packages/rhn/SSL.py", line 218, in _poll
    raise TimeoutException, "Connection timed out on %s" % caller_name
<class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was:
Connection timed out on readline

One way to help the situation is make the client not to upload
files if they're not desired -- i.e. if they're over the size limit
or if the file uploading is not allowed at all.

spacewalk.git master: 1d83f1eb87814b1c4b8715197daa50ef7c3d1cc0

Comment 11 Matej Kollar 2013-09-23 11:35:41 UTC

Spacewalk.git master: 3e7bc878aed77e04f92ff91875ffa436b2c2a907

Comment 14 Lukas Pramuk 2013-09-30 09:16:03 UTC

VERIFIED. spacewalk-abrt-2.0.1-4.el6sat

Newly spacewalk-abrt doesn't try to upload huge files.
These files aren't uploaded on SAT anymore.
Used initial reproducer in comment #0

Comment 15 Clifford Perry 2013-10-01 19:59:25 UTC

This bug was fixed with updated packages being released within the RHN Tools channels. 


RHN Tools (for Sat 5.6 GA) Errata text:

https://rhn.redhat.com/errata/RHEA-2013-1391.html