Bug 1177044

Summary: Using POST request rhnpush fails
Product: [Community] Spacewalk Reporter: Nick Tailor <ntailor>
Component: ServerAssignee: Michael Mráka <mmraka>
Status: CLOSED NOTABUG QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.2   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-07 19:50:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1484117    

Description Nick Tailor 2014-12-23 23:02:18 UTC 
It appears the rhnpush fails running out of memory when writting from buffer to whatever address space it trying to write to. I have confirmed its not physical space and I have also increased the buffer size in the rhn_pkg.py file with no such luck.

It was working and all of a sudden on this package which is about 455M in size it keeps failing. How can we fix this?

Script running 
=======================
[root@kam1odapp99<DEV>:~]# perl spacewalk-debian-sync.pl --url http://ftp.us.debian.org/debian/dists/stable/main/binary-amd64/ --username space-admin --password blahblah --channel debian_7.7_amd64
INFO: Repo URL: http://ftp.us.debian.org/debian/dists/stable/main/binary-amd64/
INFO: Debian root is http://ftp.us.debian.org/debian/
INFO: Fetching Packages.gz... done
INFO: Packages in repo:         35923
INFO: Packages already synced:  16302
INFO: Packages to sync:         19621
INFO: 1/19621 : w3af-console_1.0-rc3svn3489-1_all.deb
Connecting to https://localhost/APP
url is https://localhost/PACKAGE-PUSH
Result codes: 200 OK
Computing checksum and package info. This may take some time ...
Package /root/test/w3af-console_1.0-rc3svn3489-1_all.deb already exists on the RHN Server-- Skipping Upload....
INFO: 2/19621 : openclipart2-png_2.0+dfsg-1_all.deb
Connecting to https://localhost/APP
url is https://localhost/PACKAGE-PUSH
Result codes: 200 OK
Computing checksum and package info. This may take some time ...
Package /root/test/openclipart2-png_2.0+dfsg-1_all.deb Not Found on RHN Server -- Uploading
Uploading package /root/test/openclipart2-png_2.0+dfsg-1_all.deb
Using POST request
Internal server error 500 Internal Server Error
Error pushing /root/test/openclipart2-png_2.0+dfsg-1_all.deb:  (500)
1
Waiting 4 seconds and trying again...
Uploading package /root/test/openclipart2-png_2.0+dfsg-1_all.deb
Using POST request
Internal server error 500 Internal Server Error
Error pushing /root/test/openclipart2-png_2.0+dfsg-1_all.deb:  (500)
1
Waiting 5 seconds and trying again...
Uploading package /root/test/openclipart2-png_2.0+dfsg-1_all.deb
Using POST request
Internal server error 500 Internal Server Error
Error pushing /root/test/openclipart2-png_2.0+dfsg-1_all.deb:  (500)
1
Waiting 1 seconds and trying again...
Giving up after 3 attempts
ERROR: rhnpush failed

=========================
apache error log below

======================
[Mon Dec 22 14:55:44 2014] [error] Exception reported from kam1odapp99.connex.bclc.com
[Mon Dec 22 14:55:44 2014] [error] Time: Mon Dec 22 14:55:44 2014
[Mon Dec 22 14:55:44 2014] [error] Exception type <type 'exceptions.IOError'>
[Mon Dec 22 14:55:44 2014] [error] Exception while handling function server.apacheUploadServer._wrapper
[Mon Dec 22 14:55:44 2014] [error] Request object information:
[Mon Dec 22 14:55:44 2014] [error] URI: /PACKAGE-PUSH
[Mon Dec 22 14:55:44 2014] [error] Remote Host: localhost
[Mon Dec 22 14:55:44 2014] [error] Server Name: localhost:443
[Mon Dec 22 14:55:44 2014] [error] Headers passed in:
[Mon Dec 22 14:55:44 2014] [error] \tAccept-Encoding: identity
[Mon Dec 22 14:55:44 2014] [error] \tCONTENT_LENGTH: 477100858
[Mon Dec 22 14:55:44 2014] [error] \tCONTENT_TYPE: application/x-rpm
[Mon Dec 22 14:55:44 2014] [error] \tDOCUMENT_ROOT: /var/www/html
[Mon Dec 22 14:55:44 2014] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Mon Dec 22 14:55:44 2014] [error] \tHTTPS: 1
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_ACCEPT_ENCODING: identity
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_HOST: localhost
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_USER_AGENT: rhnpush
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_AUTH_SESSION: 16960x228e6ed2f818a7d58bf000f1dd7a9c3daefd99e7a31d3e1d673719f583b22f64
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_FILE_MD5SUM: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_FORCE: 0
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_ARCH: all-deb
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_NAME: openclipart2-png
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_RELEASE: 1
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_VERSION: 2.0+dfsg
[Mon Dec 22 14:55:44 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGING: deb
[Mon Dec 22 14:55:44 2014] [error] \tHost: localhost
[Mon Dec 22 14:55:44 2014] [error] \tPATH_INFO: 
[Mon Dec 22 14:55:44 2014] [error] \tQUERY_STRING: 
[Mon Dec 22 14:55:44 2014] [error] \tREMOTE_ADDR: ::1
[Mon Dec 22 14:55:44 2014] [error] \tREMOTE_PORT: 48091
[Mon Dec 22 14:55:44 2014] [error] \tREQUEST_METHOD: POST
[Mon Dec 22 14:55:44 2014] [error] \tREQUEST_URI: /PACKAGE-PUSH
[Mon Dec 22 14:55:44 2014] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/package_push.py
[Mon Dec 22 14:55:44 2014] [error] \tSCRIPT_NAME: /PACKAGE-PUSH
[Mon Dec 22 14:55:44 2014] [error] \tSCRIPT_URI: https://localhost/PACKAGE-PUSH
[Mon Dec 22 14:55:44 2014] [error] \tSCRIPT_URL: /PACKAGE-PUSH
[Mon Dec 22 14:55:44 2014] [error] \tSERVER_ADDR: ::1
[Mon Dec 22 14:55:44 2014] [error] \tSERVER_ADMIN: root@localhost
[Mon Dec 22 14:55:44 2014] [error] \tSERVER_NAME: localhost
[Mon Dec 22 14:55:44 2014] [error] \tSERVER_PORT: 443
[Mon Dec 22 14:55:44 2014] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Mon Dec 22 14:55:44 2014] [error] \tSERVER_SIGNATURE: <address>Apache Server at localhost Port 443</address>
[Mon Dec 22 14:55:44 2014] [error] 
[Mon Dec 22 14:55:44 2014] [error] \tSERVER_SOFTWARE: Apache
[Mon Dec 22 14:55:44 2014] [error] \tUser-Agent: rhnpush
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-Auth-Session: 16960x228e6ed2f818a7d58bf000f1dd7a9c3daefd99e7a31d3e1d673719f583b22f64
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-File-Checksum: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-File-Checksum-Type: md5
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-File-MD5sum: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-Force: 0
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-Package-Arch: all-deb
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-Package-Name: openclipart2-png
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-Package-Release: 1
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-Package-Version: 2.0+dfsg
[Mon Dec 22 14:55:44 2014] [error] \tX-RHN-Upload-Packaging: deb
[Mon Dec 22 14:55:44 2014] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7f6b9bcf83f0>
[Mon Dec 22 14:55:44 2014] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f6b9bcf83f0>
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.application_group: kam1odapp99.connex.bclc.com|/package-push
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.callable_object: application
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.handler_script: 
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.input_chunked: 0
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.listener_host: 
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.listener_port: 443
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.process_group: 
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.request_handler: wsgi-script
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.script_reloading: 1
[Mon Dec 22 14:55:44 2014] [error] \tmod_wsgi.version: (3, 2)
[Mon Dec 22 14:55:44 2014] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7f6b9bd1fdf0>
[Mon Dec 22 14:55:44 2014] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f6b9bcf83f0>
[Mon Dec 22 14:55:44 2014] [error] \twsgi.input: <mod_wsgi.Input object at 0x7f6b9bd1fe30>
[Mon Dec 22 14:55:44 2014] [error] \twsgi.multiprocess: True
[Mon Dec 22 14:55:44 2014] [error] \twsgi.multithread: False
[Mon Dec 22 14:55:44 2014] [error] \twsgi.run_once: False
[Mon Dec 22 14:55:44 2014] [error] \twsgi.url_scheme: https
[Mon Dec 22 14:55:44 2014] [error] \twsgi.version: (1, 1)
[Mon Dec 22 14:55:44 2014] [error] 
[Mon Dec 22 14:55:44 2014] [error] Exception Handler Information
[Mon Dec 22 14:55:44 2014] [error] Traceback (most recent call last):
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/apacheUploadServer.py", line 100, in _wrapper
[Mon Dec 22 14:55:44 2014] [error]     ret = function(req)
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/share/rhn/upload_server/handlers/package_push/package_push.py", line 124, in handler
[Mon Dec 22 14:55:44 2014] [error]     relative_path=self.rel_package_path, org_id=self.org_id)
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnPackageUpload.py", line 143, in push_package
[Mon Dec 22 14:55:44 2014] [error]     importer.run()
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/importLib.py", line 650, in run
[Mon Dec 22 14:55:44 2014] [error]     self.submit()
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/packageImport.py", line 317, in submit
[Mon Dec 22 14:55:44 2014] [error]     self._import_signatures()
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/packageImport.py", line 509, in _import_signatures
[Mon Dec 22 14:55:44 2014] [error]     header = rhn_pkg.get_package_header(filename=full_path)
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhn_pkg.py", line 38, in get_package_header
[Mon Dec 22 14:55:44 2014] [error]     a_pkg.read_header()
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhn_deb.py", line 113, in read_header
[Mon Dec 22 14:55:44 2014] [error]     self._stream_copy(self.input_stream, self.header_data)
[Mon Dec 22 14:55:44 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhn_pkg.py", line 113, in _stream_copy
[Mon Dec 22 14:55:44 2014] [error]     dest.write(buf)
[Mon Dec 22 14:55:44 2014] [error] IOError: [Errno 28] No space left on device
[Mon Dec 22 14:55:44 2014] [error] 
[Mon Dec 22 14:55:45 2014] [error] Spacewalk 8365 2014/12/22 14:55:45 -07:00: ('Unhandled exception',)
[Mon Dec 22 14:55:55 2014] [error] Exception reported from kam1odapp99.connex.bclc.com
[Mon Dec 22 14:55:55 2014] [error] Time: Mon Dec 22 14:55:55 2014
[Mon Dec 22 14:55:55 2014] [error] Exception type <type 'exceptions.IOError'>
[Mon Dec 22 14:55:55 2014] [error] Exception while handling function server.apacheUploadServer._wrapper
[Mon Dec 22 14:55:55 2014] [error] Request object information:
[Mon Dec 22 14:55:55 2014] [error] URI: /PACKAGE-PUSH
[Mon Dec 22 14:55:55 2014] [error] Remote Host: localhost
[Mon Dec 22 14:55:55 2014] [error] Server Name: localhost:443
[Mon Dec 22 14:55:55 2014] [error] Headers passed in:
[Mon Dec 22 14:55:55 2014] [error] \tAccept-Encoding: identity
[Mon Dec 22 14:55:55 2014] [error] \tCONTENT_LENGTH: 477100858
[Mon Dec 22 14:55:55 2014] [error] \tCONTENT_TYPE: application/x-rpm
[Mon Dec 22 14:55:55 2014] [error] \tDOCUMENT_ROOT: /var/www/html
[Mon Dec 22 14:55:55 2014] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Mon Dec 22 14:55:55 2014] [error] \tHTTPS: 1
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_ACCEPT_ENCODING: identity
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_HOST: localhost
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_USER_AGENT: rhnpush
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_AUTH_SESSION: 16960x228e6ed2f818a7d58bf000f1dd7a9c3daefd99e7a31d3e1d673719f583b22f64
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_FILE_MD5SUM: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_FORCE: 0
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_ARCH: all-deb
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_NAME: openclipart2-png
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_RELEASE: 1
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_VERSION: 2.0+dfsg
[Mon Dec 22 14:55:55 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGING: deb
[Mon Dec 22 14:55:55 2014] [error] \tHost: localhost
[Mon Dec 22 14:55:55 2014] [error] \tPATH_INFO: 
[Mon Dec 22 14:55:55 2014] [error] \tQUERY_STRING: 
[Mon Dec 22 14:55:55 2014] [error] \tREMOTE_ADDR: ::1
[Mon Dec 22 14:55:55 2014] [error] \tREMOTE_PORT: 48109
[Mon Dec 22 14:55:55 2014] [error] \tREQUEST_METHOD: POST
[Mon Dec 22 14:55:55 2014] [error] \tREQUEST_URI: /PACKAGE-PUSH
[Mon Dec 22 14:55:55 2014] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/package_push.py
[Mon Dec 22 14:55:55 2014] [error] \tSCRIPT_NAME: /PACKAGE-PUSH
[Mon Dec 22 14:55:55 2014] [error] \tSCRIPT_URI: https://localhost/PACKAGE-PUSH
[Mon Dec 22 14:55:55 2014] [error] \tSCRIPT_URL: /PACKAGE-PUSH
[Mon Dec 22 14:55:55 2014] [error] \tSERVER_ADDR: ::1
[Mon Dec 22 14:55:55 2014] [error] \tSERVER_ADMIN: root@localhost
[Mon Dec 22 14:55:55 2014] [error] \tSERVER_NAME: localhost
[Mon Dec 22 14:55:55 2014] [error] \tSERVER_PORT: 443
[Mon Dec 22 14:55:55 2014] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Mon Dec 22 14:55:55 2014] [error] \tSERVER_SIGNATURE: <address>Apache Server at localhost Port 443</address>
[Mon Dec 22 14:55:55 2014] [error] 
[Mon Dec 22 14:55:55 2014] [error] \tSERVER_SOFTWARE: Apache
[Mon Dec 22 14:55:55 2014] [error] \tUser-Agent: rhnpush
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-Auth-Session: 16960x228e6ed2f818a7d58bf000f1dd7a9c3daefd99e7a31d3e1d673719f583b22f64
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-File-Checksum: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-File-Checksum-Type: md5
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-File-MD5sum: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-Force: 0
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-Package-Arch: all-deb
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-Package-Name: openclipart2-png
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-Package-Release: 1
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-Package-Version: 2.0+dfsg
[Mon Dec 22 14:55:55 2014] [error] \tX-RHN-Upload-Packaging: deb
[Mon Dec 22 14:55:55 2014] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7f6b9b93c3f0>
[Mon Dec 22 14:55:55 2014] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f6b9b93c3f0>
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.application_group: kam1odapp99.connex.bclc.com|/package-push
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.callable_object: application
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.handler_script: 
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.input_chunked: 0
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.listener_host: 
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.listener_port: 443
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.process_group: 
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.request_handler: wsgi-script
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.script_reloading: 1
[Mon Dec 22 14:55:55 2014] [error] \tmod_wsgi.version: (3, 2)
[Mon Dec 22 14:55:55 2014] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7f6b9b965db0>
[Mon Dec 22 14:55:55 2014] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f6b9b93c3f0>
[Mon Dec 22 14:55:55 2014] [error] \twsgi.input: <mod_wsgi.Input object at 0x7f6b9b965df0>
[Mon Dec 22 14:55:55 2014] [error] \twsgi.multiprocess: True
[Mon Dec 22 14:55:55 2014] [error] \twsgi.multithread: False
[Mon Dec 22 14:55:55 2014] [error] \twsgi.run_once: False
[Mon Dec 22 14:55:55 2014] [error] \twsgi.url_scheme: https
[Mon Dec 22 14:55:55 2014] [error] \twsgi.version: (1, 1)
[Mon Dec 22 14:55:55 2014] [error] 
[Mon Dec 22 14:55:55 2014] [error] Exception Handler Information
[Mon Dec 22 14:55:55 2014] [error] Traceback (most recent call last):
[Mon Dec 22 14:55:55 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/apacheUploadServer.py", line 100, in _wrapper
[Mon Dec 22 14:55:55 2014] [error]     ret = function(req)
[Mon Dec 22 14:55:55 2014] [error]   File "/usr/share/rhn/upload_server/handlers/package_push/package_push.py", line 114, in handler
[Mon Dec 22 14:55:55 2014] [error]     self.file_checksum_type, self.file_checksum)
[Mon Dec 22 14:55:55 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnPackageUpload.py", line 264, in save_uploaded_package
[Mon Dec 22 14:55:55 2014] [error]     a_pkg.read_header()
[Mon Dec 22 14:55:55 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhn_deb.py", line 113, in read_header
[Mon Dec 22 14:55:55 2014] [error]     self._stream_copy(self.input_stream, self.header_data)
[Mon Dec 22 14:55:55 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhn_pkg.py", line 113, in _stream_copy
[Mon Dec 22 14:55:55 2014] [error]     dest.write(buf)
[Mon Dec 22 14:55:55 2014] [error] IOError: [Errno 28] No space left on device
[Mon Dec 22 14:55:55 2014] [error] 
[Mon Dec 22 14:55:55 2014] [error] Spacewalk 8370 2014/12/22 14:55:55 -07:00: ('Unhandled exception',)
[Mon Dec 22 14:55:59 2014] [error] Spacewalk 8369 2014/12/22 14:55:59 -07:00: ("DATABASE CONNECTION TO 'spacewalk@//10.0.9.107/XE' LOST", "Exception information: (3113, 'ORA-03113: end-of-file on communication channel\\\\nProcess ID: 1285\\\\nSession ID: 39 Serial number: 587\\\\n', 'select 1 from dual')")
[Mon Dec 22 14:56:03 2014] [error] Exception reported from kam1odapp99.connex.bclc.com
[Mon Dec 22 14:56:03 2014] [error] Time: Mon Dec 22 14:56:03 2014
[Mon Dec 22 14:56:03 2014] [error] Exception type <type 'exceptions.IOError'>
[Mon Dec 22 14:56:03 2014] [error] Exception while handling function server.apacheUploadServer._wrapper
[Mon Dec 22 14:56:03 2014] [error] Request object information:
[Mon Dec 22 14:56:03 2014] [error] URI: /PACKAGE-PUSH
[Mon Dec 22 14:56:03 2014] [error] Remote Host: localhost
[Mon Dec 22 14:56:03 2014] [error] Server Name: localhost:443
[Mon Dec 22 14:56:03 2014] [error] Headers passed in:
[Mon Dec 22 14:56:03 2014] [error] \tAccept-Encoding: identity
[Mon Dec 22 14:56:03 2014] [error] \tCONTENT_LENGTH: 477100858
[Mon Dec 22 14:56:03 2014] [error] \tCONTENT_TYPE: application/x-rpm
[Mon Dec 22 14:56:03 2014] [error] \tDOCUMENT_ROOT: /var/www/html
[Mon Dec 22 14:56:03 2014] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Mon Dec 22 14:56:03 2014] [error] \tHTTPS: 1
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_ACCEPT_ENCODING: identity
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_HOST: localhost
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_USER_AGENT: rhnpush
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_AUTH_SESSION: 16960x228e6ed2f818a7d58bf000f1dd7a9c3daefd99e7a31d3e1d673719f583b22f64
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_FILE_MD5SUM: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_FORCE: 0
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_ARCH: all-deb
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_NAME: openclipart2-png
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_RELEASE: 1
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGE_VERSION: 2.0+dfsg
[Mon Dec 22 14:56:03 2014] [error] \tHTTP_X_RHN_UPLOAD_PACKAGING: deb
[Mon Dec 22 14:56:03 2014] [error] \tHost: localhost
[Mon Dec 22 14:56:03 2014] [error] \tPATH_INFO: 
[Mon Dec 22 14:56:03 2014] [error] \tQUERY_STRING: 
[Mon Dec 22 14:56:03 2014] [error] \tREMOTE_ADDR: ::1
[Mon Dec 22 14:56:03 2014] [error] \tREMOTE_PORT: 48116
[Mon Dec 22 14:56:03 2014] [error] \tREQUEST_METHOD: POST
[Mon Dec 22 14:56:03 2014] [error] \tREQUEST_URI: /PACKAGE-PUSH
[Mon Dec 22 14:56:03 2014] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/package_push.py
[Mon Dec 22 14:56:03 2014] [error] \tSCRIPT_NAME: /PACKAGE-PUSH
[Mon Dec 22 14:56:03 2014] [error] \tSCRIPT_URI: https://localhost/PACKAGE-PUSH
[Mon Dec 22 14:56:03 2014] [error] \tSCRIPT_URL: /PACKAGE-PUSH
[Mon Dec 22 14:56:03 2014] [error] \tSERVER_ADDR: ::1
[Mon Dec 22 14:56:03 2014] [error] \tSERVER_ADMIN: root@localhost
[Mon Dec 22 14:56:03 2014] [error] \tSERVER_NAME: localhost
[Mon Dec 22 14:56:03 2014] [error] \tSERVER_PORT: 443
[Mon Dec 22 14:56:03 2014] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Mon Dec 22 14:56:03 2014] [error] \tSERVER_SIGNATURE: <address>Apache Server at localhost Port 443</address>
[Mon Dec 22 14:56:03 2014] [error] 
[Mon Dec 22 14:56:03 2014] [error] \tSERVER_SOFTWARE: Apache
[Mon Dec 22 14:56:03 2014] [error] \tUser-Agent: rhnpush
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-Auth-Session: 16960x228e6ed2f818a7d58bf000f1dd7a9c3daefd99e7a31d3e1d673719f583b22f64
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-File-Checksum: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-File-Checksum-Type: md5
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-File-MD5sum: 46f814f023424ceedbf4b7653293ce17
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-Force: 0
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-Package-Arch: all-deb
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-Package-Name: openclipart2-png
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-Package-Release: 1
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-Package-Version: 2.0+dfsg
[Mon Dec 22 14:56:03 2014] [error] \tX-RHN-Upload-Packaging: deb
[Mon Dec 22 14:56:03 2014] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7f6b9a668af8>
[Mon Dec 22 14:56:03 2014] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f6b9a668af8>
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.application_group: kam1odapp99.connex.bclc.com|/package-push
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.callable_object: application
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.handler_script: 
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.input_chunked: 0
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.listener_host: 
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.listener_port: 443
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.process_group: 
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.request_handler: wsgi-script
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.script_reloading: 1
[Mon Dec 22 14:56:03 2014] [error] \tmod_wsgi.version: (3, 2)
[Mon Dec 22 14:56:03 2014] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7f6b9af57cf0>
[Mon Dec 22 14:56:03 2014] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f6b9a668af8>
[Mon Dec 22 14:56:03 2014] [error] \twsgi.input: <mod_wsgi.Input object at 0x7f6b9a974fb0>
[Mon Dec 22 14:56:03 2014] [error] \twsgi.multiprocess: True
[Mon Dec 22 14:56:03 2014] [error] \twsgi.multithread: False
[Mon Dec 22 14:56:03 2014] [error] \twsgi.run_once: False
[Mon Dec 22 14:56:03 2014] [error] \twsgi.url_scheme: https
[Mon Dec 22 14:56:03 2014] [error] \twsgi.version: (1, 1)
[Mon Dec 22 14:56:03 2014] [error] 
[Mon Dec 22 14:56:03 2014] [error] Exception Handler Information
[Mon Dec 22 14:56:03 2014] [error] Traceback (most recent call last):
[Mon Dec 22 14:56:03 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/apacheUploadServer.py", line 100, in _wrapper
[Mon Dec 22 14:56:03 2014] [error]     ret = function(req)
[Mon Dec 22 14:56:03 2014] [error]   File "/usr/share/rhn/upload_server/handlers/package_push/package_push.py", line 114, in handler
[Mon Dec 22 14:56:03 2014] [error]     self.file_checksum_type, self.file_checksum)
[Mon Dec 22 14:56:03 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnPackageUpload.py", line 264, in save_uploaded_package
[Mon Dec 22 14:56:03 2014] [error]     a_pkg.read_header()
[Mon Dec 22 14:56:03 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhn_deb.py", line 113, in read_header
[Mon Dec 22 14:56:03 2014] [error]     self._stream_copy(self.input_stream, self.header_data)
[Mon Dec 22 14:56:03 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhn_pkg.py", line 113, in _stream_copy
[Mon Dec 22 14:56:03 2014] [error]     dest.write(buf)
[Mon Dec 22 14:56:03 2014] [error] IOError: [Errno 28] No space left on device
[Mon Dec 22 14:56:03 2014] [error] 
[Mon Dec 22 14:56:03 2014] [error] Spacewalk 8369 2014/12/22 14:56:03 -07:00: ('Unhandled exception',)
========================================
Comment 1 Nick Tailor 2015-01-07 19:50:44 UTC 
I was able to resolve the issue by increasing the /tmp to 5gigs. Seems like that script still uses /tmp as a temporary buffer during stream writing.
Comment 2 Eric Herget 2017-09-28 18:11:15 UTC 
This BZ closed some time during 2.5, 2.6 or 2.7.  Adding to 2.7 tracking bug.