Bug 1246392

Summary: foreman-proxy Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
Product: Red Hat Satellite 6 Reporter: Sebastian Hetze <shetze>
Component: SCAP PluginAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Kedar Bidarkar <kbidarka>
Severity: high Docs Contact:
Priority: high    
Version: 6.1.0CC: bbuckingham, cwelton, dpathak, echen, kbidarka, mreinke, mtessun, will_darton
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 09:16:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Example of openscap report creating this error
none
Example of openscap report from same client working correctly none

Description Sebastian Hetze 2015-07-24 08:17:55 UTC
Created attachment 1055647 [details]
Example of openscap report creating this error

Description of problem:

Ocassionally openscap reports are not digested properly by foreman-proxy.
Reports are uploaded correctly to /var/spool/foreman-proxy/openscap/arf but fail to be transferred to the foreman server.
This error occurs randomly in series of correctly handled reports from the same client. Once one bogus report occurs, foreman-proxy leaves all subsequent reports in the queue.

foreman-proxy debug output:

D, [2015-07-24T08:00:01.980290 #3822] DEBUG -- : Uploading 5c83c000e4119d92538ba04a8ba9c406c72916cfc5ce0d40aff5bc77eae0bdf5 to /api/v2/compliance/arf_reports/7be7b239-27bb-4b85-aa3d-7d58667ea5d2/1/2015-07-23
D, [2015-07-24T08:00:04.071839 #3822] DEBUG -- : {
  "error": {"message":"Could not read from bz_stream: BZ2_bzDecompress returns -4 [bz2.c:157]\nCould not create Result DataStream session: File is not Result DataStream. [ds_rds_session.c:54]\n"}
}
D, [2015-07-24T08:00:04.072150 #3822] DEBUG -- : /usr/share/ruby/net/http/response.rb:119:in `error!'
        /usr/share/ruby/net/http/response.rb:128:in `value'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:182:in `forward_arf_file'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:167:in `block in forward_date_dir'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:163:in `foreach'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:163:in `forward_date_dir'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:155:in `block in forward_policy_dir'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:152:in `foreach'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:152:in `forward_policy_dir'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:145:in `block in forward_cname_dir'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:142:in `foreach'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:142:in `forward_cname_dir'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:135:in `block in do'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:132:in `foreach'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:132:in `do'
        /usr/share/gems/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:79:in `send_spool_to_foreman'
        /usr/bin/smart-proxy-openscap-send:38:in `<main>'
E, [2015-07-24T08:00:04.072190 #3822] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"

Version-Release number of selected component (if applicable):
foreman-proxy-1.7.2.5-1
ruby193-rubygem-foreman_openscap-0.3.2.10-1
puppet-foreman_scap_client-0.3.3-9

How reproducible:
Unfortunately this happens randomly.

Attached you find a report that generates the error. Copy it to /var/spool/foreman-proxy/openscap/arf/7be7b239-27bb-4b85-aa3d-7d58667ea5d2/1/2015-07-23/5c83c000e4119d92538ba04a8ba9c406c72916cfc5ce0d40aff5bc77eae0bdf5 and you should get the described behaviour. 

Actual results:
Listing of new reports in Satellite Hosts->Compliance->Reports stalls until you manually delete the bogus report from /var/spool/foreman-proxy/openscap/arf

Expected results:
- the actual reports do not look different and I expect all of them to work
- if somehow real bogus data is delivered to foreman-proxy, this one data set must be sorted out with meaningful message and processing of subsequent reports must proceed.

Additional info:

Comment 1 Sebastian Hetze 2015-07-24 08:20:10 UTC
Created attachment 1055648 [details]
Example of openscap report from same client working correctly

Comment 7 Kedar Bidarkar 2016-04-20 14:59:48 UTC
This issue being random is hard to reproduce reliably or to test.

But having faced this issue with sat6.1 few times myself  and me testing it with Sat6.2, I have not faced this issue with any sat6.2 builds.

Please re-open this bug if faced with sat6.2 beta or GA.

VERIFIED with sat62-snap8.1

Comment 8 Kedar Bidarkar 2016-04-20 15:08:44 UTC
I had raised a bug here for this issue, https://bugzilla.redhat.com/show_bug.cgi?id=1250065

Comment 10 errata-xmlrpc 2016-07-27 09:16:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501