My satellite6.1 setup is running on RHEL67 with the latest snap15 2015-07-31.1

I don't see any reports on sat61 running on rhel67 and with rhel71 I see this randomly with some clients.

With SNAP13-c2 it was working smoothly but that was on rhel66.

can you double check the reports are sent to satellite server? (e.g. logs)
also double check the host in question is in the right org / location context?

thanks

I see the below errors in the log files.


[root@xyz settings.d]# tail -f /var/log/foreman-proxy/openscap-send.log
# Logfile created on Tue Aug 04 17:00:01 +0530 2015 by logger.rb/1.2.6
E, [2015-08-04T17:50:26.942980 #21438] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T17:54:02.618644 #21635] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T17:59:42.298199 #22631] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T18:00:11.368085 #22683] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T18:30:05.910917 #26321] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T18:53:20.014825 #27813] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"

can you please find the equivalent error on production.log? you might need to bump log to debug.

thanks.

This is what is seen in /var/log/foreman/production.log file.

2015-08-04 22:00:02 [I] Processing by Api::V2::Compliance::ArfReportsController#create as */*
2015-08-04 22:00:02 [I]   Parameters: {"apiv"=>"v2", "cname"=>"e47fdd36-eabd-4356-bb12-964fb202ddc5", "policy_id"=>"2", "date"=>"2015-08-04"}
2015-08-04 22:00:03 [E] Could not read from bz_stream: BZ2_bzDecompress returns -4 [bz2.c:157]
Could not create Result DataStream session: File is not Result DataStream. [ds_rds_session.c:54]
 (OpenSCAP::OpenSCAPError)
/opt/rh/ruby193/root/usr/share/gems/gems/openscap-0.4.2/lib/openscap/openscap.rb:34:in `raise!'
/opt/rh/ruby193/root/usr/share/gems/gems/openscap-0.4.2/lib/openscap/ds/arf.rb:30:in `initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/scaptimony-0.3.0.1/app/models/scaptimony/arf_report_raw.rb:48:in `new'
/opt/rh/ruby193/root/usr/share/gems/gems/scaptimony-0.3.0.1/app/models/scaptimony/arf_report_raw.rb:48:in `build_arf'
/opt/rh/ruby193/root/usr/share/gems/gems/scaptimony-0.3.0.1/app/models/scaptimony/arf_report_raw.rb:26:in `save_dependent_entities'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__3419742543643310877__create__4504087908705251513__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_create_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:268:in `create'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:347:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:414:in `_run__3419742543643310877__save__4504087908705251513__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:104:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:56:in `save!'

Created attachment 1059192 [details]
foreman debug

Attaching the foreman-debug which has the entire logs.

I have tested this on another setup (with sat6.1.1 on RHEL67) and it seems to work fine, the reports are getting uploaded. 

Not sure what is the root cause for this issue, but it seems to be quiet random.

I will remove the blocker flag and move severity settings to medium. But would like to have this bug in open state.

/var/log/foreman-proxy/openscap-send.log with debug logging:

D, [2015-08-05T12:27:32.749639 #2455] DEBUG -- : Uploading 9fbd0d32601ac60460f09fad94672d74919c1586baf1a7bb052b3228b5f77f5f to /api/v2/compliance/arf_reports/e47fdd36-eabd-4356-bb12-964fb202ddc5/2/2015-08-04
D, [2015-08-05T12:27:33.775909 #2455] DEBUG -- : {
  "error": {"message":"Could not read from bz_stream: BZ2_bzDecompress returns -4 [bz2.c:157]\nCould not create Result DataStream session: File is not Result DataStream. [ds_rds_session.c:54]\n"}
}

D, [2015-08-05T12:27:33.776099 #2455] DEBUG -- : /usr/lib/ruby/1.8/net/http.rb:2105:in `error!'
        /usr/lib/ruby/1.8/net/http.rb:2114:in `value'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:182:in `forward_arf_file'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:167:in `forward_date_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:163:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:163:in `forward_date_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:155:in `forward_policy_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:152:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:152:in `forward_policy_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:145:in `forward_cname_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:142:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:142:in `forward_cname_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:135:in `do'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:132:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:132:in `do'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:79:in `send_spool_to_foreman'
        /usr/bin/smart-proxy-openscap-send:38
E, [2015-08-05T12:27:33.776167 #2455] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"

The inability to read bz_stream seems to be the reason why the reports are not showing up in UI. As for selectivity, it is probably related to this:
http://projects.theforeman.org/issues/9991, there is already PR for this issue.

I will keep on searching for the cause of bz_stream corruption.

Update: the corrupted bz_stream is definitely a cause of this, how it happened is unclear since I failed to reproduce even on machine that initially reported this error.

Workaround steps:
1) on proxy, delete /var/spool/foreman-proxy/openscap/arf/#{id}/#{policy}/#{date}
that gives the error (for error above it is /var/spool/foreman-proxy/openscap/arf/e47fdd36-eabd-4356-bb12-964fb202ddc5/2/2015-08-04)

2) re-run smart-proxy-openscap-send, there should be no errors. You may also want to re-run foreman_scap_client before this step.

True this is very random issue, hence was not seen earlier.

can you collect the report file and attach it ?

Simon, could you please have a look? thanks!

Looks like I don't have the report to upload as I had used the workaround mentioned above and had deleted.

Also as this issue is random, currently it's hard for me to reproduce this, will upload the report and the oscap-ds-stream.xml file, if I will be facing this issue again.

I'll try to sneak this into our next sprint, but no promises.

Created attachment 1066946 [details]
oscap_content_rhel67

the oscap content for rhel67

I just experienced the same behaviour on my lab setup:

Satellite 6.1.1 on RHEL 7.1

- reports where visible in WEBUI when 'smart-proxy-openscap-send' was run
- reports stopped being visible, same errors in openscap-send.log
- applying the steps to fix from above, reports are visible again

Moving 6.2 bugs out to sat-backlog.

We can close this bug, oscap reports where visible for sat6.1 and sat6.2 GA builds.

This is no longer a bug with Sat6.2 atleast.

Also this issue should not be faced at all when using Sat6.2, but there are probably chances that we may see this issue with Sat6.1 rarely.