Bug 1250065 - oscap reports are not seen in the webui
oscap reports are not seen in the webui
Status: CLOSED WORKSFORME
Product: Red Hat Satellite 6
Classification: Red Hat
Component: SCAP Plugin (Show other bugs)
6.1.0
Unspecified Unspecified
medium Severity medium (vote)
: Unspecified
: --
Assigned To: Ohad Levy
Kedar Bidarkar
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-04 08:37 EDT by Kedar Bidarkar
Modified: 2016-11-14 03:09 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-14 03:06:25 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
foreman debug (674.53 KB, application/x-xz)
2015-08-04 13:28 EDT, Kedar Bidarkar
no flags Details
oscap_content_rhel67 (358.25 KB, application/x-gzip)
2015-08-25 12:33 EDT, Kedar Bidarkar
no flags Details

  None (edit)
Description Kedar Bidarkar 2015-08-04 08:37:33 EDT
Description of problem:

Currently we are unable to see oscap reports in the webui.

Tested with SNAP15-31.1 on RHEL67

Version-Release number of selected component (if applicable):

Tested with SNAP15-31.1 on RHEL67

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
reports are not visible in WEBUI when 'smart-proxy-openscap-send' is run on the capsule.

Expected results:
Reports should be visible.

Additional info:
Comment 1 Kedar Bidarkar 2015-08-04 08:40:20 EDT
My satellite6.1 setup is running on RHEL67 with the latest snap15 2015-07-31.1

I don't see any reports on sat61 running on rhel67 and with rhel71 I see this randomly with some clients.

With SNAP13-c2 it was working smoothly but that was on rhel66.
Comment 2 Ohad Levy 2015-08-04 08:40:55 EDT
can you double check the reports are sent to satellite server? (e.g. logs)
also double check the host in question is in the right org / location context?

thanks
Comment 4 Kedar Bidarkar 2015-08-04 09:31:49 EDT
I see the below errors in the log files.


[root@xyz settings.d]# tail -f /var/log/foreman-proxy/openscap-send.log
# Logfile created on Tue Aug 04 17:00:01 +0530 2015 by logger.rb/1.2.6
E, [2015-08-04T17:50:26.942980 #21438] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T17:54:02.618644 #21635] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T17:59:42.298199 #22631] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T18:00:11.368085 #22683] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T18:30:05.910917 #26321] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
E, [2015-08-04T18:53:20.014825 #27813] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"
Comment 5 Ohad Levy 2015-08-04 09:33:34 EDT
can you please find the equivalent error on production.log? you might need to bump log to debug.

thanks.
Comment 6 Kedar Bidarkar 2015-08-04 13:13:10 EDT
This is what is seen in /var/log/foreman/production.log file.

2015-08-04 22:00:02 [I] Processing by Api::V2::Compliance::ArfReportsController#create as */*
2015-08-04 22:00:02 [I]   Parameters: {"apiv"=>"v2", "cname"=>"e47fdd36-eabd-4356-bb12-964fb202ddc5", "policy_id"=>"2", "date"=>"2015-08-04"}
2015-08-04 22:00:03 [E] Could not read from bz_stream: BZ2_bzDecompress returns -4 [bz2.c:157]
Could not create Result DataStream session: File is not Result DataStream. [ds_rds_session.c:54]
 (OpenSCAP::OpenSCAPError)
/opt/rh/ruby193/root/usr/share/gems/gems/openscap-0.4.2/lib/openscap/openscap.rb:34:in `raise!'
/opt/rh/ruby193/root/usr/share/gems/gems/openscap-0.4.2/lib/openscap/ds/arf.rb:30:in `initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/scaptimony-0.3.0.1/app/models/scaptimony/arf_report_raw.rb:48:in `new'
/opt/rh/ruby193/root/usr/share/gems/gems/scaptimony-0.3.0.1/app/models/scaptimony/arf_report_raw.rb:48:in `build_arf'
/opt/rh/ruby193/root/usr/share/gems/gems/scaptimony-0.3.0.1/app/models/scaptimony/arf_report_raw.rb:26:in `save_dependent_entities'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__3419742543643310877__create__4504087908705251513__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_create_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:268:in `create'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:347:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:414:in `_run__3419742543643310877__save__4504087908705251513__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:104:in `save!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:56:in `save!'
Comment 7 Kedar Bidarkar 2015-08-04 13:28:03 EDT
Created attachment 1059192 [details]
foreman debug

Attaching the foreman-debug which has the entire logs.
Comment 8 Kedar Bidarkar 2015-08-04 13:33:38 EDT
I have tested this on another setup (with sat6.1.1 on RHEL67) and it seems to work fine, the reports are getting uploaded. 

Not sure what is the root cause for this issue, but it seems to be quiet random.

I will remove the blocker flag and move severity settings to medium. But would like to have this bug in open state.
Comment 9 Ondřej Pražák 2015-08-05 03:36:11 EDT
/var/log/foreman-proxy/openscap-send.log with debug logging:

D, [2015-08-05T12:27:32.749639 #2455] DEBUG -- : Uploading 9fbd0d32601ac60460f09fad94672d74919c1586baf1a7bb052b3228b5f77f5f to /api/v2/compliance/arf_reports/e47fdd36-eabd-4356-bb12-964fb202ddc5/2/2015-08-04
D, [2015-08-05T12:27:33.775909 #2455] DEBUG -- : {
  "error": {"message":"Could not read from bz_stream: BZ2_bzDecompress returns -4 [bz2.c:157]\nCould not create Result DataStream session: File is not Result DataStream. [ds_rds_session.c:54]\n"}
}

D, [2015-08-05T12:27:33.776099 #2455] DEBUG -- : /usr/lib/ruby/1.8/net/http.rb:2105:in `error!'
        /usr/lib/ruby/1.8/net/http.rb:2114:in `value'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:182:in `forward_arf_file'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:167:in `forward_date_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:163:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:163:in `forward_date_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:155:in `forward_policy_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:152:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:152:in `forward_policy_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:145:in `forward_cname_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:142:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:142:in `forward_cname_dir'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:135:in `do'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:132:in `foreach'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:132:in `do'
        /usr/lib/ruby/gems/1.8/gems/smart_proxy_openscap-0.3.0.9/lib/smart_proxy_openscap/openscap_lib.rb:79:in `send_spool_to_foreman'
        /usr/bin/smart-proxy-openscap-send:38
E, [2015-08-05T12:27:33.776167 #2455] ERROR -- : Failed to send SCAP results to the Foreman server: 500 "Internal Server Error"

The inability to read bz_stream seems to be the reason why the reports are not showing up in UI. As for selectivity, it is probably related to this:
http://projects.theforeman.org/issues/9991, there is already PR for this issue.

I will keep on searching for the cause of bz_stream corruption.
Comment 10 Ondřej Pražák 2015-08-05 06:38:30 EDT
Update: the corrupted bz_stream is definitely a cause of this, how it happened is unclear since I failed to reproduce even on machine that initially reported this error.

Workaround steps:
1) on proxy, delete /var/spool/foreman-proxy/openscap/arf/#{id}/#{policy}/#{date}
that gives the error (for error above it is /var/spool/foreman-proxy/openscap/arf/e47fdd36-eabd-4356-bb12-964fb202ddc5/2/2015-08-04)

2) re-run smart-proxy-openscap-send, there should be no errors. You may also want to re-run foreman_scap_client before this step.
Comment 11 Kedar Bidarkar 2015-08-05 09:01:21 EDT
True this is very random issue, hence was not seen earlier.
Comment 12 Ohad Levy 2015-08-06 02:40:39 EDT
can you collect the report file and attach it ?
Comment 13 Ohad Levy 2015-08-06 02:42:22 EDT
Simon, could you please have a look? thanks!
Comment 14 Kedar Bidarkar 2015-08-10 07:09:20 EDT
Looks like I don't have the report to upload as I had used the workaround mentioned above and had deleted.

Also as this issue is random, currently it's hard for me to reproduce this, will upload the report and the oscap-ds-stream.xml file, if I will be facing this issue again.
Comment 15 Šimon Lukašík 2015-08-25 08:03:11 EDT
I'll try to sneak this into our next sprint, but no promises.
Comment 17 Kedar Bidarkar 2015-08-25 12:33:56 EDT
Created attachment 1066946 [details]
oscap_content_rhel67

the oscap content for rhel67
Comment 18 Götz Rieger 2015-09-18 05:31:35 EDT
I just experienced the same behaviour on my lab setup:

Satellite 6.1.1 on RHEL 7.1

- reports where visible in WEBUI when 'smart-proxy-openscap-send' was run
- reports stopped being visible, same errors in openscap-send.log
- applying the steps to fix from above, reports are visible again
Comment 22 Bryan Kearney 2016-08-04 16:19:24 EDT
Moving 6.2 bugs out to sat-backlog.
Comment 24 Kedar Bidarkar 2016-11-14 03:06:25 EST
We can close this bug, oscap reports where visible for sat6.1 and sat6.2 GA builds.

This is no longer a bug with Sat6.2 atleast.

Also this issue should not be faced at all when using Sat6.2, but there are probably chances that we may see this issue with Sat6.1 rarely.

Note You need to log in before you can comment on or make changes to this bug.