Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
If proxy sending SCAP reports fails to send a report, it stores it into /var/spool/foreman-proxy/openscap/arf directory and re-tries in next run (invoked via */30 cronjob by default).
But there are multiple scenarios leaving some reports orphaned in the spool forever, e.g. if we:
- delete a host (any sending returns 404)
- delete a policy (any sending returns 404)
- restart proxy just after it successfully sent the report but before it deleted it from the spool - any subsequent sending fails with ISE 500 "ActiveRecord::RecordInvalid: Validation failed: Reported at has already been taken" on app/models/foreman_openscap/arf_report.rb:109:in `create_arf'
- there can be other scenarios unknown ATM
So there are scenarios leaving stalled/orphaned ARF reports in the spool - forever. Each and every execution of /etc/cron.d/rubygem-smart_proxy_openscap fails in sending them, each and every 30 minutes (by default).
We should have a cleanup policy (ideally configurable in foreman-proxy settings) that if a report is older than X days, it is not (again) attempted to be sent but it is deleted - ideally with some (warning?) log to proxy log.
Version-Release number of selected component (if applicable):
Sat 6.4
rubygem-smart_proxy_openscap-0.6.11-1.el7sat.noarch
How reproducible:
100%
Steps to Reproduce:
1. attach few OpenSCAP policies to few hosts
2. remove some Host (but keep the VM running for a while)
3. remove some policy
4. optionally, try to restart foreman-proxy after sending a report but before deleting it from spool (or mimic this by copying some report from spool, letting it processed successfully and then copying it back to the spool)
5. wait several half-hours and see what arf reports are (re)sent to Satellite:
tail -f /var/log/httpd/foreman-ssl_access_ssl.log | grep "POST /api/v2/compliance/arf_reports/"
Actual results:
5. shows the same reports are sent again and again, with timestamp (unix TS since The Epoch) at the end of URI stalled in past (even years in past, if you will be enough patient :) ).
The requests will fail with 404 or 500 error response code every time, forever.
Expected results:
5. shows too old reports are not further attempted to be sent, after some number of (failing) attempts.
Additional info:
verified
@satellite 6.5.0 snap 23
@tfm-rubygem-foreman_openscap-0.11.5-1.el7sat.noarch
@rubygem-smart_proxy_openscap-0.7.1-1.el7sat.noarch
steps:
1. attached few OpenSCAP policies to few hosts
2. have some reports in spool
3. remove some Hosts
4. remove some policies
observation:
- If we delete host, policy from satellite then reports in spool
also gets deleted.
- If reports from spool successfully gets uploaded to satellite, then it's
deleted from spool.
- If policy is not found on satellite during uploading reports from spool
then error "Policy with id 4 not found." is shown in production.log
- If host is not found on satellite during uploading reports from spool
then error "Could not find host identified by: 2cdf25d0-eafd-44e8-9e86-74918001f5e9" is shown in production.log
- after deleting scap policy or deleting host itself from satellite, host still sends reports satellite/capsule.
filed separate bugzilla for this issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1699260
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2019:1222