Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1934210 - Bad HTTP method requests filling up /var/log/messages with stack traces
Summary: Bad HTTP method requests filling up /var/log/messages with stack traces
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Packaging
Version: 6.7.0
Hardware: All
OS: Linux
unspecified
high
Target Milestone: 6.12.0
Assignee: satellite6-bugs
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-02 17:24 UTC by Julio Entrena Perez
Modified: 2022-12-09 18:28 UTC (History)
6 users (show)

Fixed In Version: tfm-rubygem-rails-6.0.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-16 13:32:29 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker SAT-12933 0 None None None 2022-09-14 11:06:42 UTC
Red Hat Issue Tracker SAT-8660 0 None None None 2022-09-14 10:49:51 UTC
Red Hat Knowledge Base (Solution) 5847851 0 None None None 2021-03-02 17:41:29 UTC
Red Hat Product Errata RHSA-2022:8506 0 None None None 2022-11-16 13:32:36 UTC

Description Julio Entrena Perez 2021-03-02 17:24:57 UTC 
Description of problem:
When httpd in Satellite receives a request with an invalid HTTP method, it logs the condition to /var/log/httpd/error_log with the following text:

[ 2021-02-14 10:09:15.7381 28224/7f6e0e815700 Pool2/Implementation.cpp:1274 ]: [App 28283 stderr] [ 2021-02-14 10:09:15.7380 6537/0x000000000c291948(Worker 1) utils.rb:74 ]: *** Exception ActionController::UnknownHttpMethod in Rack application object (ABCD, accepted HTTP methods are OPTIONS, GET, HEAD, POST, PUT, DELETE, TRACE, CONNECT, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, VERSION-CONTROL, REPORT, CHECKOUT, CHECKIN, UNCHECKOUT, MKWORKSPACE, UPDATE, LABEL, MERGE, BASELINE-CONTROL, MKACTIVITY, ORDERPATCH, ACL, SEARCH, MKCALENDAR, and PATCH) (process 6537, thread 0x000000000c291948(Worker 1)):

This is reasonable, but after that there are 33 more lines of stack trace:

[ 2021-03-02 17:21:13.0880 40626/7f438426d700 Pool2/Implementation.cpp:1274 ]: [App 40766 stderr] 	from /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/request.rb:431:in `check_method'
[ 2021-03-02 17:21:13.0880 40626/7f438426d700 Pool2/Implementation.cpp:1274 ]: [App 40766 stderr] 	from /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/request.rb:143:in `request_method'
[...]

A Qualys security scanner is regularly hitting the Satellite with bad HTTP methods and this is filling up the filesystem where /var/log/httpd/ resides.

Version-Release number of selected component (if applicable):
satellite-6.7.1-1.el7sat.noarch

How reproducible:
Always

Steps to Reproduce:
1. Issue a bad HTTP method to httpd in Satellite:
$ curl -X ABCD http://<fqdn_of_satellite>

Actual results:
The above line is logged to /var/log/httpd/error.log followed by 30+ lines of stack trace.

Expected results:
Only the above line is logged to /var/log/httpd/error.log .

Additional info:
This condition is being triggered very frequently by a Qualys security scanner sending all sorts of bad methods to the Satellite:

$ grep "accepted HTTP methods are" error_log | awk '{print $24}' | sed -e 's/^(//' -e 's/,$//' | sort | uniq -c
     57 ABCD
     18 BADMETHOD
     60 BADMTHD
     18 BDMTHD
     18 CFYZ
     18 DEBUG
    143 get
     18 INDEX
     38 QUALYS
     37 rndmmtd
     18 RNDMMTD
     72 TRACK

The resulting stack traces are filling up the filesystem where /var/log resides making the Satellite unavailable.
The same behaviour is observable on Satellite 6.8.
Comment 1 Julio Entrena Perez 2021-03-02 17:27:40 UTC 
Apologies, it's not /var/log/foreman/production.log but /var/log/httpd/error_log
Comment 2 Ewoud Kohl van Wijngaarden 2021-03-03 14:47:25 UTC 
It looks like this is https://github.com/rails/rails/issues/38998. It has https://github.com/rails/rails/pull/40246 as a resolution and https://github.com/rails/rails/commit/5ccc67605ec462a8e164635c75be84a3af0234f5 as a cherry pick to 6.0-stable but no Rails release includes it yet. It's aligned to Rails 6.0.4 according to the current milestones.
Comment 7 Stephen Wadeley 2021-03-08 11:32:40 UTC 
(In reply to Julio Entrena Perez from comment #1)
> Apologies, it's not /var/log/foreman/production.log but
> /var/log/httpd/error_log

then we should edit the summary
Comment 15 Ewoud Kohl van Wijngaarden 2021-07-22 19:57:46 UTC 
(In reply to Ewoud Kohl van Wijngaarden from comment #2)
> It looks like this is https://github.com/rails/rails/issues/38998. It has
> https://github.com/rails/rails/pull/40246 as a resolution and
> https://github.com/rails/rails/commit/
> 5ccc67605ec462a8e164635c75be84a3af0234f5 as a cherry pick to 6.0-stable but
> no Rails release includes it yet. It's aligned to Rails 6.0.4 according to
> the current milestones.

Rails 6.0.4 is out since June 15th: https://rubygems.org/gems/rails/versions/6.0.4. I think this should be solved by a packaging update.
Comment 27 Lukas Pramuk 2022-09-14 10:43:03 UTC 
VERIFIED.

@Satellite 6.12.0 Snap10
rubygem-rails-6.0.4.7-1.el8sat.noarch

by the manual reproducer described in comment#0:

1) Issue a bad HTTP method to Satellite while monitoring logs:

# curl -X ABCD http://$(hostname)

# journalctl -efu foreman
Sep 14 06:33:04 dhcp-2-87.vms.sat.rdu2.redhat.com foreman[2000]: 2022-09-14 06:33:04 -0400 Rack app ("ABCD /" - (10.1.2.87)): #<ActionController::UnknownHttpMethod: ABCD, accepted HTTP methods are OPTIONS, GET, HEAD, POST, PUT, DELETE, TRACE, CONNECT, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, VERSION-CONTROL, REPORT, CHECKOUT, CHECKIN, UNCHECKOUT, MKWORKSPACE, UPDATE, LABEL, MERGE, BASELINE-CONTROL, MKACTIVITY, ORDERPATCH, ACL, SEARCH, MKCALENDAR, and PATCH>

>>> the error in the log is no longer followed by stacktrace
Comment 31 errata-xmlrpc 2022-11-16 13:32:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506
Note You need to log in before you can comment on or make changes to this bug.