Bug 1298571 - [RFE] Log X-Forwarded-For information
[RFE] Log X-Forwarded-For information
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
x86_64 Linux
unspecified Severity high
: rc
: 2.2
Assigned To: Matt Benjamin (redhat)
Ramakrishnan Periyasamy
Bara Ancincova
Depends On:
Blocks: 1412948
  Show dependency treegraph
Reported: 2016-01-14 08:00 EST by Benjamin Schmaus
Modified: 2017-07-30 11:47 EDT (History)
12 users (show)

See Also:
Fixed In Version: RHEL: ceph-10.2.5-7.el7cp Ubuntu: ceph_10.2.5-3redhat1xenial
Doc Type: Enhancement
Doc Text:
.The Ceph Object Gateway now supports custom HTTP header logging Sites that use the Civetweb HTTP web server previously lacked the ability to log custom HTTP headers, as they could when using the Apache web server and the FastCGI protocol. With this update, the Ceph Object Gateway supports custom HTTP header logging. To log custom HTTP headers, enable the operations log socket on the Ceph Object Gateway instance and list the HTTP headers. Add the following parameters to the Ceph configuration file: ---- rgw enable ops log = true rgw ops log socket path = <path> rgw log http headers = "<headers>" ---- Replace `<path>` with the path to the operations log socket and `<headers` with a comma-separated list of custom HTTP headers, for example: ---- rgw enable ops log = true rgw ops log socket path = /tmp/opslog rgw log http headers = "http_x_forwarded_for, http_expect, http_content_md5" ---- The operations log stream then lists the headers as a JSON-formatted key-value list with the "http_x_headers" key.
Story Points: ---
Clone Of:
Last Closed: 2017-03-14 11:43:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Benjamin Schmaus 2016-01-14 08:00:50 EST
Description of problem:

RGW (civetweb) does not allow addition of cache control headers, nor does it allow logging of custom headers. A few headers that I can think of are:

X-Forwarded-For (must be able to log this, and other custom headers in the RGW logs)
last-modified (or X-Copied-From-Last-Modified)
etags (including ability to disable)

What are the recommended practices for the requirement above?

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:
Comment 7 Neil Levine 2016-02-10 19:58:19 EST
Upon further conversation with the customer, the use-case is where RGW act as a origin for a caching CDN and the cache needs to know when to re-contact the origin to see if a newer version of an object exists. The customer wasn't aware that you can arbitrarily set headers on the objects via the application which will hint this to the cache software so there is no need for RGW to set a default header.

However, the customer did request that RGW log the X-Forwarded-For header as this is necessary for debugging and analytics where you have multiple cache front-ends all talking to multiple RGW instances. Changing the RFE subject accordingly.
Comment 16 chrone 2016-11-18 21:35:03 EST
I could not find any documentation how to log X-FORWARDED-FOR or X-REAL-IP addresses in the default Ceph RGW civetweb log without enabling "debug ms = 1" and "debug rgw = 20" mode.

2016-11-18 18:18:14.137551 7f5e93b87700  1 civetweb: 0x7f5eb00018b0: - - [18/Nov/2016:18:18:14 -0800] "GET /auth/1.0 HTTP/1.1" 204 0 - -

Will this feature enable Ceph RGW civetweb logging X-FORWARDED-FOR and/or X-REAL-IP addresses by default? This will be a great feature to have since enabling debug mode will filling up the disk space rapidly.
Comment 17 Matt Benjamin (redhat) 2016-12-15 11:52:57 EST
The capability log any HTTP header (and also other server variables) selectively to the RGW ops log has merged to master here:


Backport to Jewel in progress:


This is expected to reach RHCS 2.2, as planned.
Comment 21 shilpa 2017-01-13 02:27:12 EST
Hi Matt,

Could you share the steps to reproduce the BZ?

My understanding is that you would have to set the config options in ceph.conf as in https://github.com/ceph/ceph/pull/7639 and look for the X-Forwarded-For header in civetweb logs?
Comment 23 Matt Benjamin (redhat) 2017-01-17 07:24:17 EST
(In reply to shilpa from comment #21)
> Hi Matt,
> Could you share the steps to reproduce the BZ?
> My understanding is that you would have to set the config options in
> ceph.conf as in https://github.com/ceph/ceph/pull/7639 and look for the
> X-Forwarded-For header in civetweb logs?

Hi Shilpa,

No.  You need to:

1. enable usage logging
2. set a header to custom log--you can pick any header that we're actually receiving to verify;  I found http_content_md5 convenient for this
2. attach a program like nc to the output log socket on the radosgw host--if you use nc, it will stream the custom log records to stdout, and you should be able to verify the custom headers appear in a json labeled array in each matching record

Here's a snippet from my ceph.conf:
	# ops log
	rgw enable ops log = true
	rgw ops log socket path = /tmp/opslog
	#rgw log http headers = "http_x_forwarded_for, http_expect, http_content_md5"

Comment 27 Ramakrishnan Periyasamy 2017-02-17 04:12:17 EST
Moving this bug to verified state. verified in ceph: 10.2.5-26.el7cp
Thanks for the clarifications Matt.

Followed steps in comment23 and observed http_content_md5 header in nc stdout as mentioned in it.

Ceph.conf params:

rgw enable ops log = true
rgw ops log socket path = /tmp/opslog
rgw enable usage log = true
rgw log http headers = "http_x_forwarded_for, http_expect, http_content_md5"

after restarting rgw service a socket created in below path


nc stdout:

[ubuntu@host023 ~]$ sudo nc -U /tmp/systemd-private-459cca697e8740108d3134fb6601def2-ceph-radosgw@rgw.host023.service-7SlMGl/tmp/opslog
[{"bucket":"client2_run3","time":"2017-02-17 06:22:22.420641Z","time_local":"2017-02-17 06:22:22.420641","remote_addr":"","user":"nfsuser","operation":"PUT","uri":"\/client2_run3\/","http_status":"200","error_code":"","bytes_sent":19,"bytes_received":0,"object_size":0,"total_time":0,"user_agent":"Boto\/2.45.0 Python\/2.7.5 Linux\/3.10.0-514.6.1.el7.x86_64","referrer":""},
{"bucket":"client2_run3","time":"2017-02-17 06:22:22.619912Z","time_local":"2017-02-17 06:22:22.619912","remote_addr":"","user":"nfsuser","operation":"PUT","uri":"\/client2_run3\/redhat1607329","http_status":"200","error_code":"","bytes_sent":0,"bytes_received":1595408,"object_size":1595408,"total_time":0,"user_agent":"Boto\/2.45.0 Python\/2.7.5 Linux\/3.10.0-514.6.1.el7.x86_64","referrer":"","http_x_headers":[{"HTTP_CONTENT_MD5":"O3T1zQdAoFgCojsBnOV5ow=="},{"HTTP_EXPECT":"100-Continue"}]},
{"bucket":"client2_run3","time":"2017-02-17 06:22:22.861734Z","time_local":"2017-02-17 06:22:22.861734","remote_addr":"","user":"nfsuser","operation":"PUT","uri":"\/client2_run3\/redhat1607330","http_status":"200","error_code":"","bytes_sent":0,"bytes_received":1595408,"object_size":1595408,"total_time":0,"user_agent":"Boto\/2.45.0 Python\/2.7.5 Linux\/3.10.0-514.6.1.el7.x86_64","referrer":"","http_x_headers":[{"HTTP_CONTENT_MD5":"O3T1zQdAoFgCojsBnOV5ow=="},{"HTTP_EXPECT":"100-Continue"}]},
{"bucket":"client2_run3","time":"2017-02-17 06:22:22.958222Z","time_local":"2017-02-17 06:22:22.958222","remote_addr":"","user":"nfsuser","operation":"PUT","uri":"\/client2_run3\/redhat1607331","http_status":"200","error_code":"","bytes_sent":0,"bytes_received":1595408,"object_size":1595408,"total_time":0,"user_agent":"Boto\/2.45.0 Python\/2.7.5 Linux\/3.10.0-514.6.1.el7.x86_64","referrer":"","http_x_headers":[{"HTTP_CONTENT_MD5":"O3T1zQdAoFgCojsBnOV5ow=="},{"HTTP_EXPECT":"100-Continue"}]},
{"bucket":"client2_run3","time":"2017-02-17 06:22:23.107756Z","time_local":"2017-02-17 06:22:23.107756","remote_addr":"","user":"nfsuser","operation":"PUT","uri":"\/client2_run3\/redhat1607332","http_status":"200","error_code":"","bytes_sent":0,"bytes_received":1595408,"object_size":1595408,"total_time":0,"user_agent":"Boto\/2.45.0 Python\/2.7.5 Linux\/3.10.0-514.6.1.el7.x86_64","referrer":"","http_x_headers":[{"HTTP_CONTENT_MD5":"O3T1zQdAoFgCojsBnOV5ow=="},{"HTTP_EXPECT":"100-Continue"}]},
Comment 32 errata-xmlrpc 2017-03-14 11:43:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.