Description of problem: When trying to send bucket notifications to ElasticSearch HTTP endpoint (http://elasticsearch:9200/index/_doc/), the following error appears in RGW logs: --- debug 2021-02-24 14:34:34.180 7f6185119700 1 ====== starting new request req=0x7f629e30a680 ===== debug 2021-02-24 14:34:34.190 7f6183916700 1 push to endpoint HTTP/S Endpoint URI: http://elasticsearch-sample-es-http.rgw-es.svc:9200/s3_index/_doc/ Ack Level: don't verify SSL failed, with error: -5 debug 2021-02-24 14:34:34.190 7f6183916700 1 ====== req done req=0x7f629e30a680 op status=0 http_status=200 latency=0.0100002s ====== debug 2021-02-24 14:34:34.190 7f6183916700 1 beast: 0x7f629e30a680: 10.131.2.41 - - [2021-02-24 14:34:34.0.190742s] "PUT /std-user-bucket1/Pipfile HTTP/1.1" 200 180 - "Boto3/1.17.14 Python/3.6.8 Linux/4.18.0-193.41.1.el8_2.x86_64 Botocore/1.20.14" - --- Notes: - The error is not linked to SSL, this is a generic error message. Endpoint is working over http. Plus SSL verify is disabled from the bucket notification configuration. - Direct curl command over endpoint works flawlessly. - After discussion with Yuval Lifshitz, "its is because the client send "Expect: 100-continue" - If a server respect that field, and actually answer with 100 Continue result code, the RGW treat that as an -EIO. -EIO = -5" - This bug seems to have been corrected by this upstream and in the 4.2z1 branch: https://github.com/ceph/ceph/pull/34414 So apparently it would only need to make its way to OCS. Version-Release number of selected component (if applicable): OCS 4.6.2 How reproducible: Send an RGW bucket notification to an ElasticSearch endpoint. Actual results: - Notification sending error. Expected results: - Notification sent and logged into ElasticSearch. Additional info: This problem should occur with any http endpoint respecting the "Expect: 100-continue" flag in the request.
issue was fixed by the following commit: commit 75b17dd1193d63f60eb677d7523321717626299c Author: Yuval Lifshitz <yuvalif> Date: Mon Apr 6 12:50:37 2020 +0300 rgw/http: add timeout to http client also, prevent "Expect: 100-continue" from being sent when not needed Signed-off-by: Yuval Lifshitz <yuvalif> (cherry picked from commit dd49cc83078c7e268ce3de7ab0bfbf3035ed5d50)
Confirmed with Yuval, above commit is already there in RHCS4.2z1. Moving this BZ to MODIFIED.
Not able to verifiy this bz due to put object error. See https://bugzilla.redhat.com/show_bug.cgi?id=1932396#c10 for more detail.
Hi Yuval, PTAL, do you know in which exact ceph version this fix went in? Thanks
Hi Yuval, I have tracked the commit, it is here: https://gitlab.cee.redhat.com/ceph/ceph/-/commit/75b17dd1193d63f60eb677d7523321717626299c Which means the build we are testing with has the patch but if we are still hitting the issue we may have to investigate more. Let me know if I have to open a Ceph BZ for this. Thanks
This seems to be a tracker for an RHCS issue that does not have a BZ. Please create an RHCS BZ for this issue and link it here.
Created a tracker and moving out of 4.7 as we need a fix in Ceph. Will set the acks accordingly.
Fix should be available in the latest ODF builds
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.9.0 enhancement, security, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:5086