Bug 2230234 - [s3select][json]: rgw crashes seen for query "select * from s3object[*].phonenumbers where phoneNumbers.addr is NULL;"
Summary: [s3select][json]: rgw crashes seen for query "select * from s3object[*].phone...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 7.0
Assignee: gal salomon
QA Contact: Hemanth Sai
Rivka Pollack
URL:
Whiteboard:
Depends On:
Blocks: 2237662
TreeView+ depends on / blocked
 
Reported: 2023-08-09 04:45 UTC by Hemanth Sai
Modified: 2023-12-13 15:21 UTC (History)
6 users (show)

Fixed In Version: ceph-18.2.0-105.el9cp
Doc Type: Bug Fix
Doc Text:
.Ceph Object Gateway daemon no longer crashes where `phoneNumbers.addr` is `NULL` Previously, due to a syntax error, the query for `select * from s3object[*].phonenumbers where phoneNumbers.addr is NULL;` would cause the Ceph Object Gateway daemon to crash. With this fix the wrong syntax is identified and reported, no longer causing the daemon to crash.
Clone Of:
Environment:
Last Closed: 2023-12-13 15:21:51 UTC
Embargoed:
gsalomon: needinfo-
gsalomon: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-7179 0 None None None 2023-08-09 04:46:59 UTC
Red Hat Product Errata RHBA-2023:7780 0 None None None 2023-12-13 15:21:56 UTC

Description Hemanth Sai 2023-08-09 04:45:55 UTC
Description of problem:
rgw daemon is crashing while executing the below query
[cephuser@ceph-hmaheswa-rhcs7-fiisfo-node6 ~]$ aws s3api select-object-content --endpoint-url http://10.0.210.237:80 --bucket bkt1 --key example1_json --expression-type 'SQL' --input-serialization '{"JSON": {"Type": "DOCUMENT"}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}' --expression "select * from s3object[*].phonenumbers where phoneNumbers.addr is NULL;" /dev/stdout

("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
[cephuser@ceph-hmaheswa-rhcs7-fiisfo-node6 ~]$


crash info:
[root@ceph-hmaheswa-rhcs7-fiisfo-node5 2023-08-09T04:39:49.720640Z_13631980-1ab3-472c-be1b-e8291f85c1c7]# cat meta 
{
    "crash_id": "2023-08-09T04:39:49.720640Z_13631980-1ab3-472c-be1b-e8291f85c1c7",
    "timestamp": "2023-08-09T04:39:49.720640Z",
    "process_name": "radosgw",
    "entity_name": "client.rgw.rgw.1.ceph-hmaheswa-rhcs7-fiisfo-node5.vnrgwe",
    "ceph_version": "18.1.2-1.el9cp",
    "utsname_hostname": "ceph-hmaheswa-rhcs7-fiisfo-node5",
    "utsname_sysname": "Linux",
    "utsname_release": "5.14.0-284.25.1.el9_2.x86_64",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Jul 20 09:11:28 EDT 2023",
    "utsname_machine": "x86_64",
    "os_name": "Red Hat Enterprise Linux",
    "os_id": "rhel",
    "os_version_id": "9.2",
    "os_version": "9.2 (Plow)",
    "backtrace": [
        "/lib64/libc.so.6(+0x54df0) [0x7f5330ddbdf0]",
        "/lib64/libc.so.6(+0xa154c) [0x7f5330e2854c]",
        "raise()",
        "abort()",
        "/usr/bin/radosgw(+0x309588) [0x564f9f044588]",
        "/usr/bin/radosgw(+0x6126f7) [0x564f9f34d6f7]",
        "/usr/bin/radosgw(+0x61e292) [0x564f9f359292]",
        "/usr/bin/radosgw(+0x6286c2) [0x564f9f3636c2]",
        "/usr/bin/radosgw(+0xb972e9) [0x564f9f8d22e9]",
        "/usr/bin/radosgw(+0x634c00) [0x564f9f36fc00]",
        "/usr/bin/radosgw(+0x61dc0d) [0x564f9f358c0d]",
        "/usr/bin/radosgw(+0xb144d4) [0x564f9f84f4d4]",
        "/usr/bin/radosgw(+0x61fe06) [0x564f9f35ae06]",
        "(RGWSelectObj_ObjStore_S3::run_s3select_on_json(char const*, char const*, unsigned long)+0x9de) [0x564f9f37f2be]",
        "(RGWSelectObj_ObjStore_S3::json_processing(ceph::buffer::v15_2_0::list&, long, long)+0x633) [0x564f9f3826f3]",
        "(RGWRados::get_obj_iterate_cb(DoutPrefixProvider const*, rgw_raw_obj const&, long, long, long, bool, RGWObjState*, void*)+0x11a) [0x564f9f470aaa]",
        "/usr/bin/radosgw(+0x730dd6) [0x564f9f46bdd6]",
        "(RGWRados::iterate_obj(DoutPrefixProvider const*, RGWObjectCtx&, RGWBucketInfo&, rgw_obj const&, long, long, unsigned long, int (*)(DoutPrefixProvider const*, rgw_raw_obj const&, long, long, long, bool, RGWObjState*, void*), void*, optional_yield)+0x428) [0x564f9f4713a8]",
        "(RGWRados::Object::Read::iterate(DoutPrefixProvider const*, long, long, RGWGetDataCB*, optional_yield)+0x134) [0x564f9f471b64]",
        "(RGWGetObj::execute(optional_yield)+0xf59) [0x564f9f29c809]",
        "(RGWSelectObj_ObjStore_S3::execute(optional_yield)+0xc1) [0x564f9f3845b1]",
        "(rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xa72) [0x564f9f14e1a2]",
        "(process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x1039) [0x564f9f151719]",
        "/usr/bin/radosgw(+0xb4f176) [0x564f9f88a176]",
        "/usr/bin/radosgw(+0x371ce1) [0x564f9f0acce1]",
        "make_fcontext()"
    ]
}
[root@ceph-hmaheswa-rhcs7-fiisfo-node5 2023-08-09T04:39:49.720640Z_13631980-1ab3-472c-be1b-e8291f85c1c7]# 



json data uploaded:
{
 "firstName": "Joe",
 "lastName": "Jackson",
 "gender": "male",
 "age": "twenty",
 "address": {
 "streetAddress": "101",
 "city": "San Diego",
 "state": "CA"
 },

 "firstName": "Joe_2",
 "lastName": "Jackson_2",
 "gender": "male",
 "age": 21,
 "address": {
 "streetAddress": "101",
 "city": "San Diego",
 "state": "CA"
 },

 "phoneNumbers": [
   { "type": "home1", "number": "734928_1","addr": 11 },
   { "type": "home2", "number": "734928_2","addr": 22 },
   { "type": "home3", "number": "734928_3","addr": 33 },
   { "type": "home4", "number": "734928_4","addr": 44 },
   { "type": "home5", "number": "734928_5","addr": 55 },
   { "type": "home6", "number": "734928_6","addr": 66 },
   { "type": "home7", "number": "734928_7","addr": 77 },
   { "type": "home8", "number": "734928_8","addr": 88 },
   { "type": "home9", "number": "734928_9","addr": 99 },
   { "type": "home10", "number": "734928_10","addr": 100 }
 ],

 "key_after_array": "XXX",

 "description" : {
   "main_desc" : "value_1",
   "second_desc" : "value_2"
 }
}


Version-Release number of selected component (if applicable):
ceph version 18.1.2-1.el9cp

How reproducible:
always

Steps to Reproduce:
1.deploy rhcs7.0 ceph cluster
2.upload a json object using aws-cli
3.execute the query "select * from s3object[*].phonenumbers where phoneNumbers.addr is NULL;" 

Actual results:
rgw daemon is crashing

Expected results:
It should display proper error instead of crashing

Additional info:
rgw logs and crash logs are present here: http://magna002.ceph.redhat.com/ceph-qe-logs/HemanthSai/s3select_json_bz_logs/ceph-client.rgw.rgw.1.ceph-hmaheswa-rhcs7-fiisfo-node5.vnrgwe.log

rgw node:10.0.210.237
creds: root/passwd ; cephuser/cephuser

Comment 2 gal salomon 2023-08-16 15:44:55 UTC
https://github.com/ceph/ceph/pull/52651

Comment 3 gal salomon 2023-08-16 15:47:02 UTC
the crash is fixed on https://github.com/ceph/ceph/pull/52651

Comment 9 gal salomon 2023-10-30 10:05:46 UTC
`An error occurred (s3select-Syntax-Error) when calling the SelectObjectContent operation: engine_version  function not found`

with that returns the correct value 
there is no point in testing `18.2.0-102.el9cp`

Comment 15 errata-xmlrpc 2023-12-13 15:21:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7780


Note You need to log in before you can comment on or make changes to this bug.