Bug 2290775 - [rgw][s3select]: rgw going down executing query "select ( not ( 'abc' + 'def' ) ) from s3object;"
Summary: [rgw][s3select]: rgw going down executing query "select ( not ( 'abc' + 'def'...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 8.0
Assignee: Matt Benjamin (redhat)
QA Contact: Anuchaithra
URL:
Whiteboard:
Depends On:
Blocks: 2317218
TreeView+ depends on / blocked
 
Reported: 2024-06-06 19:34 UTC by Hemanth Sai
Modified: 2025-03-26 04:25 UTC (History)
8 users (show)

Fixed In Version: ceph-19.1.1-20.el9cp
Doc Type: Bug Fix
Doc Text:
.The AST structure SQL statement no longer causes a crash Previously, in some cases, due to an erroneous semantic combined with the Parquet flow, the AST creation that is produced by the SQL engine was wrong a crash would occur. With this fix, more safety checks are in place for the AST structure and the statement processing time is fixed and a crash is avoided.
Clone Of:
Environment:
Last Closed: 2024-11-25 09:01:44 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-9150 0 None None None 2024-06-06 19:36:16 UTC
Red Hat Product Errata RHBA-2024:10216 0 None None None 2024-11-25 09:01:49 UTC

Description Hemanth Sai 2024-06-06 19:34:51 UTC
Description of problem:
rgw going down executing query "select ( not ( 'abc' + 'def' ) ) from s3object;"

[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content  --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}'  --expression "select ( not ( 'abc' + 'def' ) ) from s3object;" /dev/stdout

("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ 

[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content  --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}'  --expression "select ( not ( '2027-02-23T08:05:12.787679' + _3 ) ) from s3object;" /dev/stdout

("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ 




crash log snippet in rgw logs:


   -20> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s s3:get_obj s3-select query: <SelectObjectContentRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Expression>select ( not ( 'abc' + 'def' ) ) from s3object;</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><Parquet /><CompressionType>NONE</CompressionType></InputSerialization><OutputSerialization><CSV /></OutputSerialization></SelectObjectContentRequest>
   -19> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s s3:get_obj s3select: engine is set to process Parquet objects
   -18> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: calling execute(async): request-offset :0 request-length :4 buffer size : 0
   -17> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj get_obj_state: rctx=0x561d0f3df960 obj=jessicaj.385-bucky-3749-0:Key_jessicaj.385-bucky-3749-0_parquet state=0x561d116659e8 s->prefetch_data=1
   -16> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.acl
   -15> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.etag
   -14> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.idtag
   -13> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.manifest
   -12> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver
   -11> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone
   -10> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag
    -9> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256
    -8> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date
    -7> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s cache get: name=default.rgw.log++script.getdata. : hit (negative entry)
    -6> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 15 req 5377627009901700961 0.003000056s Encryption mode:
    -5> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj get_obj_state: rctx=0x561d0f3df960 obj=jessicaj.385-bucky-3749-0:Key_jessicaj.385-bucky-3749-0_parquet state=0x561d116659e8 s->prefetch_data=1
    -4> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: part 1 it.length() = 4516
    -3> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select:append_in_callback = 4516
    -2> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: buffer is complete 4 request range length:4
    -1> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: done waiting, buffer is complete buffer-size:4
     0> 2024-06-06T19:25:05.054+0000 7fbbe9ed7640 -1 *** Caught signal (Aborted) **
 in thread 7fbbe9ed7640 thread_name:radosgw

 ceph version 18.2.1-191.el9cp (cea42e150963ca5b7dc9a61e483e3043dcf8dfbd) reef (stable)
 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbc2e34d6f0]
 2: /lib64/libc.so.6(+0x8b94c) [0x7fbc2e39a94c]
 3: raise()
 4: abort()
 5: /usr/bin/radosgw(+0x324da8) [0x561d0b73eda8]
 6: (s3selectEngine::push_projection::builder(s3selectEngine::s3select*, char const*, char const*) const+0xf7) [0x561d0ba29537]
 7: /usr/bin/radosgw(+0x653e2f) [0x561d0ba6de2f]
 8: /usr/bin/radosgw(+0x6584c5) [0x561d0ba724c5]
 9: /usr/bin/radosgw(+0x6584fe) [0x561d0ba724fe]
 10: /usr/bin/radosgw(+0x6541ba) [0x561d0ba6e1ba]
 11: /usr/bin/radosgw(+0x65427c) [0x561d0ba6e27c]
 12: /usr/bin/radosgw(+0x65870e) [0x561d0ba7270e]
 13: /usr/bin/radosgw(+0x649636) [0x561d0ba63636]
 14: /usr/bin/radosgw(+0x61fbb9) [0x561d0ba39bb9]
 15: (RGWSelectObj_ObjStore_S3::run_s3select_on_parquet(char const*)+0x3f8) [0x561d0ba49ef8]
 16: (RGWSelectObj_ObjStore_S3::execute(optional_yield)+0x2d2) [0x561d0ba4ae92]
 17: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xa72) [0x561d0b8891b2]
 18: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x1039) [0x561d0b88ff79]
 19: /usr/bin/radosgw(+0xb951b0) [0x561d0bfaf1b0]
 20: /usr/bin/radosgw(+0x383e56) [0x561d0b79de56]
 21: make_fcontext()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.






parquet object:

[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content  --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}'  --expression "select * from s3object;" /dev/stdout
102557980536,170971523010216.66,SMc877hOpzVIznQGYws2,2027-02-23T12:05:12.666406+00:00
73722184098,151307343195801.44,XbLT3ezRDw4VY Gqd9Xe,2027-02-23T12:05:12.666429+00:00
4932309237,173877766869206.38,S ooLRT1 21Mh8d7u7H6,2027-02-23T12:05:12.666435+00:00
91678062661,159434544390944.88,dgso7xITc7EQY pP9f7Z,2027-02-23T12:05:12.666439+00:00
16176573691,211991748629029.03,pxzCCRfGIh iLlCg3hQl,2027-02-23T12:05:12.666444+00:00
16170177342,183610771622168.16,yJyEXATnhuqSJx8nWmKm,2027-02-23T12:05:12.666448+00:00
14748582300,156257235709899.31,36tVSMI1jUv59jsce9gt,2027-02-23T12:05:12.666453+00:00
7852551394,143732517395955.12,6sK5kHAsoOUR7Z7e9bkj,2027-02-23T12:05:12.666458+00:00
23897599683,129186835035137.27,h uXSBaMelhmn7pPJa5R,2027-02-23T12:05:12.666463+00:00
68971853006,143102205681198.78,8le8aAFCpnVUBEVIZxgP,2027-02-23T12:05:12.666467+00:00
84264539452,148920440750671.69,i0Mr024VgeuyPeyVk55E,2027-02-23T12:05:12.666472+00:00
74329796277,182390925296235.81, WCrXfNKcCT8RXyOOog ,2027-02-23T12:05:12.666476+00:00
6433380359,188437160016409.31,UBFLknlHUYJUoUdEQAAl,2027-02-23T12:05:12.666481+00:00
98392703576,159375205191966.03,pQzZ69ovVhvhhMKtAl4Q,2027-02-23T12:05:12.666485+00:00
34924458294,126544727839598.28,HF97Pgpdpx8JrOHhns4V,2027-02-23T12:05:12.666490+00:00
32730509179,185776479113536.88,KMnOaPLGxJq7hvn8U9uN,2027-02-23T12:05:12.666494+00:00
62111244673,201417320418787.25,0vKmJzeqpoD9q8ifq12k,2027-02-23T12:05:12.666499+00:00
102615089192,200666197093090,A3dltQeViw5SUVplZ o8,2027-02-23T12:05:12.666506+00:00
58390797321,155818765611447.03,QiN8CMW3XPIEb9DnIO9N,2027-02-23T12:05:12.666510+00:00
95648381174,179228444827592.31,zRzI6tAEyoeb HhoeHD2,2027-02-23T12:05:12.666515+00:00
81131952257,179096643300878.03,PoLg5WzpznHwHcEY4zUD,2027-02-23T12:05:12.666519+00:00
65534701929,182204058688273,sZzJIXeoIEi5ujy68rE1,2027-02-23T12:05:12.666523+00:00
8406707874,186958563731280.62,lonZx BJ7f1nATg2Hz6O,2027-02-23T12:05:12.666528+00:00
47999458421,181796716255134.47,u27d2qqFHzBLdc1x7TMg,2027-02-23T12:05:12.666532+00:00
49077978011,125896214780070.72,gVUnVbhb2ccy cSLMaT5,2027-02-23T12:05:12.666537+00:00
50581453102,111496494847716.5,tt31SNtytut2Jj6z5McE,2027-02-23T12:05:12.666542+00:00
102145530113,121083314121489.23,bh1aJ2kil13siG8rW3nU,2027-02-23T12:05:12.666546+00:00
52460678937,165885757440844.19,NrwQ0WGjNoKuPIQxOOtT,2027-02-23T12:05:12.666551+00:00
34306251347,135961078527472.48,El7nDW6T9Qg7IZ4NSLRN,2027-02-23T12:05:12.666555+00:00
14480007386,164309631283457.66,ih38Q rOkfe6y3iTNP8m,2027-02-23T12:05:12.666560+00:00
[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ 




Version-Release number of selected component (if applicable):
ceph version 18.2.1-191.el9cp

How reproducible:
always

Steps to Reproduce:
1.deploy rhcs7.1 ceph cluster with rgw daemons
2.upload parquet object using aws-cli
3.execute the query "select ( not ( 'abc' + 'def' ) ) from s3object;"

Actual results:
rgw going down executing query

Expected results:
rgw should not crash

Additional info:
automation fail log: http://magna002.ceph.redhat.com/cephci-jenkins/hsm/s3select_parquet_and_tfa/cephci-run-8J01OC/test_s3select_with_depth2_queries_on_parquet_objects_0.log

Comment 1 gal salomon 2024-06-08 04:18:34 UTC
the crash happened during the parsing stage, it happened upon Parquet-flow and not CSV-flow (for the same statement) 
that is because the Parquet flow is parsing twice (wrongly) the SQL statement(AST creation), this may cause unexpected behavior.

Comment 2 gal salomon 2024-06-09 12:22:22 UTC
https://github.com/ceph/ceph/pull/57944

Comment 10 errata-xmlrpc 2024-11-25 09:01:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:10216

Comment 11 Red Hat Bugzilla 2025-03-26 04:25:34 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days


Note You need to log in before you can comment on or make changes to this bug.