Bug 2290775

Summary: [rgw][s3select]: rgw going down executing query "select ( not ( 'abc' + 'def' ) ) from s3object;"
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Hemanth Sai <hmaheswa>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED ERRATA QA Contact: Anuchaithra <anrao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.1CC: akraj, anrao, ceph-eng-bugs, cephqe-warriors, gsalomon, mbenjamin, rpollack, tserlin
Target Milestone: ---Keywords: Automation
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-19.1.1-20.el9cp Doc Type: Bug Fix
Doc Text:
.The AST structure SQL statement no longer causes a crash Previously, in some cases, due to an erroneous semantic combined with the Parquet flow, the AST creation that is produced by the SQL engine was wrong a crash would occur. With this fix, more safety checks are in place for the AST structure and the statement processing time is fixed and a crash is avoided.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2024-11-25 09:01:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2317218    

Description Hemanth Sai 2024-06-06 19:34:51 UTC 
Description of problem:
rgw going down executing query "select ( not ( 'abc' + 'def' ) ) from s3object;"

[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content  --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}'  --expression "select ( not ( 'abc' + 'def' ) ) from s3object;" /dev/stdout

("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ 

[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content  --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}'  --expression "select ( not ( '2027-02-23T08:05:12.787679' + _3 ) ) from s3object;" /dev/stdout

("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ 




crash log snippet in rgw logs:


   -20> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s s3:get_obj s3-select query: <SelectObjectContentRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Expression>select ( not ( 'abc' + 'def' ) ) from s3object;</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><Parquet /><CompressionType>NONE</CompressionType></InputSerialization><OutputSerialization><CSV /></OutputSerialization></SelectObjectContentRequest>
   -19> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s s3:get_obj s3select: engine is set to process Parquet objects
   -18> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: calling execute(async): request-offset :0 request-length :4 buffer size : 0
   -17> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj get_obj_state: rctx=0x561d0f3df960 obj=jessicaj.385-bucky-3749-0:Key_jessicaj.385-bucky-3749-0_parquet state=0x561d116659e8 s->prefetch_data=1
   -16> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.acl
   -15> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.etag
   -14> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.idtag
   -13> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.manifest
   -12> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver
   -11> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone
   -10> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag
    -9> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256
    -8> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date
    -7> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s cache get: name=default.rgw.log++script.getdata. : hit (negative entry)
    -6> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 15 req 5377627009901700961 0.003000056s Encryption mode:
    -5> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj get_obj_state: rctx=0x561d0f3df960 obj=jessicaj.385-bucky-3749-0:Key_jessicaj.385-bucky-3749-0_parquet state=0x561d116659e8 s->prefetch_data=1
    -4> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: part 1 it.length() = 4516
    -3> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select:append_in_callback = 4516
    -2> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: buffer is complete 4 request range length:4
    -1> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: done waiting, buffer is complete buffer-size:4
     0> 2024-06-06T19:25:05.054+0000 7fbbe9ed7640 -1 *** Caught signal (Aborted) **
 in thread 7fbbe9ed7640 thread_name:radosgw

 ceph version 18.2.1-191.el9cp (cea42e150963ca5b7dc9a61e483e3043dcf8dfbd) reef (stable)
 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbc2e34d6f0]
 2: /lib64/libc.so.6(+0x8b94c) [0x7fbc2e39a94c]
 3: raise()
 4: abort()
 5: /usr/bin/radosgw(+0x324da8) [0x561d0b73eda8]
 6: (s3selectEngine::push_projection::builder(s3selectEngine::s3select*, char const*, char const*) const+0xf7) [0x561d0ba29537]
 7: /usr/bin/radosgw(+0x653e2f) [0x561d0ba6de2f]
 8: /usr/bin/radosgw(+0x6584c5) [0x561d0ba724c5]
 9: /usr/bin/radosgw(+0x6584fe) [0x561d0ba724fe]
 10: /usr/bin/radosgw(+0x6541ba) [0x561d0ba6e1ba]
 11: /usr/bin/radosgw(+0x65427c) [0x561d0ba6e27c]
 12: /usr/bin/radosgw(+0x65870e) [0x561d0ba7270e]
 13: /usr/bin/radosgw(+0x649636) [0x561d0ba63636]
 14: /usr/bin/radosgw(+0x61fbb9) [0x561d0ba39bb9]
 15: (RGWSelectObj_ObjStore_S3::run_s3select_on_parquet(char const*)+0x3f8) [0x561d0ba49ef8]
 16: (RGWSelectObj_ObjStore_S3::execute(optional_yield)+0x2d2) [0x561d0ba4ae92]
 17: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xa72) [0x561d0b8891b2]
 18: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x1039) [0x561d0b88ff79]
 19: /usr/bin/radosgw(+0xb951b0) [0x561d0bfaf1b0]
 20: /usr/bin/radosgw(+0x383e56) [0x561d0b79de56]
 21: make_fcontext()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.






parquet object:

[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content  --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}'  --expression "select * from s3object;" /dev/stdout
102557980536,170971523010216.66,SMc877hOpzVIznQGYws2,2027-02-23T12:05:12.666406+00:00
73722184098,151307343195801.44,XbLT3ezRDw4VY Gqd9Xe,2027-02-23T12:05:12.666429+00:00
4932309237,173877766869206.38,S ooLRT1 21Mh8d7u7H6,2027-02-23T12:05:12.666435+00:00
91678062661,159434544390944.88,dgso7xITc7EQY pP9f7Z,2027-02-23T12:05:12.666439+00:00
16176573691,211991748629029.03,pxzCCRfGIh iLlCg3hQl,2027-02-23T12:05:12.666444+00:00
16170177342,183610771622168.16,yJyEXATnhuqSJx8nWmKm,2027-02-23T12:05:12.666448+00:00
14748582300,156257235709899.31,36tVSMI1jUv59jsce9gt,2027-02-23T12:05:12.666453+00:00
7852551394,143732517395955.12,6sK5kHAsoOUR7Z7e9bkj,2027-02-23T12:05:12.666458+00:00
23897599683,129186835035137.27,h uXSBaMelhmn7pPJa5R,2027-02-23T12:05:12.666463+00:00
68971853006,143102205681198.78,8le8aAFCpnVUBEVIZxgP,2027-02-23T12:05:12.666467+00:00
84264539452,148920440750671.69,i0Mr024VgeuyPeyVk55E,2027-02-23T12:05:12.666472+00:00
74329796277,182390925296235.81, WCrXfNKcCT8RXyOOog ,2027-02-23T12:05:12.666476+00:00
6433380359,188437160016409.31,UBFLknlHUYJUoUdEQAAl,2027-02-23T12:05:12.666481+00:00
98392703576,159375205191966.03,pQzZ69ovVhvhhMKtAl4Q,2027-02-23T12:05:12.666485+00:00
34924458294,126544727839598.28,HF97Pgpdpx8JrOHhns4V,2027-02-23T12:05:12.666490+00:00
32730509179,185776479113536.88,KMnOaPLGxJq7hvn8U9uN,2027-02-23T12:05:12.666494+00:00
62111244673,201417320418787.25,0vKmJzeqpoD9q8ifq12k,2027-02-23T12:05:12.666499+00:00
102615089192,200666197093090,A3dltQeViw5SUVplZ o8,2027-02-23T12:05:12.666506+00:00
58390797321,155818765611447.03,QiN8CMW3XPIEb9DnIO9N,2027-02-23T12:05:12.666510+00:00
95648381174,179228444827592.31,zRzI6tAEyoeb HhoeHD2,2027-02-23T12:05:12.666515+00:00
81131952257,179096643300878.03,PoLg5WzpznHwHcEY4zUD,2027-02-23T12:05:12.666519+00:00
65534701929,182204058688273,sZzJIXeoIEi5ujy68rE1,2027-02-23T12:05:12.666523+00:00
8406707874,186958563731280.62,lonZx BJ7f1nATg2Hz6O,2027-02-23T12:05:12.666528+00:00
47999458421,181796716255134.47,u27d2qqFHzBLdc1x7TMg,2027-02-23T12:05:12.666532+00:00
49077978011,125896214780070.72,gVUnVbhb2ccy cSLMaT5,2027-02-23T12:05:12.666537+00:00
50581453102,111496494847716.5,tt31SNtytut2Jj6z5McE,2027-02-23T12:05:12.666542+00:00
102145530113,121083314121489.23,bh1aJ2kil13siG8rW3nU,2027-02-23T12:05:12.666546+00:00
52460678937,165885757440844.19,NrwQ0WGjNoKuPIQxOOtT,2027-02-23T12:05:12.666551+00:00
34306251347,135961078527472.48,El7nDW6T9Qg7IZ4NSLRN,2027-02-23T12:05:12.666555+00:00
14480007386,164309631283457.66,ih38Q rOkfe6y3iTNP8m,2027-02-23T12:05:12.666560+00:00
[cephuser@ceph-hsm-71-8j01oc-node6 ~]$ 




Version-Release number of selected component (if applicable):
ceph version 18.2.1-191.el9cp

How reproducible:
always

Steps to Reproduce:
1.deploy rhcs7.1 ceph cluster with rgw daemons
2.upload parquet object using aws-cli
3.execute the query "select ( not ( 'abc' + 'def' ) ) from s3object;"

Actual results:
rgw going down executing query

Expected results:
rgw should not crash

Additional info:
automation fail log: http://magna002.ceph.redhat.com/cephci-jenkins/hsm/s3select_parquet_and_tfa/cephci-run-8J01OC/test_s3select_with_depth2_queries_on_parquet_objects_0.log
Comment 1 gal salomon 2024-06-08 04:18:34 UTC 
the crash happened during the parsing stage, it happened upon Parquet-flow and not CSV-flow (for the same statement) 
that is because the Parquet flow is parsing twice (wrongly) the SQL statement(AST creation), this may cause unexpected behavior.
Comment 2 gal salomon 2024-06-09 12:22:22 UTC 
https://github.com/ceph/ceph/pull/57944
Comment 10 errata-xmlrpc 2024-11-25 09:01:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:10216
Comment 11 Red Hat Bugzilla 2025-03-26 04:25:34 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days