Description of problem: rgw going down executing query "select ( not ( 'abc' + 'def' ) ) from s3object;" [cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}' --expression "select ( not ( 'abc' + 'def' ) ) from s3object;" /dev/stdout ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) [cephuser@ceph-hsm-71-8j01oc-node6 ~]$ [cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}' --expression "select ( not ( '2027-02-23T08:05:12.787679' + _3 ) ) from s3object;" /dev/stdout ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)) [cephuser@ceph-hsm-71-8j01oc-node6 ~]$ crash log snippet in rgw logs: -20> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s s3:get_obj s3-select query: <SelectObjectContentRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Expression>select ( not ( 'abc' + 'def' ) ) from s3object;</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><Parquet /><CompressionType>NONE</CompressionType></InputSerialization><OutputSerialization><CSV /></OutputSerialization></SelectObjectContentRequest> -19> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s s3:get_obj s3select: engine is set to process Parquet objects -18> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: calling execute(async): request-offset :0 request-length :4 buffer size : 0 -17> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj get_obj_state: rctx=0x561d0f3df960 obj=jessicaj.385-bucky-3749-0:Key_jessicaj.385-bucky-3749-0_parquet state=0x561d116659e8 s->prefetch_data=1 -16> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.acl -15> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.etag -14> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.idtag -13> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.manifest -12> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver -11> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone -10> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag -9> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 -8> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date -7> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 req 5377627009901700961 0.003000056s cache get: name=default.rgw.log++script.getdata. : hit (negative entry) -6> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 15 req 5377627009901700961 0.003000056s Encryption mode: -5> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 20 req 5377627009901700961 0.003000056s s3:get_obj get_obj_state: rctx=0x561d0f3df960 obj=jessicaj.385-bucky-3749-0:Key_jessicaj.385-bucky-3749-0_parquet state=0x561d116659e8 s->prefetch_data=1 -4> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: part 1 it.length() = 4516 -3> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select:append_in_callback = 4516 -2> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: buffer is complete 4 request range length:4 -1> 2024-06-06T19:25:05.049+0000 7fbbe9ed7640 10 S3select: done waiting, buffer is complete buffer-size:4 0> 2024-06-06T19:25:05.054+0000 7fbbe9ed7640 -1 *** Caught signal (Aborted) ** in thread 7fbbe9ed7640 thread_name:radosgw ceph version 18.2.1-191.el9cp (cea42e150963ca5b7dc9a61e483e3043dcf8dfbd) reef (stable) 1: /lib64/libc.so.6(+0x3e6f0) [0x7fbc2e34d6f0] 2: /lib64/libc.so.6(+0x8b94c) [0x7fbc2e39a94c] 3: raise() 4: abort() 5: /usr/bin/radosgw(+0x324da8) [0x561d0b73eda8] 6: (s3selectEngine::push_projection::builder(s3selectEngine::s3select*, char const*, char const*) const+0xf7) [0x561d0ba29537] 7: /usr/bin/radosgw(+0x653e2f) [0x561d0ba6de2f] 8: /usr/bin/radosgw(+0x6584c5) [0x561d0ba724c5] 9: /usr/bin/radosgw(+0x6584fe) [0x561d0ba724fe] 10: /usr/bin/radosgw(+0x6541ba) [0x561d0ba6e1ba] 11: /usr/bin/radosgw(+0x65427c) [0x561d0ba6e27c] 12: /usr/bin/radosgw(+0x65870e) [0x561d0ba7270e] 13: /usr/bin/radosgw(+0x649636) [0x561d0ba63636] 14: /usr/bin/radosgw(+0x61fbb9) [0x561d0ba39bb9] 15: (RGWSelectObj_ObjStore_S3::run_s3select_on_parquet(char const*)+0x3f8) [0x561d0ba49ef8] 16: (RGWSelectObj_ObjStore_S3::execute(optional_yield)+0x2d2) [0x561d0ba4ae92] 17: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xa72) [0x561d0b8891b2] 18: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x1039) [0x561d0b88ff79] 19: /usr/bin/radosgw(+0xb951b0) [0x561d0bfaf1b0] 20: /usr/bin/radosgw(+0x383e56) [0x561d0b79de56] 21: make_fcontext() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. parquet object: [cephuser@ceph-hsm-71-8j01oc-node6 ~]$ aws s3api --endpoint-url http://10.0.208.194:80 select-object-content --bucket jessicaj.385-bucky-3749-0 --key Key_jessicaj.385-bucky-3749-0_parquet --expression-type 'SQL' --input-serialization '{"Parquet": {}, "CompressionType": "NONE"}' --output-serialization '{"CSV": {}}' --expression "select * from s3object;" /dev/stdout 102557980536,170971523010216.66,SMc877hOpzVIznQGYws2,2027-02-23T12:05:12.666406+00:00 73722184098,151307343195801.44,XbLT3ezRDw4VY Gqd9Xe,2027-02-23T12:05:12.666429+00:00 4932309237,173877766869206.38,S ooLRT1 21Mh8d7u7H6,2027-02-23T12:05:12.666435+00:00 91678062661,159434544390944.88,dgso7xITc7EQY pP9f7Z,2027-02-23T12:05:12.666439+00:00 16176573691,211991748629029.03,pxzCCRfGIh iLlCg3hQl,2027-02-23T12:05:12.666444+00:00 16170177342,183610771622168.16,yJyEXATnhuqSJx8nWmKm,2027-02-23T12:05:12.666448+00:00 14748582300,156257235709899.31,36tVSMI1jUv59jsce9gt,2027-02-23T12:05:12.666453+00:00 7852551394,143732517395955.12,6sK5kHAsoOUR7Z7e9bkj,2027-02-23T12:05:12.666458+00:00 23897599683,129186835035137.27,h uXSBaMelhmn7pPJa5R,2027-02-23T12:05:12.666463+00:00 68971853006,143102205681198.78,8le8aAFCpnVUBEVIZxgP,2027-02-23T12:05:12.666467+00:00 84264539452,148920440750671.69,i0Mr024VgeuyPeyVk55E,2027-02-23T12:05:12.666472+00:00 74329796277,182390925296235.81, WCrXfNKcCT8RXyOOog ,2027-02-23T12:05:12.666476+00:00 6433380359,188437160016409.31,UBFLknlHUYJUoUdEQAAl,2027-02-23T12:05:12.666481+00:00 98392703576,159375205191966.03,pQzZ69ovVhvhhMKtAl4Q,2027-02-23T12:05:12.666485+00:00 34924458294,126544727839598.28,HF97Pgpdpx8JrOHhns4V,2027-02-23T12:05:12.666490+00:00 32730509179,185776479113536.88,KMnOaPLGxJq7hvn8U9uN,2027-02-23T12:05:12.666494+00:00 62111244673,201417320418787.25,0vKmJzeqpoD9q8ifq12k,2027-02-23T12:05:12.666499+00:00 102615089192,200666197093090,A3dltQeViw5SUVplZ o8,2027-02-23T12:05:12.666506+00:00 58390797321,155818765611447.03,QiN8CMW3XPIEb9DnIO9N,2027-02-23T12:05:12.666510+00:00 95648381174,179228444827592.31,zRzI6tAEyoeb HhoeHD2,2027-02-23T12:05:12.666515+00:00 81131952257,179096643300878.03,PoLg5WzpznHwHcEY4zUD,2027-02-23T12:05:12.666519+00:00 65534701929,182204058688273,sZzJIXeoIEi5ujy68rE1,2027-02-23T12:05:12.666523+00:00 8406707874,186958563731280.62,lonZx BJ7f1nATg2Hz6O,2027-02-23T12:05:12.666528+00:00 47999458421,181796716255134.47,u27d2qqFHzBLdc1x7TMg,2027-02-23T12:05:12.666532+00:00 49077978011,125896214780070.72,gVUnVbhb2ccy cSLMaT5,2027-02-23T12:05:12.666537+00:00 50581453102,111496494847716.5,tt31SNtytut2Jj6z5McE,2027-02-23T12:05:12.666542+00:00 102145530113,121083314121489.23,bh1aJ2kil13siG8rW3nU,2027-02-23T12:05:12.666546+00:00 52460678937,165885757440844.19,NrwQ0WGjNoKuPIQxOOtT,2027-02-23T12:05:12.666551+00:00 34306251347,135961078527472.48,El7nDW6T9Qg7IZ4NSLRN,2027-02-23T12:05:12.666555+00:00 14480007386,164309631283457.66,ih38Q rOkfe6y3iTNP8m,2027-02-23T12:05:12.666560+00:00 [cephuser@ceph-hsm-71-8j01oc-node6 ~]$ Version-Release number of selected component (if applicable): ceph version 18.2.1-191.el9cp How reproducible: always Steps to Reproduce: 1.deploy rhcs7.1 ceph cluster with rgw daemons 2.upload parquet object using aws-cli 3.execute the query "select ( not ( 'abc' + 'def' ) ) from s3object;" Actual results: rgw going down executing query Expected results: rgw should not crash Additional info: automation fail log: http://magna002.ceph.redhat.com/cephci-jenkins/hsm/s3select_parquet_and_tfa/cephci-run-8J01OC/test_s3select_with_depth2_queries_on_parquet_objects_0.log
the crash happened during the parsing stage, it happened upon Parquet-flow and not CSV-flow (for the same statement) that is because the Parquet flow is parsing twice (wrongly) the SQL statement(AST creation), this may cause unexpected behavior.
https://github.com/ceph/ceph/pull/57944
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:10216
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days