+++ This bug was initially created as a clone of Bug #2256969 +++ +++ This bug was initially created as a clone of Bug #2256967 +++ Description of problem: RGW lacks support for new content checksums, and, in particular, two new forms of AWS STREAMING request formats. This bz tracks support for the new STREAMING-AWS4-HMAC-SHA256-PAYLOAD STREAMING-UNSIGNED-PAYLOAD-TRAILER forms, which introduce new aws-chunked behavior, which cause RGW to fail otherwise valid requests --- Additional comment from RHEL Program Management on 2024-01-05 18:31:06 UTC --- Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity. --- Additional comment from Madhavi Kasturi on 2024-01-17 00:53:08 UTC --- Hi Matt, Could you please share a reproducer to verify the fix. Thanks, Madhavi --- Additional comment from errata-xmlrpc on 2024-01-17 04:34:37 UTC --- This bug has been added to advisory RHBA-2024:126401 by Thomas Serlin (tserlin) --- Additional comment from errata-xmlrpc on 2024-01-17 04:34:52 UTC --- Bug report changed to ON_QA status by Errata System. A QE request has been submitted for advisory RHBA-2024:126401-01 https://errata.engineering.redhat.com/advisory/126401 --- Additional comment from Vidushi Mishra on 2024-02-06 06:30:53 UTC --- Hi Matt and Casey, Please ack if the tests mentioned in https://github.com/ceph/s3-tests/pull/540/files would hold good for verification for 7.0z1. - test_streaming_upload_checksum_crc32 - test_streaming_upload_checksum_crc32c - test_streaming_upload_checksum_sha1 - test_streaming_upload_checksum_sha256 Thanks, Vidushi --- Additional comment from on 2024-02-07 03:58:33 UTC --- QE, please use the build that was generated today. I just updated the Fixed in Version. Thanks, Thomas --- Additional comment from Tejas on 2024-02-14 14:18:31 UTC --- Hi Casey , I am seeing failures while running the new s3tests added as part of this PR : https://github.com/ceph/s3-tests/pull/540/files Attached the log, pls take a look --- Additional comment from Casey Bodley on 2024-02-14 15:06:42 UTC --- (In reply to Tejas from comment #6) > Created attachment 2016703 [details] > run log > > Hi Casey , I am seeing failures while running the new s3tests added as part > of this PR : https://github.com/ceph/s3-tests/pull/540/files > Attached the log, pls take a look lets not use those s3tests to validate these sigv4 streaming changes. they test checksum functionality that isn't implemented here. there's ongoing work upstream in https://github.com/ceph/ceph/pull/55076 to fully support the checksum features Matt's java tests should be used for test coverage of STREAMING-AWS4-HMAC-SHA256-PAYLOAD and STREAMING-UNSIGNED-PAYLOAD-TRAILER --- Additional comment from Tejas on 2024-02-15 05:31:32 UTC --- hi Casey, Matt, Can you please let us know on how to run the java tests that Matt has provided? --- Additional comment from Matt Benjamin (redhat) on 2024-02-20 17:04:47 UTC --- instructions for running the java test suite: 0. ensure openjdk17 or higher equivalent,and maven, are installed 1. from ceph/main checkout, in ./qa/workunits/rgw/jcksum (i.e., cd to that directory) (or a snapshot of that directory) 2. run tests for http and https: 2.1 $ export RGW_HTTP_ENDPOINT_URL=http://<hostname>::<port> 2.2 $ ./mvnw clean package 2.3 $ ./mvnw test -Dtest=PutObjects (should print some output and report success 2.4 $ export RGW_HTTP_ENDPOINT_URL=https://<hostname>::<ssl port> (repeat steps 2.2 and 2.3) thanks! Matt --- Additional comment from Tejas on 2024-02-21 08:49:36 UTC --- hi Matt , I am seeing a Java error with the steps you mentioned. the './mvnw clean package' command executed correctly, but seeing an error with test run. Please let me know if there is anything Im missing. System details : rgw node 10.0.209.166 (root/passwd) root@ceph-ck-mdlog-0l8bb9-node5 jcksum]# ./mvnw test -Dtest=PutObjects Warning: JAVA_HOME environment variable is not set. [INFO] Scanning for projects... [INFO] [INFO] ---------------------------< io.ceph:jcksum >--------------------------- [INFO] Building jcksum 1.0 [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- resources:3.3.1:resources (default-resources) @ jcksum --- [INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/main/resources [INFO] [INFO] --- compiler:3.11.0:compile (default-compile) @ jcksum --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- resources:3.3.1:testResources (default-testResources) @ jcksum --- [INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/test/resources [INFO] [INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ jcksum --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- surefire:3.2.3:test (default-test) @ jcksum --- [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running io.ceph.jcksum.PutObjects PutObjects.java: starting test run: AccessKey=0555b35654ad1656d804 SecretKey=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q== EndpointUrl=http://ceph-ck-mdlog-0l8bb9-node5:80 [main] WARN software.amazon.awssdk.http.apache.ApacheHttpClient - SSL Certificate verification is disabled. This is not a safe setting and should only be used for testing. DEBUG: Environment Variables Read DEBUG: S3 Client Initialized DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generated Big Files [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.654 s [INFO] Finished at: 2024-02-21T03:46:14-05:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.3:test (default-test) on project jcksum: [ERROR] [ERROR] Please refer to /root/ceph/qa/workunits/rgw/jcksum/target/surefire-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. [ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240221034613194_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-02-21T03-46-13_099-jvmRun1' 'surefire-20240221034613194_1tmp' 'surefire_0-20240221034613194_2tmp' [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240221034613194_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-02-21T03-46-13_099-jvmRun1' 'surefire-20240221034613194_1tmp' 'surefire_0-20240221034613194_2tmp' [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1240) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1089) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:905) [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162) [ERROR] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73) [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118) [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261) [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173) [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101) [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906) [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283) [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:206) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:568) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:568) [ERROR] at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:52) [ERROR] at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:161) [ERROR] at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:73) [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException --- Additional comment from Tejas on 2024-02-26 05:20:38 UTC --- Hi Matt, We need help with the verification of this BZ. Moving it back to Assigned. --- Additional comment from Matt Benjamin (redhat) on 2024-03-01 20:23:41 UTC --- Hi Folks, I am logged into 10.0.209.166, and do not see any of the maven environemnt present. Matt --- Additional comment from Tejas on 2024-03-02 03:24:53 UTC --- Hi Matt, Please find a new env at 10.0.211.233 (root/passwd), I am observing the same issue. --- Additional comment from Matt Benjamin (redhat) on 2024-03-02 15:19:59 UTC --- [mbenjamin@fedora ~]$ grep host ~/.s3cfg host_base = 10.0.211.233:80 host_bucket = %(bucket) [mbenjamin@fedora ~]$ s3cmd -c ~/.s3cfg-tejas ls ERROR: S3 error: 403 (InvalidAccessKeyId) [mbenjamin@fedora ~]$ env | grep ACC AWS_SECRET_ACCESS_KEY=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q== AWS_ACCESS_KEY_ID=0555b35654ad1656d804 [root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]# radosgw-admin user list [ "hsm", "tenant0$elvas.760", "repuser" ] radosgw-admin user create --uid=matt --secret='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==' --display-name="Admin User" --access-key='0555b35654ad1656d804' --system [mbenjamin@fedora ~]$ s3cmd -c ~/.s3cfg-tejas mb s3://buck1 Bucket 's3://buck1/' created From PutObjects.java: void readEnvironmentVars() { jcksum.access_key = get_envvar("AWS_ACCESS_KEY_ID", "0555b35654ad1656d804"); jcksum.secret_key = get_envvar("AWS_SECRET_ACCESS_KEY", "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT\ 3Q=="); jcksum.http_endpoint = get_envvar("RGW_HTTP_ENDPOINT_URL", ""); } /* readEnvironmentVArs */ (ie., this script requires each of those envvars to be exported, so in a remote root shell with cwd /root/ceph/qa/workunits/rgw/jcksum) export AWS_SECRET_ACCESS_KEY=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q== export AWS_ACCESS_KEY_ID=0555b35654ad1656d804 export RGW_HTTP_ENDPOINT_URL='http://localhost:80' [root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]# ./mvnw test -Dtest=PutObjects Warning: JAVA_HOME environment variable is not set. [INFO] Scanning for projects... [INFO] [INFO] ---------------------------< io.ceph:jcksum >--------------------------- [INFO] Building jcksum 1.0 [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- resources:3.3.1:resources (default-resources) @ jcksum --- [INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/main/resources [INFO] [INFO] --- compiler:3.11.0:compile (default-compile) @ jcksum --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- resources:3.3.1:testResources (default-testResources) @ jcksum --- [INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/test/resources [INFO] [INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ jcksum --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- surefire:3.2.3:test (default-test) @ jcksum --- [INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running io.ceph.jcksum.PutObjects PutObjects.java: starting test run: AccessKey=0555b35654ad1656d804 SecretKey=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q== EndpointUrl=http://localhost:80 [main] WARN software.amazon.awssdk.http.apache.ApacheHttpClient - SSL Certificate verification is disabled. This is not a safe setting and should only be used for testing. DEBUG: Environment Variables Read DEBUG: S3 Client Initialized DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generated Big Files HeadBucketResponse() sheik is ready DEBUG: Test Bucket Created putObjectFromFileNoCksum called with file-0b putObjectFromFileNoCksum called with file-8b putObjectFromFileNoCksum called with file-200b putObjectFromFileNoCksum called with file-21983b putObjectFromFileNoCksum called with file-5519b putObjectFromFileNoCksum called with file-204329b putObjectFromFileNoCksum called with file-256k putObjectFromFileNoCksum called with file-1m putObjectFromFileNoCksum called with file-1038757b mpuObjectFromFileNoCksum called with file-5m mpuObjectFromFileNoCksum called with file-10m mpuObjectFromFileNoCksum called with file-100m arg: file-0b arg: file-8b arg: file-200b arg: file-21983b arg: file-5519b arg: file-204329b arg: file-256k arg: file-1m arg: file-1038757b putObjectFromFileCksum called with file-0b (Service: S3, Status Code: 403, Request ID: tx00000b52e43b38c07336b-0065e34326-8679-primary) [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 8.152 s [INFO] Finished at: 2024-03-02T10:17:58-05:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.3:test (default-test) on project jcksum: [ERROR] [ERROR] Please refer to /root/ceph/qa/workunits/rgw/jcksum/target/surefire-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. [ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240302101751477_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-03-02T10-17-51_403-jvmRun1' 'surefire-20240302101751477_1tmp' 'surefire_0-20240302101751477_2tmp' [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] Crashed tests: [ERROR] io.ceph.jcksum.PutObjects [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240302101751477_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-03-02T10-17-51_403-jvmRun1' 'surefire-20240302101751477_1tmp' 'surefire_0-20240302101751477_2tmp' [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] Crashed tests: [ERROR] io.ceph.jcksum.PutObjects [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1240) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1089) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:905) [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162) [ERROR] at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73) [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118) [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261) [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173) [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101) [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906) [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283) [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:206) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:568) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:568) [ERROR] at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:52) [ERROR] at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:161) [ERROR] at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:73) [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException --- Additional comment from Matt Benjamin (redhat) on 2024-03-02 15:40:54 UTC --- ok, so that's a fail. the PutObjects test successfully uploaded and verified the following objects WITHOUT checksum: putObjectFromFileNoCksum called with file-0b putObjectFromFileNoCksum called with file-8b putObjectFromFileNoCksum called with file-200b putObjectFromFileNoCksum called with file-21983b putObjectFromFileNoCksum called with file-5519b putObjectFromFileNoCksum called with file-204329b putObjectFromFileNoCksum called with file-256k putObjectFromFileNoCksum called with file-1m putObjectFromFileNoCksum called with file-1038757b mpuObjectFromFileNoCksum called with file-5m mpuObjectFromFileNoCksum called with file-10m mpuObjectFromFileNoCksum called with file-100m However, rgw is returning a 403 in the first test with a checksum requested--which happens to be file-0b. When I change the test to omit only the 0-length file test, I get the following result: [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running io.ceph.jcksum.PutObjects PutObjects.java: starting test run: AccessKey=0555b35654ad1656d804 SecretKey=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q== EndpointUrl=http://localhost:80 [main] WARN software.amazon.awssdk.http.apache.ApacheHttpClient - SSL Certificate verification is disabled. This is not a safe setting and should only be used for testing. DEBUG: Environment Variables Read DEBUG: S3 Client Initialized DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generating File DEBUG: File Generated DEBUG: Generated Big Files HeadBucketResponse() sheik is ready DEBUG: Test Bucket Created putObjectFromFileNoCksum called with file-8b putObjectFromFileNoCksum called with file-8b putObjectFromFileNoCksum called with file-200b putObjectFromFileNoCksum called with file-21983b putObjectFromFileNoCksum called with file-5519b putObjectFromFileNoCksum called with file-204329b putObjectFromFileNoCksum called with file-256k putObjectFromFileNoCksum called with file-1m putObjectFromFileNoCksum called with file-1038757b mpuObjectFromFileNoCksum called with file-5m mpuObjectFromFileNoCksum called with file-10m mpuObjectFromFileNoCksum called with file-100m arg: file-8b arg: file-8b arg: file-200b arg: file-21983b arg: file-5519b arg: file-204329b arg: file-256k arg: file-1m arg: file-1038757b putObjectFromFileCksum called with file-8b PutObjectResponse putObjectFromFileCksum called with file-8b PutObjectResponse putObjectFromFileCksum called with file-200b PutObjectResponse putObjectFromFileCksum called with file-21983b PutObjectResponse putObjectFromFileCksum called with file-5519b PutObjectResponse putObjectFromFileCksum called with file-204329b PutObjectResponse putObjectFromFileCksum called with file-256k PutObjectResponse putObjectFromFileCksum called with file-1m PutObjectResponse putObjectFromFileCksum called with file-1038757b PutObjectResponse mpuObjectFromFileCksum called with file-5m mpuObjectFromFileCksum called with file-10m mpuObjectFromFileCksum called with file-100m [INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.180 s -- in io.ceph.jcksum.PutObjects [INFO] [INFO] Results: [INFO] [INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10.933 s [INFO] Finished at: 2024-03-02T10:31:44-05:00 [INFO] ------------------------------------------------------------------------ [root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]# Which means that checksum support is succeeding for all the other cases. adding support for a zero-length file w/checksum was one of the last things added to the upstream PR. The commits that fixed this upstream seem to be present, but obviously the zero-length upload w/checksum is failing. I will work on the 0-byte case on the 7.0z1 baseline this weekend, but, worst case, we could release the current code with a release note. There are affected workloads that we care about, but the are specialized. Matt --- Additional comment from Matt Benjamin (redhat) on 2024-03-02 15:43:54 UTC --- just fyi: [root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]# git diff diff --git a/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java b/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java index 8ac68a27e3c..c7b54ac0585 100644 --- a/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java +++ b/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java @@ -42,7 +42,7 @@ public class jcksum { /* files containing test data of the corresponding names/sizes */ public static Stream<String> inputFileNames() { return Stream.of( - "file-0b", + "file-8b", "file-8b", "file-200b", "file-21983b", --- Additional comment from Matt Benjamin (redhat) on 2024-03-02 17:32:35 UTC --- update: I've repeated the above issue in a debug environment, will update further when I have an RCA. Matt --- Additional comment from Matt Benjamin (redhat) on 2024-03-05 00:32:52 UTC --- I've proposed a candidate fix for this issue, but my proposal would be to move this bz to 7.0z2. Matt --- Additional comment from Madhavi Kasturi on 2024-03-06 06:07:20 UTC --- Based on email conversation, ack from QE to move the BZ out of 7.0z1 --- Additional comment from Rivka Pollack on 2024-03-06 07:43:24 UTC --- Matt, do we want this as a known issue for 7.0z1? --- Additional comment from errata-xmlrpc on 2024-03-06 15:13:34 UTC --- This bug has been dropped from advisory RHBA-2024:126401 by Ken Dreyer (kdreyer) --- Additional comment from Matt Benjamin (redhat) on 2024-03-06 17:33:07 UTC --- The release note might be in the nature of, applications using some S3 client SDKs could experience an unexpected 403 when uploading a zero-length object if an external checksum is requested. matt
*** This bug has been marked as a duplicate of bug 2256967 ***