Bug 2268348

Summary:	[rgw] uploads with java and golang SDKs can fail when checksum is requested
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Matt Benjamin (redhat) <mbenjamin>
Component:	RGW	Assignee:	Matt Benjamin (redhat) <mbenjamin>
Status:	CLOSED DUPLICATE	QA Contact:	Tejas <tchandra>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.1	CC:	bkunal, cbodley, ceph-eng-bugs, cephqe-warriors, mkasturi, rpollack, tchandra, tserlin, vimishra
Target Milestone:	---
Target Release:	7.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2256969	Environment:
Last Closed:	2024-03-26 15:29:00 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2256967
Bug Blocks:	2260311

Description Matt Benjamin (redhat) 2024-03-06 23:58:27 UTC

+++ This bug was initially created as a clone of Bug #2256969 +++

+++ This bug was initially created as a clone of Bug #2256967 +++

Description of problem:
RGW lacks support for new content checksums, and, in particular, two new forms of AWS STREAMING request formats.

This bz tracks support for the new STREAMING-AWS4-HMAC-SHA256-PAYLOAD STREAMING-UNSIGNED-PAYLOAD-TRAILER forms, which introduce new aws-chunked behavior, which cause RGW to fail otherwise valid requests

--- Additional comment from RHEL Program Management on 2024-01-05 18:31:06 UTC ---

Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

--- Additional comment from Madhavi Kasturi on 2024-01-17 00:53:08 UTC ---

Hi Matt,

Could you please share a reproducer to verify the fix.

Thanks,
Madhavi

--- Additional comment from errata-xmlrpc on 2024-01-17 04:34:37 UTC ---

This bug has been added to advisory RHBA-2024:126401 by Thomas Serlin (tserlin)

--- Additional comment from errata-xmlrpc on 2024-01-17 04:34:52 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2024:126401-01
https://errata.engineering.redhat.com/advisory/126401

--- Additional comment from Vidushi Mishra on 2024-02-06 06:30:53 UTC ---

Hi Matt and Casey,

Please ack if the tests mentioned in https://github.com/ceph/s3-tests/pull/540/files would hold good for verification for 7.0z1.

-  test_streaming_upload_checksum_crc32
-  test_streaming_upload_checksum_crc32c
-  test_streaming_upload_checksum_sha1
-  test_streaming_upload_checksum_sha256

Thanks,
Vidushi

--- Additional comment from  on 2024-02-07 03:58:33 UTC ---

QE, please use the build that was generated today. I just updated the Fixed in Version.

Thanks,

Thomas

--- Additional comment from Tejas on 2024-02-14 14:18:31 UTC ---

Hi Casey , I am seeing failures while running the new s3tests added as part of this PR : https://github.com/ceph/s3-tests/pull/540/files
Attached the log, pls take a look

--- Additional comment from Casey Bodley on 2024-02-14 15:06:42 UTC ---

(In reply to Tejas from comment #6)
> Created attachment 2016703 [details]
> run log
> 
> Hi Casey , I am seeing failures while running the new s3tests added as part
> of this PR : https://github.com/ceph/s3-tests/pull/540/files
> Attached the log, pls take a look

lets not use those s3tests to validate these sigv4 streaming changes. they test checksum functionality that isn't implemented here. there's ongoing work upstream in https://github.com/ceph/ceph/pull/55076 to fully support the checksum features

Matt's java tests should be used for test coverage of STREAMING-AWS4-HMAC-SHA256-PAYLOAD and STREAMING-UNSIGNED-PAYLOAD-TRAILER

--- Additional comment from Tejas on 2024-02-15 05:31:32 UTC ---

hi Casey, Matt,

Can you please let us know on how to run the java tests that Matt has provided?

--- Additional comment from Matt Benjamin (redhat) on 2024-02-20 17:04:47 UTC ---

instructions for running the java test suite:

0. ensure openjdk17 or higher equivalent,and maven, are installed
1. from ceph/main checkout, in ./qa/workunits/rgw/jcksum (i.e., cd to that directory)
 (or a snapshot of that directory)

2. run tests for http and https:

2.1  $ export RGW_HTTP_ENDPOINT_URL=http://<hostname>::<port>
2.2  $ ./mvnw clean package
2.3  $ ./mvnw test -Dtest=PutObjects

(should print some output and report success

2.4 $ export RGW_HTTP_ENDPOINT_URL=https://<hostname>::<ssl port>

(repeat steps 2.2 and 2.3)


thanks!

Matt

--- Additional comment from Tejas on 2024-02-21 08:49:36 UTC ---

hi Matt ,

I am seeing a Java error with the steps you mentioned. the './mvnw clean package' command executed correctly, but seeing an error with test run.
Please let me know if there is anything Im missing.
System details : rgw node 10.0.209.166 (root/passwd)


root@ceph-ck-mdlog-0l8bb9-node5 jcksum]# ./mvnw test -Dtest=PutObjects
Warning: JAVA_HOME environment variable is not set.
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< io.ceph:jcksum >---------------------------
[INFO] Building jcksum 1.0
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @ jcksum ---
[INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/main/resources
[INFO]
[INFO] --- compiler:3.11.0:compile (default-compile) @ jcksum ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- resources:3.3.1:testResources (default-testResources) @ jcksum ---
[INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/test/resources
[INFO]
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ jcksum ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- surefire:3.2.3:test (default-test) @ jcksum ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.ceph.jcksum.PutObjects
PutObjects.java: starting test run:
	AccessKey=0555b35654ad1656d804
	SecretKey=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
	EndpointUrl=http://ceph-ck-mdlog-0l8bb9-node5:80
[main] WARN software.amazon.awssdk.http.apache.ApacheHttpClient - SSL Certificate verification is disabled. This is not a safe setting and should only be used for testing.
DEBUG: Environment Variables Read
DEBUG: S3 Client Initialized
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generated Big Files

[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.654 s
[INFO] Finished at: 2024-02-21T03:46:14-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.3:test (default-test) on project jcksum:
[ERROR]
[ERROR] Please refer to /root/ceph/qa/workunits/rgw/jcksum/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240221034613194_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-02-21T03-46-13_099-jvmRun1' 'surefire-20240221034613194_1tmp' 'surefire_0-20240221034613194_2tmp'
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240221034613194_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-02-21T03-46-13_099-jvmRun1' 'surefire-20240221034613194_1tmp' 'surefire_0-20240221034613194_2tmp'
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1240)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1089)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:905)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
[ERROR] 	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[ERROR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[ERROR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[ERROR] 	at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:52)
[ERROR] 	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:161)
[ERROR] 	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:73)
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

--- Additional comment from Tejas on 2024-02-26 05:20:38 UTC ---

Hi Matt,
 We need help with the verification of this BZ. Moving it back to Assigned.

--- Additional comment from Matt Benjamin (redhat) on 2024-03-01 20:23:41 UTC ---

Hi Folks,

I am logged into 10.0.209.166, and do not see any of the maven environemnt present.

Matt

--- Additional comment from Tejas on 2024-03-02 03:24:53 UTC ---

Hi Matt,
Please find a new env at 10.0.211.233 (root/passwd), I am observing the same issue.

--- Additional comment from Matt Benjamin (redhat) on 2024-03-02 15:19:59 UTC ---

[mbenjamin@fedora ~]$ grep host ~/.s3cfg
host_base = 10.0.211.233:80
host_bucket = %(bucket)

[mbenjamin@fedora ~]$ s3cmd -c ~/.s3cfg-tejas ls
ERROR: S3 error: 403 (InvalidAccessKeyId)

[mbenjamin@fedora ~]$ env | grep ACC
AWS_SECRET_ACCESS_KEY=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
AWS_ACCESS_KEY_ID=0555b35654ad1656d804

[root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]# radosgw-admin user list
[
    "hsm",
    "tenant0$elvas.760",
    "repuser"
]

radosgw-admin user create --uid=matt --secret='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==' --display-name="Admin User" --access-key='0555b35654ad1656d804'  --system

[mbenjamin@fedora ~]$ s3cmd -c ~/.s3cfg-tejas mb s3://buck1
Bucket 's3://buck1/' created

From PutObjects.java:
  void readEnvironmentVars() {
    jcksum.access_key = get_envvar("AWS_ACCESS_KEY_ID", "0555b35654ad1656d804");
    jcksum.secret_key = get_envvar("AWS_SECRET_ACCESS_KEY", "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT\
3Q==");
    jcksum.http_endpoint = get_envvar("RGW_HTTP_ENDPOINT_URL", "");
  } /* readEnvironmentVArs */

(ie., this script requires each of those envvars to be exported, so in a remote root shell with cwd /root/ceph/qa/workunits/rgw/jcksum)

export AWS_SECRET_ACCESS_KEY=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
export AWS_ACCESS_KEY_ID=0555b35654ad1656d804
export RGW_HTTP_ENDPOINT_URL='http://localhost:80'

[root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]# ./mvnw test -Dtest=PutObjects
Warning: JAVA_HOME environment variable is not set.
[INFO] Scanning for projects...
[INFO] 
[INFO] ---------------------------< io.ceph:jcksum >---------------------------
[INFO] Building jcksum 1.0
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ jcksum ---
[INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/main/resources
[INFO] 
[INFO] --- compiler:3.11.0:compile (default-compile) @ jcksum ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- resources:3.3.1:testResources (default-testResources) @ jcksum ---
[INFO] skip non existing resourceDirectory /root/ceph/qa/workunits/rgw/jcksum/src/test/resources
[INFO] 
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ jcksum ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- surefire:3.2.3:test (default-test) @ jcksum ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.ceph.jcksum.PutObjects
PutObjects.java: starting test run:
	AccessKey=0555b35654ad1656d804
	SecretKey=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
	EndpointUrl=http://localhost:80
[main] WARN software.amazon.awssdk.http.apache.ApacheHttpClient - SSL Certificate verification is disabled. This is not a safe setting and should only be used for testing.
DEBUG: Environment Variables Read
DEBUG: S3 Client Initialized
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generated Big Files
HeadBucketResponse()
sheik is ready
DEBUG: Test Bucket Created
putObjectFromFileNoCksum called with file-0b
putObjectFromFileNoCksum called with file-8b
putObjectFromFileNoCksum called with file-200b
putObjectFromFileNoCksum called with file-21983b
putObjectFromFileNoCksum called with file-5519b
putObjectFromFileNoCksum called with file-204329b
putObjectFromFileNoCksum called with file-256k
putObjectFromFileNoCksum called with file-1m
putObjectFromFileNoCksum called with file-1038757b
mpuObjectFromFileNoCksum called with file-5m
mpuObjectFromFileNoCksum called with file-10m
mpuObjectFromFileNoCksum called with file-100m
arg: file-0b
arg: file-8b
arg: file-200b
arg: file-21983b
arg: file-5519b
arg: file-204329b
arg: file-256k
arg: file-1m
arg: file-1038757b
putObjectFromFileCksum called with file-0b
 (Service: S3, Status Code: 403, Request ID: tx00000b52e43b38c07336b-0065e34326-8679-primary)
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.152 s
[INFO] Finished at: 2024-03-02T10:17:58-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.3:test (default-test) on project jcksum: 
[ERROR] 
[ERROR] Please refer to /root/ceph/qa/workunits/rgw/jcksum/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240302101751477_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-03-02T10-17-51_403-jvmRun1' 'surefire-20240302101751477_1tmp' 'surefire_0-20240302101751477_2tmp'
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] io.ceph.jcksum.PutObjects
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd '/root/ceph/qa/workunits/rgw/jcksum' && '/usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el9.x86_64/bin/java' '-jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire/surefirebooter-20240302101751477_3.jar' '/root/ceph/qa/workunits/rgw/jcksum/target/surefire' '2024-03-02T10-17-51_403-jvmRun1' 'surefire-20240302101751477_1tmp' 'surefire_0-20240302101751477_2tmp'
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] Crashed tests:
[ERROR] io.ceph.jcksum.PutObjects
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:643)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1240)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1089)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:905)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:126)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:328)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:316)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:174)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:75)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:162)
[ERROR] 	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:159)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:105)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:73)
[ERROR] 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:53)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:118)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:261)
[ERROR] 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:173)
[ERROR] 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:101)
[ERROR] 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:906)
[ERROR] 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:283)
[ERROR] 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:206)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[ERROR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:283)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:226)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)
[ERROR] 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[ERROR] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[ERROR] 	at org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:52)
[ERROR] 	at org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:161)
[ERROR] 	at org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:73)
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

--- Additional comment from Matt Benjamin (redhat) on 2024-03-02 15:40:54 UTC ---

ok, so that's a fail.  the PutObjects test successfully uploaded and verified the following objects WITHOUT
checksum:

putObjectFromFileNoCksum called with file-0b
putObjectFromFileNoCksum called with file-8b
putObjectFromFileNoCksum called with file-200b
putObjectFromFileNoCksum called with file-21983b
putObjectFromFileNoCksum called with file-5519b
putObjectFromFileNoCksum called with file-204329b
putObjectFromFileNoCksum called with file-256k
putObjectFromFileNoCksum called with file-1m
putObjectFromFileNoCksum called with file-1038757b
mpuObjectFromFileNoCksum called with file-5m
mpuObjectFromFileNoCksum called with file-10m
mpuObjectFromFileNoCksum called with file-100m

However, rgw is returning a 403 in the first test with a checksum requested--which happens to be file-0b.  When
I change the test to omit only the 0-length file test, I get the following result:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.ceph.jcksum.PutObjects
PutObjects.java: starting test run:
	AccessKey=0555b35654ad1656d804
	SecretKey=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
	EndpointUrl=http://localhost:80
[main] WARN software.amazon.awssdk.http.apache.ApacheHttpClient - SSL Certificate verification is disabled. This is not a safe setting and should only be used for testing.
DEBUG: Environment Variables Read
DEBUG: S3 Client Initialized
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generating File
DEBUG: File Generated
DEBUG: Generated Big Files
HeadBucketResponse()
sheik is ready
DEBUG: Test Bucket Created
putObjectFromFileNoCksum called with file-8b
putObjectFromFileNoCksum called with file-8b
putObjectFromFileNoCksum called with file-200b
putObjectFromFileNoCksum called with file-21983b
putObjectFromFileNoCksum called with file-5519b
putObjectFromFileNoCksum called with file-204329b
putObjectFromFileNoCksum called with file-256k
putObjectFromFileNoCksum called with file-1m
putObjectFromFileNoCksum called with file-1038757b
mpuObjectFromFileNoCksum called with file-5m
mpuObjectFromFileNoCksum called with file-10m
mpuObjectFromFileNoCksum called with file-100m
arg: file-8b
arg: file-8b
arg: file-200b
arg: file-21983b
arg: file-5519b
arg: file-204329b
arg: file-256k
arg: file-1m
arg: file-1038757b
putObjectFromFileCksum called with file-8b
PutObjectResponse
putObjectFromFileCksum called with file-8b
PutObjectResponse
putObjectFromFileCksum called with file-200b
PutObjectResponse
putObjectFromFileCksum called with file-21983b
PutObjectResponse
putObjectFromFileCksum called with file-5519b
PutObjectResponse
putObjectFromFileCksum called with file-204329b
PutObjectResponse
putObjectFromFileCksum called with file-256k
PutObjectResponse
putObjectFromFileCksum called with file-1m
PutObjectResponse
putObjectFromFileCksum called with file-1038757b
PutObjectResponse
mpuObjectFromFileCksum called with file-5m
mpuObjectFromFileCksum called with file-10m
mpuObjectFromFileCksum called with file-100m
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.180 s -- in io.ceph.jcksum.PutObjects
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.933 s
[INFO] Finished at: 2024-03-02T10:31:44-05:00
[INFO] ------------------------------------------------------------------------
[root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]#

Which means that checksum support is succeeding for all the other cases.  adding support for a zero-length file w/checksum was one of the last things added to the upstream PR.  The commits that fixed this upstream seem to be present, but obviously the zero-length upload w/checksum is failing.

I will work on the 0-byte case on the 7.0z1 baseline this weekend, but, worst case, we could release the current code with a release note.  There are affected workloads that we care about, but the are specialized.

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-03-02 15:43:54 UTC ---

just fyi:

[root@ceph-pri-hsm-7-0z1-8trdy5-node5 jcksum]# git diff
diff --git a/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java b/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java
index 8ac68a27e3c..c7b54ac0585 100644
--- a/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java
+++ b/qa/workunits/rgw/jcksum/src/main/java/io/ceph/jcksum/jcksum.java
@@ -42,7 +42,7 @@ public class jcksum {
        /* files containing test data of the corresponding names/sizes */
        public static Stream<String> inputFileNames() {
            return Stream.of(
-          "file-0b",
+          "file-8b",
                        "file-8b",
                        "file-200b",
                        "file-21983b",

--- Additional comment from Matt Benjamin (redhat) on 2024-03-02 17:32:35 UTC ---

update:  I've repeated the above issue in a debug environment, will update further when I have an RCA.

Matt

--- Additional comment from Matt Benjamin (redhat) on 2024-03-05 00:32:52 UTC ---

I've proposed a candidate fix for this issue, but my proposal would be to move this bz to 7.0z2.

Matt

--- Additional comment from Madhavi Kasturi on 2024-03-06 06:07:20 UTC ---

Based on email conversation, ack from QE to move the BZ out of 7.0z1

--- Additional comment from Rivka Pollack on 2024-03-06 07:43:24 UTC ---

Matt, do we want this as a known issue for 7.0z1?

--- Additional comment from errata-xmlrpc on 2024-03-06 15:13:34 UTC ---

This bug has been dropped from advisory RHBA-2024:126401 by Ken Dreyer (kdreyer)

--- Additional comment from Matt Benjamin (redhat) on 2024-03-06 17:33:07 UTC ---

The release note might be in the nature of, applications using some S3 client SDKs could experience an unexpected 403 when uploading a zero-length object if an external checksum is requested.

matt

Comment 3 Matt Benjamin (redhat) 2024-03-26 15:29:00 UTC


*** This bug has been marked as a duplicate of bug 2256967 ***