1456060 – [RHCeph 2.3/10.2.7-21]: s3a hadoop 2.8 integration test ITestS3AFileContextURI - Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000000b6d-005928b901-104a-default

Bug 1456060 - [RHCeph 2.3/10.2.7-21]: s3a hadoop 2.8 integration test ITestS3AFileContextURI - Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000000b6d-005928b901-104a-default

Summary: [RHCeph 2.3/10.2.7-21]: s3a hadoop 2.8 integration test ITestS3AFileContextUR...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RGW
Sub Component:
Version:	2.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	2.4
Assignee:	Marcus Watts
QA Contact:	Vasu Kulkarni
Docs Contact:	Bara Ancincova
URL:
Whiteboard:
Depends On:	1498323
Blocks:	1473188 1473436 1479701
TreeView+	depends on / blocked

Reported:	2017-05-26 23:52 UTC by Vasu Kulkarni
Modified:	2017-10-19 20:12 UTC (History)
CC List:	16 users (show)
Fixed In Version:	RHEL: ceph-10.2.7-41.el7cp Ubuntu: ceph_10.2.7-40redhat1
Doc Type:	Bug Fix
Doc Text:	.Several AWS version 4 signature bugs are fixed This update fixes several Amazon Web Service (AWS) version 4 signature bugs.
Clone Of:
Clones:	1498323 (view as bug list)
Environment:
Last Closed:	2017-10-17 18:12:51 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	20501	0	None	None	None	2017-09-18 08:20:02 UTC
Red Hat Product Errata	RHBA-2017:2903	0	normal	SHIPPED_LIVE	Red Hat Ceph Storage 2.4 enhancement and bug fix update	2017-10-17 22:12:30 UTC

Description Vasu Kulkarni 2017-05-26 23:52:39 UTC

Description of problem:


In hadoop 2.8.0, test structure has changed and many moved to integration tests, while trying to run with 2.8.0, 5 of 325 failed 

Tests in error:
  ITestS3AFileContextURI>FileContextURIBase.testDeleteDirectory:404 Â» AccessDenied

ITestS3AFileContextURI>FileContextURIBase.testListStatus:528 Â» AccessDenied us...

ITestS3AFileContextURI>FileContextURIBase.testCreateFile:103 Â» AccessDenied us...

ITestS3AFileContextURI>FileContextURIBase.testCreateDirectory:231 Â» AccessDenied

ITestS3AAWSCredentialsProvider.testAnonymousProvider:131 Â» FileNotFound Bucket..


2017-05-26T19:38:17.310 INFO:teuthology.orchestra.run.pluto004.stdout:Tests in error:
2017-05-26T19:38:17.310 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testDeleteDirectory:404 Â» AccessDenied
2017-05-26T19:38:17.311 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testListStatus:528 Â» AccessDenied us...
2017-05-26T19:38:17.311 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testCreateFile:103 Â» AccessDenied us...
2017-05-26T19:38:17.311 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testCreateDirectory:231 Â» AccessDenied
2017-05-26T19:38:17.312 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AAWSCredentialsProvider.testAnonymousProvider:131 Â» FileNotFound Bucket...
2017-05-26T19:38:17.312 INFO:teuthology.orchestra.run.pluto004.stdout:
2017-05-26T19:38:17.312 INFO:teuthology.orchestra.run.pluto004.stdout:Tests 

run: 323, Failures: 0, Errors: 5, Skipped: 161


2017-05-26T19:17:03.269 INFO:teuthology.orchestra.run.pluto004.stdout: T E S T S
2017-05-26T19:17:03.269 INFO:teuthology.orchestra.run.pluto004.stdout:-------------------------------------------------------
2017-05-26T19:17:03.536 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
2017-05-26T19:17:09.648 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.106 sec - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
2017-05-26T19:17:09.649 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
2017-05-26T19:17:09.680 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.027 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
2017-05-26T19:17:09.680 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
2017-05-26T19:17:09.788 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.104 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
2017-05-26T19:17:09.789 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
2017-05-26T19:17:09.884 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.091 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
2017-05-26T19:17:09.884 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
2017-05-26T19:17:10.380 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.496 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
2017-05-26T19:17:10.381 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
2017-05-26T19:17:10.396 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.014 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
2017-05-26T19:17:10.396 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
2017-05-26T19:17:49.559 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 60, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 39.162 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
2017-05-26T19:17:49.560 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
2017-05-26T19:17:49.568 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
2017-05-26T19:17:49.568 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
2017-05-26T19:37:50.133 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 17, Failures: 0, Errors: 4, Skipped: 1, Time elapsed: 1,200.561 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
2017-05-26T19:37:50.134 INFO:teuthology.orchestra.run.pluto004.stdout:testDeleteDirectory(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI)  Time elapsed: 298.849 sec  <<< ERROR!
2017-05-26T19:37:50.134 INFO:teuthology.orchestra.run.pluto004.stdout:java.nio.file.AccessDeniedException: user/ubuntu/target/test/data/testContextURI/deleteTest/test Dir/: getFileStatus on user/ubuntu/target/test/data/testContextURI/deleteTest/test Dir/: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000000b6d-005928b901-104a-default), S3 Extended Request ID: 104a-default-default
2017-05-26T19:37:50.134 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1635)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:117)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1177)
2017-05-26T19:37:50.136 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1173)
2017-05-26T19:37:50.136 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
2017-05-26T19:37:50.136 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1173)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1638)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContextURIBase.testDeleteDirectory(FileContextURIBase.java:404)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at java.lang.reflect.Method.invoke(Method.java:498)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
2017-05-26T19:37:50.139 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2017-05-26T19:37:50.139 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
2017-05-26T19:37:50.139 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
2017-05-26T19:37:50.142 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
2017-05-26T19:37:50.142 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
2017-05-26T19:37:50.142 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
2017-05-26T19:37:50.144 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
2017-05-26T19:37:50.144 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
2017-05-26T19:37:50.144 INFO:teuthology.orchestra.run.pluto004.stdout:Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000000b6d-005928b901-104a-default)


Full logs:
http://magna002.ceph.redhat.com/vasu-2017-05-26_19:02:14-smoke-jewel---basic-multi/269384/teuthology.log

Comment 24 John Poelstra 2017-05-31 15:27:11 UTC

development believes there is not a bug 

product needs more time to review

QE defers to product

Comment 36 Vasu Kulkarni 2017-07-31 19:13:39 UTC

Sorry I held the testbed for quite some time before moving on, Since your build includes AWS signature changes I believe it is fixing this issue,  Can you talk to Thomas and generate a puddle build so that I can verify with new build and close this rather than try with old build(it is consistent with old build so there is no point in recreating it).

Also thanks for the landsat changes, I will update the scripts which will make everything green.

Comment 38 Kyle Bader 2017-08-01 20:16:34 UTC

I've been able to reproduce this with:

ceph version 10.2.7-28.0.TEST.bz1470301.el7cp (8950adbd3b787f57ba50a41d5289dee56dc02e44)

RGW logs (debug ms = 1, debug rgw = 20)

https://gist.github.com/mmgaggle/dc706e8a592e10e7831e7fb14ca52870

Comment 41 Vasu Kulkarni 2017-08-02 16:59:31 UTC

Doesn't seem to be fixed at, build used is ceph-radosgw-1:10.2.7-28.0.TEST.bz1476865.el7cp.x86_64 as provided by Thomas, Not sure if it has all the fixes.

http://magna002.ceph.redhat.com/vasu-2017-08-02_11:48:46-smoke-jewel---basic-multi/271165/teuthology.log

https://paste.fedoraproject.org/paste/RUW-9NnYuUTlDKlQkDhOtQ/raw

Comment 57 Vasu Kulkarni 2017-08-11 17:20:53 UTC

Thomas,

I cant use the tar builds, mind generating dev-compose or providing a repo for the test build.  Thanks

Comment 66 Vasu Kulkarni 2017-08-17 00:35:20 UTC

Other than the bucket-landsat issue (for which I need to fix the script to include what marcus has suggested in comment 34) all tests have passed. 

Tests run: 323, Failures: 0, Errors: 1, Skipped: 161

I will close this once I verify this again in async build, since this is a test build run.

http://magna002.ceph.redhat.com/vasu-2017-08-16_19:58:32-smoke-jewel---basic-multi/272322/teuthology.log

Comment 71 Tamil 2017-09-19 00:49:25 UTC

hi Vasu, can you please verify this bug?

Comment 74 Vasu Kulkarni 2017-09-21 17:53:24 UTC

I can actually close this issue, since I see this fixed in 2.8.0 run (ignoring the bucket-landsat issue)

http://magna002.ceph.redhat.com/vasu-2017-09-21_12:21:23-smoke-jewel---basic-multi/274636/teuthology.log

2017-09-21T12:54:00.698 INFO:teuthology.orchestra.run.pluto007.stdout:Tests run: 323, Failures: 0, Errors: 1, Skipped: 161


Also I ran on 2.8.1 just to check if anything fails, but It was clean in my run.

http://magna002.ceph.redhat.com/vasu-2017-09-21_12:21:23-smoke-jewel---basic-multi/274637/teuthology.log

2017-09-21T13:21:56.425 INFO:teuthology.orchestra.run.pluto007.stdout:Tests run: 323, Failures: 0, Errors: 1, Skipped: 161

Comment 87 errata-xmlrpc 2017-10-17 18:12:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2903

Note You need to log in before you can comment on or make changes to this bug.