Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1456060 - [RHCeph 2.3/10.2.7-21]: s3a hadoop 2.8 integration test ITestS3AFileContextURI - Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000000b6d-005928b901-104a-default
[RHCeph 2.3/10.2.7-21]: s3a hadoop 2.8 integration test ITestS3AFileContextUR...
Status: CLOSED ERRATA
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
2.3
Unspecified Unspecified
medium Severity medium
: rc
: 2.4
Assigned To: Marcus Watts
Vasu Kulkarni
Bara Ancincova
:
Depends On: 1498323
Blocks: 1473188 1473436 1479701
  Show dependency tree / graph
 
Reported: 2017-05-26 19:52 EDT by Vasu Kulkarni
Modified: 2017-10-19 16:12 EDT (History)
16 users (show)

See Also:
Fixed In Version: RHEL: ceph-10.2.7-41.el7cp Ubuntu: ceph_10.2.7-40redhat1
Doc Type: Bug Fix
Doc Text:
.Several AWS version 4 signature bugs are fixed This update fixes several Amazon Web Service (AWS) version 4 signature bugs.
Story Points: ---
Clone Of:
: 1498323 (view as bug list)
Environment:
Last Closed: 2017-10-17 14:12:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 20501 None None None 2017-09-18 04:20 EDT
Red Hat Product Errata RHBA-2017:2903 normal SHIPPED_LIVE Red Hat Ceph Storage 2.4 enhancement and bug fix update 2017-10-17 18:12:30 EDT

  None (edit)
Description Vasu Kulkarni 2017-05-26 19:52:39 EDT 
Description of problem:


In hadoop 2.8.0, test structure has changed and many moved to integration tests, while trying to run with 2.8.0, 5 of 325 failed 

Tests in error:
  ITestS3AFileContextURI>FileContextURIBase.testDeleteDirectory:404 » AccessDenied

ITestS3AFileContextURI>FileContextURIBase.testListStatus:528 » AccessDenied us...

ITestS3AFileContextURI>FileContextURIBase.testCreateFile:103 » AccessDenied us...

ITestS3AFileContextURI>FileContextURIBase.testCreateDirectory:231 » AccessDenied

ITestS3AAWSCredentialsProvider.testAnonymousProvider:131 » FileNotFound Bucket..


2017-05-26T19:38:17.310 INFO:teuthology.orchestra.run.pluto004.stdout:Tests in error:
2017-05-26T19:38:17.310 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testDeleteDirectory:404 » AccessDenied
2017-05-26T19:38:17.311 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testListStatus:528 » AccessDenied us...
2017-05-26T19:38:17.311 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testCreateFile:103 » AccessDenied us...
2017-05-26T19:38:17.311 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AFileContextURI>FileContextURIBase.testCreateDirectory:231 » AccessDenied
2017-05-26T19:38:17.312 INFO:teuthology.orchestra.run.pluto004.stdout:  ITestS3AAWSCredentialsProvider.testAnonymousProvider:131 » FileNotFound Bucket...
2017-05-26T19:38:17.312 INFO:teuthology.orchestra.run.pluto004.stdout:
2017-05-26T19:38:17.312 INFO:teuthology.orchestra.run.pluto004.stdout:Tests 

run: 323, Failures: 0, Errors: 5, Skipped: 161


2017-05-26T19:17:03.269 INFO:teuthology.orchestra.run.pluto004.stdout: T E S T S
2017-05-26T19:17:03.269 INFO:teuthology.orchestra.run.pluto004.stdout:-------------------------------------------------------
2017-05-26T19:17:03.536 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
2017-05-26T19:17:09.648 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.106 sec - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
2017-05-26T19:17:09.649 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
2017-05-26T19:17:09.680 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.027 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
2017-05-26T19:17:09.680 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
2017-05-26T19:17:09.788 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.104 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
2017-05-26T19:17:09.789 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
2017-05-26T19:17:09.884 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.091 sec - in org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
2017-05-26T19:17:09.884 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
2017-05-26T19:17:10.380 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.496 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A
2017-05-26T19:17:10.381 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
2017-05-26T19:17:10.396 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.014 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
2017-05-26T19:17:10.396 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
2017-05-26T19:17:49.559 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 60, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 39.162 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
2017-05-26T19:17:49.560 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
2017-05-26T19:17:49.568 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
2017-05-26T19:17:49.568 INFO:teuthology.orchestra.run.pluto004.stdout:Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
2017-05-26T19:37:50.133 INFO:teuthology.orchestra.run.pluto004.stdout:Tests run: 17, Failures: 0, Errors: 4, Skipped: 1, Time elapsed: 1,200.561 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
2017-05-26T19:37:50.134 INFO:teuthology.orchestra.run.pluto004.stdout:testDeleteDirectory(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI)  Time elapsed: 298.849 sec  <<< ERROR!
2017-05-26T19:37:50.134 INFO:teuthology.orchestra.run.pluto004.stdout:java.nio.file.AccessDeniedException: user/ubuntu/target/test/data/testContextURI/deleteTest/test Dir/: getFileStatus on user/ubuntu/target/test/data/testContextURI/deleteTest/test Dir/: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000000b6d-005928b901-104a-default), S3 Extended Request ID: 104a-default-default
2017-05-26T19:37:50.134 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1635)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:117)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
2017-05-26T19:37:50.135 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1177)
2017-05-26T19:37:50.136 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1173)
2017-05-26T19:37:50.136 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
2017-05-26T19:37:50.136 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1173)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1638)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.hadoop.fs.FileContextURIBase.testDeleteDirectory(FileContextURIBase.java:404)
2017-05-26T19:37:50.137 INFO:teuthology.orchestra.run.pluto004.stdout:	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at java.lang.reflect.Method.invoke(Method.java:498)
2017-05-26T19:37:50.138 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
2017-05-26T19:37:50.139 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2017-05-26T19:37:50.139 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
2017-05-26T19:37:50.139 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
2017-05-26T19:37:50.140 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
2017-05-26T19:37:50.141 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
2017-05-26T19:37:50.142 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
2017-05-26T19:37:50.142 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
2017-05-26T19:37:50.142 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
2017-05-26T19:37:50.143 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
2017-05-26T19:37:50.144 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
2017-05-26T19:37:50.144 INFO:teuthology.orchestra.run.pluto004.stdout:	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
2017-05-26T19:37:50.144 INFO:teuthology.orchestra.run.pluto004.stdout:Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx000000000000000000b6d-005928b901-104a-default)


Full logs:
http://magna002.ceph.redhat.com/vasu-2017-05-26_19:02:14-smoke-jewel---basic-multi/269384/teuthology.log
Comment 24 John Poelstra 2017-05-31 11:27:11 EDT 
development believes there is not a bug 

product needs more time to review

QE defers to product
Comment 36 Vasu Kulkarni 2017-07-31 15:13:39 EDT 
Sorry I held the testbed for quite some time before moving on, Since your build includes AWS signature changes I believe it is fixing this issue,  Can you talk to Thomas and generate a puddle build so that I can verify with new build and close this rather than try with old build(it is consistent with old build so there is no point in recreating it).

Also thanks for the landsat changes, I will update the scripts which will make everything green.
Comment 38 Kyle Bader 2017-08-01 16:16:34 EDT 
I've been able to reproduce this with:

ceph version 10.2.7-28.0.TEST.bz1470301.el7cp (8950adbd3b787f57ba50a41d5289dee56dc02e44)

RGW logs (debug ms = 1, debug rgw = 20)

https://gist.github.com/mmgaggle/dc706e8a592e10e7831e7fb14ca52870
Comment 41 Vasu Kulkarni 2017-08-02 12:59:31 EDT 
Doesn't seem to be fixed at, build used is ceph-radosgw-1:10.2.7-28.0.TEST.bz1476865.el7cp.x86_64 as provided by Thomas, Not sure if it has all the fixes.

http://magna002.ceph.redhat.com/vasu-2017-08-02_11:48:46-smoke-jewel---basic-multi/271165/teuthology.log

https://paste.fedoraproject.org/paste/RUW-9NnYuUTlDKlQkDhOtQ/raw
Comment 57 Vasu Kulkarni 2017-08-11 13:20:53 EDT 
Thomas,

I cant use the tar builds, mind generating dev-compose or providing a repo for the test build.  Thanks
Comment 66 Vasu Kulkarni 2017-08-16 20:35:20 EDT 
Other than the bucket-landsat issue (for which I need to fix the script to include what marcus has suggested in comment 34) all tests have passed. 

Tests run: 323, Failures: 0, Errors: 1, Skipped: 161

I will close this once I verify this again in async build, since this is a test build run.

http://magna002.ceph.redhat.com/vasu-2017-08-16_19:58:32-smoke-jewel---basic-multi/272322/teuthology.log
Comment 71 Tamil 2017-09-18 20:49:25 EDT 
hi Vasu, can you please verify this bug?
Comment 74 Vasu Kulkarni 2017-09-21 13:53:24 EDT 
I can actually close this issue, since I see this fixed in 2.8.0 run (ignoring the bucket-landsat issue)

http://magna002.ceph.redhat.com/vasu-2017-09-21_12:21:23-smoke-jewel---basic-multi/274636/teuthology.log

2017-09-21T12:54:00.698 INFO:teuthology.orchestra.run.pluto007.stdout:Tests run: 323, Failures: 0, Errors: 1, Skipped: 161


Also I ran on 2.8.1 just to check if anything fails, but It was clean in my run.

http://magna002.ceph.redhat.com/vasu-2017-09-21_12:21:23-smoke-jewel---basic-multi/274637/teuthology.log

2017-09-21T13:21:56.425 INFO:teuthology.orchestra.run.pluto007.stdout:Tests run: 323, Failures: 0, Errors: 1, Skipped: 161
Comment 87 errata-xmlrpc 2017-10-17 14:12:51 EDT 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2903
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.