[RHCeph2.2/10.2.5-37] S3A part upload failed Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
Description of problem:
while running s3a hadoop test, it fails with below error:
Part upload failed in multi-part upload with id '2~RiBbajYQ6pH7xTaxUacOvGD0N02FEUe'
Steps:
1) Configure Ceph rgw for hadoop S3A tests (steps below)
my current configuration includes dnsmasq name of s3.ceph.com and ip binding to 172.* address
[client.rgw.ceph-vakulkar-run657-node8-rgw]
rgw dns name = s3.ceph.com
keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw/keyring
rgw data = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw
host = ceph-vakulkar-run657-node8-rgw
rgw socket path = /tmp/radosgw-ceph-vakulkar-run657-node8-rgw.sock
log file = /var/log/ceph/ceph-rgw-ceph-vakulkar-run657-node8-rgw.log
rgw frontends = civetweb port=172.16.115.178:80 num_threads=50
dnsmasq configuration:
$ sudo cat /etc/dnsmasq.d/ceph
address=/s3.ceph.com/172.16.115.178
Hadoop/Java Versions:
Prereq:
sudo yum install protobuf-c.x86_64 java java-1.8.0-openjdk-devel
1) wget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
2) tar xvzf apache-maven-3.3.9-bin.tar.gz
3) git clone https://github.com/apache/hadoop
4) cd hadoop/
5) git checkout -b hadoop-2.7.3 rel/release-2.7.3
6) [cephuser@ceph-vakulkar-run657-node8-rgw ~]$ sudo cat hadoop/hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml
<configuration>
<property>
<name>fs.s3a.endpoint</name>
<value>s3.ceph.com</value>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>false</value>
</property>
<property>
<name>test.fs.s3n.name</name>
<value>s3n://buck4/</value>
</property>
<property>
<name>test.fs.s3a.name</name>
<value>s3a://buck4/</value>
</property>
<property>
<name>test.fs.s3.name</name>
<value>s3://buck4/</value>
</property>
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>EGAQRD2ULOIFKFSKCT4F</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
</property>
<property>
<name>fs.s3n.awsAccessKeyId</name>
<value>EGAQRD2ULOIFKFSKCT4F</value>
</property>
<property>
<name>fs.s3n.awsSecretAccessKey</name>
<value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<description>AWS access key ID. Omit for Role-based authentication.</description>
<value>EGAQRD2ULOIFKFSKCT4F</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<description>AWS secret key. Omit for Role-based authentication.</description>
<value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
</property>
</configuration>
Version-Release number of selected component (if applicable):
ceph version 10.2.5-37.el7cp (033f137cde8573cfc5a4662b4ed6a63b8a8d1464)
How reproducible:
Always
Steps to Reproduce:
cd hadoop/hadoop-tools/hadoop-aws/
~/apache-maven-3.3.9/bin/mvn test -Dtest=S3a*,TestS3A*
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.684 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.203 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3ABlocksize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.275 sec - in org.apache.hadoop.fs.s3a.TestS3ABlocksize
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 187.977 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AFastOutputStream
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.516 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AFastOutputStream
testMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3AFastOutputStream) Time elapsed: 1.032 sec <<< ERROR!
java.io.IOException: Part upload failed in multi-part upload with id '2~nQaVapZvmxLCXmBgwv4-h-Vg_nhoO4r':java.util.concurrent.ExecutionException: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1408)
at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1276)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.waitForAllPartUploads(S3AFastOutputStream.java:358)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:252)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:859)
at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:887)
at org.apache.hadoop.fs.s3a.TestS3AFastOutputStream.testMultiPartUpload(TestS3AFastOutputStream.java:71)
Caused by: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:792)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
at com.amazonaws.util.XpathUtils.documentFrom(XpathUtils.java:64)
at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:63)
at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:38)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:772)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2017:1497