Description of problem: while running s3a hadoop test, it fails with below error: Part upload failed in multi-part upload with id '2~RiBbajYQ6pH7xTaxUacOvGD0N02FEUe' Steps: 1) Configure Ceph rgw for hadoop S3A tests (steps below) my current configuration includes dnsmasq name of s3.ceph.com and ip binding to 172.* address [client.rgw.ceph-vakulkar-run657-node8-rgw] rgw dns name = s3.ceph.com keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw/keyring rgw data = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw host = ceph-vakulkar-run657-node8-rgw rgw socket path = /tmp/radosgw-ceph-vakulkar-run657-node8-rgw.sock log file = /var/log/ceph/ceph-rgw-ceph-vakulkar-run657-node8-rgw.log rgw frontends = civetweb port=172.16.115.178:80 num_threads=50 dnsmasq configuration: $ sudo cat /etc/dnsmasq.d/ceph address=/s3.ceph.com/172.16.115.178 Hadoop/Java Versions: Prereq: sudo yum install protobuf-c.x86_64 java java-1.8.0-openjdk-devel 1) wget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz 2) tar xvzf apache-maven-3.3.9-bin.tar.gz 3) git clone https://github.com/apache/hadoop 4) cd hadoop/ 5) git checkout -b hadoop-2.7.3 rel/release-2.7.3 6) [cephuser@ceph-vakulkar-run657-node8-rgw ~]$ sudo cat hadoop/hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml <configuration> <property> <name>fs.s3a.endpoint</name> <value>s3.ceph.com</value> </property> <property> <name>fs.s3a.connection.ssl.enabled</name> <value>false</value> </property> <property> <name>test.fs.s3n.name</name> <value>s3n://buck4/</value> </property> <property> <name>test.fs.s3a.name</name> <value>s3a://buck4/</value> </property> <property> <name>test.fs.s3.name</name> <value>s3://buck4/</value> </property> <property> <name>fs.s3.awsAccessKeyId</name> <value>EGAQRD2ULOIFKFSKCT4F</value> </property> <property> <name>fs.s3.awsSecretAccessKey</name> <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value> </property> <property> <name>fs.s3n.awsAccessKeyId</name> <value>EGAQRD2ULOIFKFSKCT4F</value> </property> <property> <name>fs.s3n.awsSecretAccessKey</name> <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value> </property> <property> <name>fs.s3a.access.key</name> <description>AWS access key ID. Omit for Role-based authentication.</description> <value>EGAQRD2ULOIFKFSKCT4F</value> </property> <property> <name>fs.s3a.secret.key</name> <description>AWS secret key. Omit for Role-based authentication.</description> <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value> </property> </configuration> Version-Release number of selected component (if applicable): ceph version 10.2.5-37.el7cp (033f137cde8573cfc5a4662b4ed6a63b8a8d1464) How reproducible: Always Steps to Reproduce: cd hadoop/hadoop-tools/hadoop-aws/ ~/apache-maven-3.3.9/bin/mvn test -Dtest=S3a*,TestS3A* OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.s3a.TestS3AConfiguration Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.684 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.203 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.s3a.TestS3ABlocksize Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.275 sec - in org.apache.hadoop.fs.s3a.TestS3ABlocksize OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 187.977 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.s3a.TestS3AFastOutputStream Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.516 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AFastOutputStream testMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3AFastOutputStream) Time elapsed: 1.032 sec <<< ERROR! java.io.IOException: Part upload failed in multi-part upload with id '2~nQaVapZvmxLCXmBgwv4-h-Vg_nhoO4r':java.util.concurrent.ExecutionException: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1408) at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1276) at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.waitForAllPartUploads(S3AFastOutputStream.java:358) at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:252) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:859) at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:887) at org.apache.hadoop.fs.s3a.TestS3AFastOutputStream.testMultiPartUpload(TestS3AFastOutputStream.java:71) Caused by: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:792) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711) at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350) at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) at com.amazonaws.util.XpathUtils.documentFrom(XpathUtils.java:64) at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:63) at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:38) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:772) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711) at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350) at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
I can consistently recreate this on different nodes
Marcus, You Rock!!!
Fixed in 10.2.7-13.el7cp Tests run: 88, Failures: 0, Errors: 0, Skipped: 48 http://pulpito.ceph.redhat.com/vasu-2017-04-28_12:29:33-smoke-jewel---basic-multi/268017/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1497