Bug 1435003 - [RHCeph2.2/10.2.5-37] S3A part upload failed Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
Summary: [RHCeph2.2/10.2.5-37] S3A part upload failed Unable to unmarshall error resp...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 2.3
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 2.3
Assignee: Marcus Watts
QA Contact: Vasu Kulkarni
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-22 21:33 UTC by Vasu Kulkarni
Modified: 2022-02-21 18:42 UTC (History)
9 users (show)

Fixed In Version: RHEL: ceph-10.2.7-12.el7cp Ubuntu: ceph_10.2.7-14redhat1xenial
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-19 13:30:48 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 19736 0 None None None 2017-04-21 07:25:07 UTC
Red Hat Issue Tracker RHCEPH-3534 0 None None None 2022-02-21 18:42:57 UTC
Red Hat Product Errata RHBA-2017:1497 0 normal SHIPPED_LIVE Red Hat Ceph Storage 2.3 bug fix and enhancement update 2017-06-19 17:24:11 UTC

Description Vasu Kulkarni 2017-03-22 21:33:52 UTC
Description of problem:

while running s3a hadoop test, it fails with below error:
 Part upload failed in multi-part upload with id '2~RiBbajYQ6pH7xTaxUacOvGD0N02FEUe'

Steps:
1) Configure Ceph rgw for hadoop S3A tests (steps below)
  my current configuration includes dnsmasq name of s3.ceph.com and ip binding to 172.* address

[client.rgw.ceph-vakulkar-run657-node8-rgw]
rgw dns name = s3.ceph.com
keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw/keyring
rgw data = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw
host = ceph-vakulkar-run657-node8-rgw
rgw socket path = /tmp/radosgw-ceph-vakulkar-run657-node8-rgw.sock
log file = /var/log/ceph/ceph-rgw-ceph-vakulkar-run657-node8-rgw.log
rgw frontends = civetweb port=172.16.115.178:80 num_threads=50

dnsmasq configuration:
$ sudo cat /etc/dnsmasq.d/ceph 
address=/s3.ceph.com/172.16.115.178

Hadoop/Java Versions:
  Prereq: 
    sudo yum install protobuf-c.x86_64 java java-1.8.0-openjdk-devel

1) wget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
2) tar xvzf apache-maven-3.3.9-bin.tar.gz
3) git clone https://github.com/apache/hadoop
4) cd hadoop/
5) git checkout -b hadoop-2.7.3 rel/release-2.7.3
6) [cephuser@ceph-vakulkar-run657-node8-rgw ~]$ sudo cat hadoop/hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml
<configuration>

  <property>
    <name>fs.s3a.endpoint</name>
    <value>s3.ceph.com</value>
  </property>

  <property>
    <name>fs.s3a.connection.ssl.enabled</name>
    <value>false</value>
  </property>

  <property>
    <name>test.fs.s3n.name</name>
    <value>s3n://buck4/</value>
  </property>

  <property>
    <name>test.fs.s3a.name</name>
    <value>s3a://buck4/</value>
  </property>

  <property>
    <name>test.fs.s3.name</name>
    <value>s3://buck4/</value>
  </property>

  <property>
    <name>fs.s3.awsAccessKeyId</name>
    <value>EGAQRD2ULOIFKFSKCT4F</value>
  </property>

  <property>
    <name>fs.s3.awsSecretAccessKey</name>
    <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
  </property>

  <property>
    <name>fs.s3n.awsAccessKeyId</name>
    <value>EGAQRD2ULOIFKFSKCT4F</value>
  </property>

  <property>
    <name>fs.s3n.awsSecretAccessKey</name>
    <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
  </property>

  <property>
    <name>fs.s3a.access.key</name>
    <description>AWS access key ID. Omit for Role-based authentication.</description>
    <value>EGAQRD2ULOIFKFSKCT4F</value>
  </property>

  <property>
    <name>fs.s3a.secret.key</name>
    <description>AWS secret key. Omit for Role-based authentication.</description>
    <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
  </property>
</configuration>



Version-Release number of selected component (if applicable):
ceph version 10.2.5-37.el7cp (033f137cde8573cfc5a4662b4ed6a63b8a8d1464)

How reproducible:
Always

Steps to Reproduce:

cd hadoop/hadoop-tools/hadoop-aws/
  ~/apache-maven-3.3.9/bin/mvn test -Dtest=S3a*,TestS3A*


OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.684 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.203 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3ABlocksize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.275 sec - in org.apache.hadoop.fs.s3a.TestS3ABlocksize
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 187.977 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AFastOutputStream
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.516 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AFastOutputStream
testMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3AFastOutputStream)  Time elapsed: 1.032 sec  <<< ERROR!
java.io.IOException: Part upload failed in multi-part upload with id '2~nQaVapZvmxLCXmBgwv4-h-Vg_nhoO4r':java.util.concurrent.ExecutionException: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
	at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1408)
	at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1276)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.waitForAllPartUploads(S3AFastOutputStream.java:358)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:252)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
	at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:859)
	at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:887)
	at org.apache.hadoop.fs.s3a.TestS3AFastOutputStream.testMultiPartUpload(TestS3AFastOutputStream.java:71)
Caused by: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:792)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
	at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog.
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
	at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
	at com.amazonaws.util.XpathUtils.documentFrom(XpathUtils.java:64)
	at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:63)
	at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:38)
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:772)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
	at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Comment 7 Vasu Kulkarni 2017-04-19 18:03:16 UTC
I can consistently recreate this on different nodes

Comment 15 Vasu Kulkarni 2017-04-20 16:47:18 UTC
Marcus, You Rock!!!

Comment 21 Vasu Kulkarni 2017-04-28 17:05:59 UTC
Fixed in 10.2.7-13.el7cp

Tests run: 88, Failures: 0, Errors: 0, Skipped: 48

http://pulpito.ceph.redhat.com/vasu-2017-04-28_12:29:33-smoke-jewel---basic-multi/268017/

Comment 24 errata-xmlrpc 2017-06-19 13:30:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497


Note You need to log in before you can comment on or make changes to this bug.