Bug 1435003 – [RHCeph2.2/10.2.5-37] S3A part upload failed Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1435003 - [RHCeph2.2/10.2.5-37] S3A part upload failed Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request

Summary:	[RHCeph2.2/10.2.5-37] S3A part upload failed Unable to unmarshall error resp...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	RGW (Show other bugs)
Sub Component:
Version:	2.3
Hardware:	Unspecified Unspecified

Priority	unspecified Severity unspecified
Target Milestone:	rc
Target Release:	2.3
Assigned To:	Marcus Watts
QA Contact:	Vasu Kulkarni
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2017-03-22 17:33 EDT by Vasu Kulkarni
Modified:	2017-07-30 12:03 EDT (History)
CC List:	9 users (show)

See Also:
Fixed In Version:	RHEL: ceph-10.2.7-12.el7cp Ubuntu: ceph_10.2.7-14redhat1xenial
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-06-19 09:30:48 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	19736	None	None	None	2017-04-21 03:25 EDT
Red Hat Product Errata	RHBA-2017:1497	normal	SHIPPED_LIVE	Red Hat Ceph Storage 2.3 bug fix and enhancement update	2017-06-19 13:24:11 EDT

Groups: None (edit)

Description Vasu Kulkarni 2017-03-22 17:33:52 EDT

Description of problem:

while running s3a hadoop test, it fails with below error:
 Part upload failed in multi-part upload with id '2~RiBbajYQ6pH7xTaxUacOvGD0N02FEUe'

Steps:
1) Configure Ceph rgw for hadoop S3A tests (steps below)
  my current configuration includes dnsmasq name of s3.ceph.com and ip binding to 172.* address

[client.rgw.ceph-vakulkar-run657-node8-rgw]
rgw dns name = s3.ceph.com
keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw/keyring
rgw data = /var/lib/ceph/radosgw/ceph-rgw.ceph-vakulkar-run657-node8-rgw
host = ceph-vakulkar-run657-node8-rgw
rgw socket path = /tmp/radosgw-ceph-vakulkar-run657-node8-rgw.sock
log file = /var/log/ceph/ceph-rgw-ceph-vakulkar-run657-node8-rgw.log
rgw frontends = civetweb port=172.16.115.178:80 num_threads=50

dnsmasq configuration:
$ sudo cat /etc/dnsmasq.d/ceph 
address=/s3.ceph.com/172.16.115.178

Hadoop/Java Versions:
  Prereq: 
    sudo yum install protobuf-c.x86_64 java java-1.8.0-openjdk-devel

1) wget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
2) tar xvzf apache-maven-3.3.9-bin.tar.gz
3) git clone https://github.com/apache/hadoop
4) cd hadoop/
5) git checkout -b hadoop-2.7.3 rel/release-2.7.3
6) [cephuser@ceph-vakulkar-run657-node8-rgw ~]$ sudo cat hadoop/hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml
<configuration>

  <property>
    <name>fs.s3a.endpoint</name>
    <value>s3.ceph.com</value>
  </property>

  <property>
    <name>fs.s3a.connection.ssl.enabled</name>
    <value>false</value>
  </property>

  <property>
    <name>test.fs.s3n.name</name>
    <value>s3n://buck4/</value>
  </property>

  <property>
    <name>test.fs.s3a.name</name>
    <value>s3a://buck4/</value>
  </property>

  <property>
    <name>test.fs.s3.name</name>
    <value>s3://buck4/</value>
  </property>

  <property>
    <name>fs.s3.awsAccessKeyId</name>
    <value>EGAQRD2ULOIFKFSKCT4F</value>
  </property>

  <property>
    <name>fs.s3.awsSecretAccessKey</name>
    <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
  </property>

  <property>
    <name>fs.s3n.awsAccessKeyId</name>
    <value>EGAQRD2ULOIFKFSKCT4F</value>
  </property>

  <property>
    <name>fs.s3n.awsSecretAccessKey</name>
    <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
  </property>

  <property>
    <name>fs.s3a.access.key</name>
    <description>AWS access key ID. Omit for Role-based authentication.</description>
    <value>EGAQRD2ULOIFKFSKCT4F</value>
  </property>

  <property>
    <name>fs.s3a.secret.key</name>
    <description>AWS secret key. Omit for Role-based authentication.</description>
    <value>zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb</value>
  </property>
</configuration>



Version-Release number of selected component (if applicable):
ceph version 10.2.5-37.el7cp (033f137cde8573cfc5a4662b4ed6a63b8a8d1464)

How reproducible:
Always

Steps to Reproduce:

cd hadoop/hadoop-tools/hadoop-aws/
  ~/apache-maven-3.3.9/bin/mvn test -Dtest=S3a*,TestS3A*


OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.684 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Tests run: 31, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.203 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3ABlocksize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.275 sec - in org.apache.hadoop.fs.s3a.TestS3ABlocksize
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 187.977 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
Running org.apache.hadoop.fs.s3a.TestS3AFastOutputStream
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.516 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AFastOutputStream
testMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3AFastOutputStream)  Time elapsed: 1.032 sec  <<< ERROR!
java.io.IOException: Part upload failed in multi-part upload with id '2~nQaVapZvmxLCXmBgwv4-h-Vg_nhoO4r':java.util.concurrent.ExecutionException: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
	at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1408)
	at com.google.common.util.concurrent.Futures$ListFuture.get(Futures.java:1276)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.waitForAllPartUploads(S3AFastOutputStream.java:358)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:252)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
	at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:859)
	at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:887)
	at org.apache.hadoop.fs.s3a.TestS3AFastOutputStream.testMultiPartUpload(TestS3AFastOutputStream.java:71)
Caused by: com.amazonaws.AmazonClientException: Unable to unmarshall error response (Content is not allowed in prolog.). Response Code: 400, Response Text: Bad Request
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:792)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
	at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog.
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
	at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
	at com.amazonaws.util.XpathUtils.documentFrom(XpathUtils.java:64)
	at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:63)
	at com.amazonaws.services.s3.internal.S3ErrorResponseHandler.handle(S3ErrorResponseHandler.java:38)
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:772)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
	at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2711)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:350)
	at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload$1.call(S3AFastOutputStream.java:343)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Comment 7 Vasu Kulkarni 2017-04-19 14:03:16 EDT

I can consistently recreate this on different nodes

Comment 15 Vasu Kulkarni 2017-04-20 12:47:18 EDT

Marcus, You Rock!!!

Comment 21 Vasu Kulkarni 2017-04-28 13:05:59 EDT

Fixed in 10.2.7-13.el7cp

Tests run: 88, Failures: 0, Errors: 0, Skipped: 48

http://pulpito.ceph.redhat.com/vasu-2017-04-28_12:29:33-smoke-jewel---basic-multi/268017/

Comment 24 errata-xmlrpc 2017-06-19 09:30:48 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497

Note You need to log in before you can comment on or make changes to this bug.