1297027 – [1.3.2 - 0.94.5-1.el7cp] S3 test connection refused after test_random_file_valid

Bug 1297027 - [1.3.2 - 0.94.5-1.el7cp] S3 test connection refused after test_random_file_valid

Summary: [1.3.2 - 0.94.5-1.el7cp] S3 test connection refused after test_random_file_valid

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RGW
Sub Component:
Version:	1.3.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	1.3.3
Assignee:	Yehuda Sadeh
QA Contact:	ceph-qe-bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-01-08 19:07 UTC by Vasu Kulkarni
Modified:	2022-02-21 18:23 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-08-17 22:49:38 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHCEPH-3404	0	None	None	None	2022-02-21 18:23:41 UTC

Description Vasu Kulkarni 2016-01-08 19:07:10 UTC

Description of problem:

While running s3 tests connection refused exception after test_random_file_valid test

Version-Release number of selected component (if applicable):

0.94.5-1.el7cp

How reproducible:

2/2

Steps to Reproduce:
Run S3 tests


Actual results:

2016-01-08T12:51:09.886 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive ... ok
2016-01-08T12:51:09.887 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_mutual ... ok
2016-01-08T12:51:09.887 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_not_too_eager ... ok
2016-01-08T12:51:09.888 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_make_choice_unweighted_with_space ... ok
2016-01-08T12:51:09.891 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_choices ... ok
2016-01-08T12:51:09.891 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_null_choices ... ok
2016-01-08T12:51:09.892 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_null_choices ... ok
2016-01-08T12:51:09.893 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_null_child ... ok
2016-01-08T12:51:09.902 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_set ... ok
2016-01-08T12:51:09.903 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_header_presence ... ok
2016-01-08T12:51:09.904 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_duplicate_header ... ok
2016-01-08T12:51:09.905 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_headers ... ok
2016-01-08T12:51:10.266 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFileValidator.test_new_file_is_valid ... ok
2016-01-08T12:51:10.629 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_on_several_calls ... ok
2016-01-08T12:51:10.630 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_when_size_is_1 ... ok
2016-01-08T12:51:10.939 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFiles.test_random_file_valid ... ok
2016-01-08T12:51:10.943 INFO:teuthology.orchestra.run.clara008.stderr:
2016-01-08T12:51:10.944 INFO:teuthology.orchestra.run.clara008.stderr:======================================================================
2016-01-08T12:51:10.945 INFO:teuthology.orchestra.run.clara008.stderr:ERROR: test suite for <module 's3tests.functional' from '/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py'>
2016-01-08T12:51:10.945 INFO:teuthology.orchestra.run.clara008.stderr:----------------------------------------------------------------------
2016-01-08T12:51:10.946 INFO:teuthology.orchestra.run.clara008.stderr:Traceback (most recent call last):
2016-01-08T12:51:10.947 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/suite.py", line 209, in run
2016-01-08T12:51:10.948 INFO:teuthology.orchestra.run.clara008.stderr:    self.setUp()
2016-01-08T12:51:10.949 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
2016-01-08T12:51:10.950 INFO:teuthology.orchestra.run.clara008.stderr:    self.setupContext(ancestor)
2016-01-08T12:51:10.951 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
2016-01-08T12:51:10.952 INFO:teuthology.orchestra.run.clara008.stderr:    try_run(context, names)
2016-01-08T12:51:10.953 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
2016-01-08T12:51:10.954 INFO:teuthology.orchestra.run.clara008.stderr:    return func()
2016-01-08T12:51:10.955 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 339, in setup
2016-01-08T12:51:10.956 INFO:teuthology.orchestra.run.clara008.stderr:    nuke_prefixed_buckets(prefix=prefix)
2016-01-08T12:51:10.957 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 98, in nuke_prefixed_buckets
2016-01-08T12:51:10.957 INFO:teuthology.orchestra.run.clara008.stderr:    nuke_prefixed_buckets_on_conn(prefix, name, conn)
2016-01-08T12:51:10.958 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 63, in nuke_prefixed_buckets_on_conn
2016-01-08T12:51:10.959 INFO:teuthology.orchestra.run.clara008.stderr:    for bucket in conn.get_all_buckets():
2016-01-08T12:51:10.960 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/s3/connection.py", line 436, in get_all_buckets
2016-01-08T12:51:10.961 INFO:teuthology.orchestra.run.clara008.stderr:    response = self.make_request('GET', headers=headers)
2016-01-08T12:51:10.962 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
2016-01-08T12:51:10.962 INFO:teuthology.orchestra.run.clara008.stderr:    retry_handler=retry_handler
2016-01-08T12:51:10.963 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-01-08T12:51:10.964 INFO:teuthology.orchestra.run.clara008.stderr:    retry_handler=retry_handler)
2016-01-08T12:51:10.965 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-01-08T12:51:10.966 INFO:teuthology.orchestra.run.clara008.stderr:    raise ex
2016-01-08T12:51:10.967 INFO:teuthology.orchestra.run.clara008.stderr:error: [Errno 111] Connection refused
2016-01-08T12:51:10.968 INFO:teuthology.orchestra.run.clara008.stderr:-------------------- >> begin captured stdout << ---------------------
2016-01-08T12:51:10.968 INFO:teuthology.orchestra.run.clara008.stderr:Deleting buckets on alt
2016-01-08T12:51:10.969 INFO:teuthology.orchestra.run.clara008.stderr:Cleaning buckets from connection alt prefix 'test-client.0-5w4uqw2hqzhrmt3-'.

Logs:
http://magna002.ceph.redhat.com/vasu-2016-01-08_12:37:07-smoke:basic-v0.94.5---basic-clara/209965/teuthology.log

Comment 3 Yehuda Sadeh 2016-01-12 18:56:20 UTC

can you provide radosgw logs for this one? It looks a bit like misconfiguration, I'm not sure it even hit rgw.

Comment 4 Vasu Kulkarni 2016-01-12 19:05:12 UTC

Yehuda,

this folder should have all the logs:
http://magna002.ceph.redhat.com/vasu-2016-01-08_12:37:07-smoke:basic-v0.94.5---basic-clara/209965/

Comment 5 Yehuda Sadeh 2016-01-19 22:04:35 UTC

looks to me that requests didn't even reach rgw. Note that the test tried to connect to rgw via port 7280, whereas rgw was configured with port 7480.

Comment 6 Vasu Kulkarni 2016-01-21 23:37:53 UTC

I see some other issue in the run as well, I know this is a config issue since the other ec pool related s3 test ran fine, but was just curious why this wouldn't even connect

Comment 7 Vasu Kulkarni 2016-02-04 17:50:05 UTC

Verified on 0.94.5-6 and its not reproducible.

Comment 10 Ken Dreyer (Red Hat) 2016-08-17 22:27:54 UTC

Sounds like this is fixed in the latest 1.3.2 builds, and we just missed attaching it to the 1.3.2 advisory, so it didn't get closed.

Vasu, can we close this CURRENTRELEASE

Note You need to log in before you can comment on or make changes to this bug.