Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1297027

Summary: [1.3.2 - 0.94.5-1.el7cp] S3 test connection refused after test_random_file_valid
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vasu Kulkarni <vakulkar>
Component: RGWAssignee: Yehuda Sadeh <yehuda>
Status: CLOSED CURRENTRELEASE QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 1.3.2CC: cbodley, ceph-eng-bugs, flucifre, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil, vakulkar, yehuda
Target Milestone: rc   
Target Release: 1.3.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-17 22:49:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vasu Kulkarni 2016-01-08 19:07:10 UTC 
Description of problem:

While running s3 tests connection refused exception after test_random_file_valid test

Version-Release number of selected component (if applicable):

0.94.5-1.el7cp

How reproducible:

2/2

Steps to Reproduce:
Run S3 tests


Actual results:

2016-01-08T12:51:09.886 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive ... ok
2016-01-08T12:51:09.887 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_mutual ... ok
2016-01-08T12:51:09.887 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_not_too_eager ... ok
2016-01-08T12:51:09.888 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_make_choice_unweighted_with_space ... ok
2016-01-08T12:51:09.891 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_choices ... ok
2016-01-08T12:51:09.891 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_null_choices ... ok
2016-01-08T12:51:09.892 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_null_choices ... ok
2016-01-08T12:51:09.893 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_null_child ... ok
2016-01-08T12:51:09.902 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_set ... ok
2016-01-08T12:51:09.903 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_header_presence ... ok
2016-01-08T12:51:09.904 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_duplicate_header ... ok
2016-01-08T12:51:09.905 INFO:teuthology.orchestra.run.clara008.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_headers ... ok
2016-01-08T12:51:10.266 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFileValidator.test_new_file_is_valid ... ok
2016-01-08T12:51:10.629 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_on_several_calls ... ok
2016-01-08T12:51:10.630 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_when_size_is_1 ... ok
2016-01-08T12:51:10.939 INFO:teuthology.orchestra.run.clara008.stderr:test_realistic.TestFiles.test_random_file_valid ... ok
2016-01-08T12:51:10.943 INFO:teuthology.orchestra.run.clara008.stderr:
2016-01-08T12:51:10.944 INFO:teuthology.orchestra.run.clara008.stderr:======================================================================
2016-01-08T12:51:10.945 INFO:teuthology.orchestra.run.clara008.stderr:ERROR: test suite for <module 's3tests.functional' from '/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py'>
2016-01-08T12:51:10.945 INFO:teuthology.orchestra.run.clara008.stderr:----------------------------------------------------------------------
2016-01-08T12:51:10.946 INFO:teuthology.orchestra.run.clara008.stderr:Traceback (most recent call last):
2016-01-08T12:51:10.947 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/suite.py", line 209, in run
2016-01-08T12:51:10.948 INFO:teuthology.orchestra.run.clara008.stderr:    self.setUp()
2016-01-08T12:51:10.949 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
2016-01-08T12:51:10.950 INFO:teuthology.orchestra.run.clara008.stderr:    self.setupContext(ancestor)
2016-01-08T12:51:10.951 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
2016-01-08T12:51:10.952 INFO:teuthology.orchestra.run.clara008.stderr:    try_run(context, names)
2016-01-08T12:51:10.953 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
2016-01-08T12:51:10.954 INFO:teuthology.orchestra.run.clara008.stderr:    return func()
2016-01-08T12:51:10.955 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 339, in setup
2016-01-08T12:51:10.956 INFO:teuthology.orchestra.run.clara008.stderr:    nuke_prefixed_buckets(prefix=prefix)
2016-01-08T12:51:10.957 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 98, in nuke_prefixed_buckets
2016-01-08T12:51:10.957 INFO:teuthology.orchestra.run.clara008.stderr:    nuke_prefixed_buckets_on_conn(prefix, name, conn)
2016-01-08T12:51:10.958 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 63, in nuke_prefixed_buckets_on_conn
2016-01-08T12:51:10.959 INFO:teuthology.orchestra.run.clara008.stderr:    for bucket in conn.get_all_buckets():
2016-01-08T12:51:10.960 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/s3/connection.py", line 436, in get_all_buckets
2016-01-08T12:51:10.961 INFO:teuthology.orchestra.run.clara008.stderr:    response = self.make_request('GET', headers=headers)
2016-01-08T12:51:10.962 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
2016-01-08T12:51:10.962 INFO:teuthology.orchestra.run.clara008.stderr:    retry_handler=retry_handler
2016-01-08T12:51:10.963 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
2016-01-08T12:51:10.964 INFO:teuthology.orchestra.run.clara008.stderr:    retry_handler=retry_handler)
2016-01-08T12:51:10.965 INFO:teuthology.orchestra.run.clara008.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
2016-01-08T12:51:10.966 INFO:teuthology.orchestra.run.clara008.stderr:    raise ex
2016-01-08T12:51:10.967 INFO:teuthology.orchestra.run.clara008.stderr:error: [Errno 111] Connection refused
2016-01-08T12:51:10.968 INFO:teuthology.orchestra.run.clara008.stderr:-------------------- >> begin captured stdout << ---------------------
2016-01-08T12:51:10.968 INFO:teuthology.orchestra.run.clara008.stderr:Deleting buckets on alt
2016-01-08T12:51:10.969 INFO:teuthology.orchestra.run.clara008.stderr:Cleaning buckets from connection alt prefix 'test-client.0-5w4uqw2hqzhrmt3-'.

Logs:
http://magna002.ceph.redhat.com/vasu-2016-01-08_12:37:07-smoke:basic-v0.94.5---basic-clara/209965/teuthology.log
Comment 3 Yehuda Sadeh 2016-01-12 18:56:20 UTC 
can you provide radosgw logs for this one? It looks a bit like misconfiguration, I'm not sure it even hit rgw.
Comment 4 Vasu Kulkarni 2016-01-12 19:05:12 UTC 
Yehuda,

this folder should have all the logs:
http://magna002.ceph.redhat.com/vasu-2016-01-08_12:37:07-smoke:basic-v0.94.5---basic-clara/209965/
Comment 5 Yehuda Sadeh 2016-01-19 22:04:35 UTC 
looks to me that requests didn't even reach rgw. Note that the test tried to connect to rgw via port 7280, whereas rgw was configured with port 7480.
Comment 6 Vasu Kulkarni 2016-01-21 23:37:53 UTC 
I see some other issue in the run as well, I know this is a config issue since the other ec pool related s3 test ran fine, but was just curious why this wouldn't even connect
Comment 7 Vasu Kulkarni 2016-02-04 17:50:05 UTC 
Verified on 0.94.5-6 and its not reproducible.
Comment 10 Ken Dreyer (Red Hat) 2016-08-17 22:27:54 UTC 
Sounds like this is fixed in the latest 1.3.2 builds, and we just missed attaching it to the 1.3.2 advisory, so it didn't get closed.

Vasu, can we close this CURRENTRELEASE