Description of problem: The tests in tempest.api.object_storage.test_object_slo.ObjectSloTest are always failing with director deploys. They pass with the packstack rhel osp7 jobs. Here is the stacktrace. It looks like a 'transfer-encoding' is expected in the header of the response. Below the stacktrace is the tempest log showing the calls from one of these tests. Traceback (most recent call last): File "/home/stack/tempest/tempest/api/object_storage/test_object_slo.py", line 166, in test_delete_large_object self.assertIn('transfer-encoding', resp) File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 354, in assertIn self.assertThat(haystack, Contains(needle), message) File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat raise mismatch_error MismatchError: 'transfer-encoding' not in {'status': '204', 'content-length': '0', 'connection': 'close', 'x-trans-id': 'txef617168977b4ce5baa15-0055cdf485', 'date': 'Fri, 14 Aug 2015 14:00:37 GMT', 'content-type': 'text/html; charset=UTF-8'} Traceback (most recent call last): _StringException: Empty attachments: stderr stdout
Please, retest with the latest ospd.
Please retest w/ ospd-7 HA 3 controllers
According to our records, this should be resolved by openstack-swift-2.3.0-4.el7ost. This build is available now.
Created attachment 1143825 [details] swift.conf
Created attachment 1143826 [details] proxy-server.conf
Created attachment 1143828 [details] tempest results
running slo tests from tempest fails for me with 3 controllers and >= the requisite RPM: [root@seal14 ~]# virsh list --all Id Name State ---------------------------------------------------- 20 undercloud running 30 compute running 31 controller2 running 32 controller1 running 33 controller3 running [root@seal14 ~]# --- [stack@undercloud ~]$ rpm -qa openstack-swift openstack-swift-2.3.0-4.el7ost.noarch [stack@undercloud ~]$ grep ^VERSION= /etc/*release /etc/os-release:VERSION="7.2 (Maipo)" [stack@undercloud ~]$ ls /etc/yum.repos.d/ redhat.repo rhos-release-7-director.repo rhos-release-7.repo rhos-release.repo rhos-release-rhel-7.2.repo [stack@undercloud ~]$ attached swift.conf, proxy-server.conf, tempest_slo_test.out Please advise.
Mike, can you do a simple curl request against the proxy please? Like this: curl http://<proxy IP>:8080/info
Created attachment 1143844 [details] tempest results #2 (after dlo is out of the pipeline)
turns out i couldn't curl to there, but after i took out dlo from the pipeline and restarted the proxy (systemctl restart openstack-swift-proxy.service) i can: [root@undercloud ~]# curl http://172.16.0.1:8080/info {"formpost": {}, "ratelimit": {"account_ratelimit": 0.0, "max_sleep_time_seconds": 60.0, "container_ratelimits": [], "container_listing_ratelimits": []}, "slo": {"max_manifest_segments": 1000, "min_segment_size": 1048576, "max_manifest_size": 2097152}, "staticweb": {}, "tempurl": {"methods": ["GET", "HEAD", "PUT", "POST", "DELETE"]}, "swift": {"max_file_size": 5368709122, "account_listing_limit": 10000, "account_autocreate": true, "max_meta_count": 90, "max_meta_value_length": 256, "container_listing_limit": 10000, "max_meta_overall_size": 4096, "version": "2.3.0", "max_meta_name_length": 128, "max_header_size": 8192, "policies": [{"default": true, "name": "Policy-0"}], "max_object_name_length": 1024, "max_account_name_length": 256, "strict_cors_mode": true, "allow_account_management": true, "max_container_name_length": 256}}[root@undercloud ~]# ...but unfortunately, the tests still fail: [stack@undercloud tempest]$ tools/run-tests.sh -- tempest.api.object_storage.test_object_slo . . . Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/home/stack/tempest/tempest/test.py", line 158, in wrapper return func(*func_args, **func_kwargs) File "/home/stack/tempest/tempest/api/object_storage/test_object_slo.py", line 125, in test_upload_manifest self._assertHeadersSLO(resp, 'PUT') File "/home/stack/tempest/tempest/api/object_storage/test_object_slo.py", line 107, in _assertHeadersSLO self.assertTrue(resp['etag'].startswith('\"')) File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 702, in assertTrue raise self.failureException(msg) AssertionError: False is not true ====== Totals ====== Ran: 4 tests in 8.0000 sec. - Passed: 0 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 4 Sum of execute time for each test: 1.8040 sec. ============== Worker Balance ============== - Worker 0 (4 tests) => 0:00:01.812156 No tests were successful during the run [stack@undercloud tempest]$
also i uploaded the latest tempest output to the file "tempest results #2".
The tempest test expects a quoted etag, which only happens when the slo middleware parses the request. However, it seems that this is not the case here - although the slo middleware is listed in the /info response. What was the reason to remove the dlo middleware? Does querying the /info endpoint doesn't work with the dlo middleware enabled? What does the pipeline look like now?
removing dlo middleware shouldn't make a difference since it gets added automatically by the code. I noticed ceilometer is in the wrong place in the pipeline, maybe that's causing a problem?? ceilometer should be placed after slo, dlo
Created attachment 1145832 [details] tempest results #3 with the dlo middleware in the pipeline, i couldn't do a 'systemctl restart openstack-swift-proxy.service'. i put ceilometer after slo in the pipeline and the test still fails (see slo.test.3 attached).
While looking again at the proxy configuration files, I noticed the dlo filter is not there. The filter can be added to the end of the file like this: [filter:dlo] use = egg:swift#dlo I don't think the dlo issue explains the SLO issue you are encountering. Since it is failing all SLO tests while using the director but passing when using packstack, it seems to be a configuration issue. Can you run the swift functional tests against this install? Or maybe even just run a manual slo test against the end point. If you can provide remote login information I could also take a closer look at to try to figure out what's happening...
Thanks Thiago. I'll send you the remote information via email.
VERIFIED. ------------------------------------------------ CONFIG ------ [stack@undercloud ~]$ nova list +--------------------------------------+------------------------+--------+------------+-------------+----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------+--------+------------+-------------+----------------------+ | 78585cfb-7ff5-45cb-a2fa-7509c722e78e | overcloud-compute-0 | ACTIVE | - | Running | ctlplane=172.16.0.22 | | 28e87a60-0ed7-41d6-8ea2-343afb3c4dba | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=172.16.0.23 | | ef45f570-1e10-429e-aa63-d3c623bcf9b9 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=172.16.0.24 | | d0d48130-53a0-46ac-a093-b60c32746067 | overcloud-controller-2 | ACTIVE | - | Running | ctlplane=172.16.0.25 | +--------------------------------------+------------------------+--------+------------+-------------+----------------------+ [stack@undercloud ~]$ [stack@undercloud tempest]$ for i in 23 24 25; do ssh heat-admin.0.$i "sudo grep slo /etc/swift/proxy-server.conf"; donepipeline = catch_errors healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb slo ceilometer proxy-logging proxy-server [filter:slo] use = egg:swift#slo pipeline = catch_errors healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb slo ceilometer proxy-logging proxy-server [filter:slo] use = egg:swift#slo pipeline = catch_errors healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb slo ceilometer proxy-logging proxy-server [filter:slo] use = egg:swift#slo [stack@undercloud tempest]$ ------------------------------------------------ TEST RUN -------- mabrams@optiplex:~$ ssh -l root seal14.qa.lab.tlv.redhat.com Last login: Tue Apr 12 19:11:06 2016 from vpn-48-88.rdu2.redhat.com [root@seal14 ~]# s^C [root@seal14 ~]# ssh stack.0.67 stack.0.67's password: Last login: Tue Apr 12 16:01:25 2016 from 10.0.0.1 [stack@undercloud ~]$ sudo su - Last login: ג' אפר 12 04:45:04 EDT 2016 on pts/0 [root@undercloud ~]# . /home/stack/overcloudrc [root@undercloud ~]# cd /etc/swift/t^C [root@undercloud ~]# cd /home/^C [root@undercloud ~]# logout [stack@undercloud ~]$ . ./overcloudrc [stack@undercloud ~]$ cd tempest [stack@undercloud tempest]$ tools/run-tests.sh -- tempest.api.object_storage.test_object_slo running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpLPV4mb running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \ OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \ ${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpLPV4mb {0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_delete_large_object [0.718254s] ... ok {0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_list_large_object_metadata [0.432042s] ... ok {0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_retrieve_large_object [0.507300s] ... ok {0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_upload_manifest [0.392570s] ... ok ====== Totals ====== Ran: 4 tests in 9.0000 sec. - Passed: 4 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 2.0502 sec. ============== Worker Balance ============== - Worker 0 (4 tests) => 0:00:02.054925 [stack@undercloud tempest]$ ------------------------------------------------