Bug 1253765 - Tempest swift ObjectSloTest tests fail with director deploy, but not packstack
Tempest swift ObjectSloTest tests fail with director deploy, but not packstack
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-swift (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
medium Severity unspecified
: z5
: 7.0 (Kilo)
Assigned To: Pete Zaitcev
Mike Abrams
: AutomationBlocker, TestOnly, Triaged, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-14 11:56 EDT by David Kranz
Modified: 2016-09-19 11:37 EDT (History)
13 users (show)

See Also:
Fixed In Version: openstack-swift-2.3.0-2.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-18 12:38:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
cschwede: needinfo-


Attachments (Terms of Use)
swift.conf (79 bytes, text/plain)
2016-04-05 08:41 EDT, Mike Abrams
no flags Details
proxy-server.conf (2.08 KB, text/plain)
2016-04-05 08:42 EDT, Mike Abrams
no flags Details
tempest results (70.78 KB, text/plain)
2016-04-05 08:42 EDT, Mike Abrams
no flags Details
tempest results #2 (after dlo is out of the pipeline) (70.82 KB, text/plain)
2016-04-05 09:38 EDT, Mike Abrams
no flags Details
tempest results #3 (70.83 KB, text/plain)
2016-04-11 01:18 EDT, Mike Abrams
no flags Details

  None (edit)
Description David Kranz 2015-08-14 11:56:03 EDT
Description of problem:

The tests in tempest.api.object_storage.test_object_slo.ObjectSloTest are always failing with director deploys. They pass with the packstack rhel osp7 jobs.

Here is the stacktrace. It looks like a 'transfer-encoding' is expected in the header of the response. Below the stacktrace is the tempest log showing the calls from one of these tests. 

Traceback (most recent call last):
      File "/home/stack/tempest/tempest/api/object_storage/test_object_slo.py", line 166, in test_delete_large_object
        self.assertIn('transfer-encoding', resp)
      File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 354, in assertIn
        self.assertThat(haystack, Contains(needle), message)
      File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 433, in assertThat
        raise mismatch_error
    MismatchError: 'transfer-encoding' not in {'status': '204', 'content-length': '0', 'connection': 'close', 'x-trans-id': 'txef617168977b4ce5baa15-0055cdf485', 'date': 'Fri, 14 Aug 2015 14:00:37 GMT', 'content-type': 'text/html; charset=UTF-8'}
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stderr
      stdout
Comment 6 Sergey Gotliv 2016-01-03 17:23:18 EST
Please, retest with the latest ospd.
Comment 7 wes hayutin 2016-01-31 08:39:37 EST
Please retest w/ ospd-7 HA 3 controllers
Comment 11 Lon Hohberger 2016-03-15 16:12:44 EDT
According to our records, this should be resolved by openstack-swift-2.3.0-4.el7ost.  This build is available now.
Comment 12 Mike Abrams 2016-04-05 08:41 EDT
Created attachment 1143825 [details]
swift.conf
Comment 13 Mike Abrams 2016-04-05 08:42 EDT
Created attachment 1143826 [details]
proxy-server.conf
Comment 14 Mike Abrams 2016-04-05 08:42 EDT
Created attachment 1143828 [details]
tempest results
Comment 15 Mike Abrams 2016-04-05 08:46:56 EDT
running slo tests from tempest fails for me with 3 controllers and >= the requisite RPM:

[root@seal14 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 20    undercloud                     running
 30    compute                        running
 31    controller2                    running
 32    controller1                    running
 33    controller3                    running

[root@seal14 ~]# 

---

[stack@undercloud ~]$ rpm -qa openstack-swift
openstack-swift-2.3.0-4.el7ost.noarch
[stack@undercloud ~]$ grep ^VERSION= /etc/*release
/etc/os-release:VERSION="7.2 (Maipo)"
[stack@undercloud ~]$ ls /etc/yum.repos.d/
redhat.repo  rhos-release-7-director.repo  rhos-release-7.repo  rhos-release.repo  rhos-release-rhel-7.2.repo
[stack@undercloud ~]$ 

attached swift.conf, proxy-server.conf, tempest_slo_test.out

Please advise.
Comment 16 Christian Schwede (cschwede) 2016-04-05 08:53:52 EDT
Mike, can you do a simple curl request against the proxy please? Like this:

curl http://<proxy IP>:8080/info
Comment 17 Mike Abrams 2016-04-05 09:38 EDT
Created attachment 1143844 [details]
tempest results #2 (after dlo is out of the pipeline)
Comment 18 Mike Abrams 2016-04-05 09:39:14 EDT
turns out i couldn't curl to there, but after i took out dlo from the pipeline and restarted the proxy (systemctl restart openstack-swift-proxy.service) i can:

[root@undercloud ~]# curl http://172.16.0.1:8080/info
{"formpost": {}, "ratelimit": {"account_ratelimit": 0.0, "max_sleep_time_seconds": 60.0, "container_ratelimits": [], "container_listing_ratelimits": []}, "slo": {"max_manifest_segments": 1000, "min_segment_size": 1048576, "max_manifest_size": 2097152}, "staticweb": {}, "tempurl": {"methods": ["GET", "HEAD", "PUT", "POST", "DELETE"]}, "swift": {"max_file_size": 5368709122, "account_listing_limit": 10000, "account_autocreate": true, "max_meta_count": 90, "max_meta_value_length": 256, "container_listing_limit": 10000, "max_meta_overall_size": 4096, "version": "2.3.0", "max_meta_name_length": 128, "max_header_size": 8192, "policies": [{"default": true, "name": "Policy-0"}], "max_object_name_length": 1024, "max_account_name_length": 256, "strict_cors_mode": true, "allow_account_management": true, "max_container_name_length": 256}}[root@undercloud ~]#

...but unfortunately, the tests still fail:

[stack@undercloud tempest]$ tools/run-tests.sh -- tempest.api.object_storage.test_object_slo
.
.
.
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/stack/tempest/tempest/test.py", line 158, in wrapper
        return func(*func_args, **func_kwargs)
      File "/home/stack/tempest/tempest/api/object_storage/test_object_slo.py", line 125, in test_upload_manifest
        self._assertHeadersSLO(resp, 'PUT')
      File "/home/stack/tempest/tempest/api/object_storage/test_object_slo.py", line 107, in _assertHeadersSLO
        self.assertTrue(resp['etag'].startswith('\"'))
      File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 702, in assertTrue
        raise self.failureException(msg)
    AssertionError: False is not true
   


======
Totals
======
Ran: 4 tests in 8.0000 sec.
 - Passed: 0
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 4
Sum of execute time for each test: 1.8040 sec.

==============
Worker Balance
==============
 - Worker 0 (4 tests) => 0:00:01.812156

No tests were successful during the run
[stack@undercloud tempest]$
Comment 19 Mike Abrams 2016-04-06 02:37:28 EDT
also i uploaded the latest tempest output to the file "tempest results #2".
Comment 20 Christian Schwede (cschwede) 2016-04-06 10:33:12 EDT
The tempest test expects a quoted etag, which only happens when the slo middleware parses the request. However, it seems that this is not the case here - although the slo middleware is listed in the /info response.

What was the reason to remove the dlo middleware? Does querying the /info endpoint doesn't work with the dlo middleware enabled? What does the pipeline look like now?
Comment 21 Thiago da Silva 2016-04-06 10:49:52 EDT
removing dlo middleware shouldn't make a difference since it gets added automatically by the code.

I noticed ceilometer is in the wrong place in the pipeline, maybe that's causing a problem?? ceilometer should be placed after slo, dlo
Comment 22 Mike Abrams 2016-04-11 01:18 EDT
Created attachment 1145832 [details]
tempest results #3

with the dlo middleware in the pipeline, i couldn't do a 'systemctl restart openstack-swift-proxy.service'.

i put ceilometer after slo in the pipeline and the test still fails (see slo.test.3 attached).
Comment 23 Thiago da Silva 2016-04-11 10:44:49 EDT
While looking again at the proxy configuration files, I noticed the dlo filter is not there. The filter can be added to the end of the file like this:

[filter:dlo]
use = egg:swift#dlo

I don't think the dlo issue explains the SLO issue you are encountering. Since it is failing all SLO tests while using the director but passing when using packstack, it seems to be a configuration issue.

Can you run the swift functional tests against this install? Or maybe even just run a manual slo test against the end point. If you can provide remote login information I could also take a closer look at to try to figure out what's happening...
Comment 24 Mike Abrams 2016-04-11 12:04:42 EDT
Thanks Thiago.

I'll send you the remote information via email.
Comment 25 Mike Abrams 2016-04-13 08:06:38 EDT
VERIFIED.
------------------------------------------------
CONFIG
------

[stack@undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+----------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks             |
+--------------------------------------+------------------------+--------+------------+-------------+----------------------+
| 78585cfb-7ff5-45cb-a2fa-7509c722e78e | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=172.16.0.22 |
| 28e87a60-0ed7-41d6-8ea2-343afb3c4dba | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=172.16.0.23 |
| ef45f570-1e10-429e-aa63-d3c623bcf9b9 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=172.16.0.24 |
| d0d48130-53a0-46ac-a093-b60c32746067 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=172.16.0.25 |
+--------------------------------------+------------------------+--------+------------+-------------+----------------------+
[stack@undercloud ~]$ 


[stack@undercloud tempest]$ for i in 23 24 25; do ssh heat-admin@172.16.0.$i "sudo grep slo /etc/swift/proxy-server.conf"; donepipeline = catch_errors healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb slo ceilometer proxy-logging proxy-server
[filter:slo]
use = egg:swift#slo
pipeline = catch_errors healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb slo ceilometer proxy-logging proxy-server
[filter:slo]
use = egg:swift#slo
pipeline = catch_errors healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb slo ceilometer proxy-logging proxy-server
[filter:slo]
use = egg:swift#slo
[stack@undercloud tempest]$ 

------------------------------------------------

TEST RUN
--------
mabrams@optiplex:~$ ssh -l root seal14.qa.lab.tlv.redhat.com
Last login: Tue Apr 12 19:11:06 2016 from vpn-48-88.rdu2.redhat.com
[root@seal14 ~]# s^C
[root@seal14 ~]# ssh stack@10.0.0.67
stack@10.0.0.67's password: 
Last login: Tue Apr 12 16:01:25 2016 from 10.0.0.1
[stack@undercloud ~]$ sudo su -
Last login: ג' אפר 12 04:45:04 EDT 2016 on pts/0
[root@undercloud ~]# . /home/stack/overcloudrc 
[root@undercloud ~]# cd /etc/swift/t^C
[root@undercloud ~]# cd /home/^C
[root@undercloud ~]# logout
[stack@undercloud ~]$ . ./overcloudrc 
[stack@undercloud ~]$ cd tempest
[stack@undercloud tempest]$ tools/run-tests.sh -- tempest.api.object_storage.test_object_slo
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpLPV4mb
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list 
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover}  --load-list /tmp/tmpLPV4mb
{0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_delete_large_object [0.718254s] ... ok
{0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_list_large_object_metadata [0.432042s] ... ok
{0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_retrieve_large_object [0.507300s] ... ok
{0} tempest.api.object_storage.test_object_slo.ObjectSloTest.test_upload_manifest [0.392570s] ... ok

======
Totals
======
Ran: 4 tests in 9.0000 sec.
 - Passed: 4
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 2.0502 sec.

==============
Worker Balance
==============
 - Worker 0 (4 tests) => 0:00:02.054925
[stack@undercloud tempest]$ 

------------------------------------------------

Note You need to log in before you can comment on or make changes to this bug.