Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2017:3462
Description of problem: Failing Tempest tests on basic director based deployment. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. 2. 3. Actual results: Tempest Basic Network Ops tests fail following from nova logs 2017-06-29 12:59:36.508 76698 DEBUG oslo_db.sqlalchemy.engines [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:280 2017-06-29 12:59:37.442 76685 DEBUG nova.osapi_compute.wsgi.server [req-4e29c8c8-f365-4676-b316-6385e86a6e7c - - - - -] (76685) accepted ('172.17.1.18', 42790) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2017-06-29 12:59:37.443 76685 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x7f3ee50>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612 2017-06-29 12:59:37.444 76685 INFO nova.osapi_compute.wsgi.server [-] 172.17.1.18 "OPTIONS / HTTP/1.0" status: 200 len: 499 time: 0.0013390 2017-06-29 12:59:37.563 76685 DEBUG nova.osapi_compute.wsgi.server [req-4e29c8c8-f365-4676-b316-6385e86a6e7c - - - - -] (76685) accepted ('172.17.1.18', 42798) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2017-06-29 12:59:37.572 76685 DEBUG nova.api.openstack.wsgi [req-42799951-de1a-46d9-bdc4-6d4ecddbb7be e3e5f14ac9604d4c9526da6cac1086ec e8179f4577354ceebc3d09cccffbe3b1 - default default] Action: 'create', calling method: <function version_select at 0xa89b5f0>, body: {"pool": "nova"} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:609 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] Failed to get metadata for instance id: e3fe962b-21de-4bce-95cb-67a48cf28e44: PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler Traceback (most recent call last): 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 285, in _get_meta_by_instance_id 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler remote_address) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 87, in get_metadata_by_instance_id 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler self._cache.set(cache_key, data) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler return self.region.set(key, value) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 973, in set 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler self.backend.set(key, self._value(value)) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler **self.set_arguments 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler return getattr(client, __name)(*args, **kwargs) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/memcache.py", line 656, in set 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler return self._set("set", key, val, time, min_compress_len) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/memcache.py", line 936, in _set 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler return _unsafe_set() 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/memcache.py", line 909, in _unsafe_set 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler store_info = self._val_to_store_info(val, min_compress_len) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/site-packages/memcache.py", line 877, in _val_to_store_info .... 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 663, in _batch_setitems 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler save(v) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 331, in save 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler self.save_reduce(obj=obj, *rv) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 419, in save_reduce 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler save(state) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 286, in save 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler f(self, obj) # Call unbound method with explicit self 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 649, in save_dict 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler self._batch_setitems(obj.iteritems()) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 663, in _batch_setitems 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler save(v) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 286, in save 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler f(self, obj) # Call unbound method with explicit self 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler File "/usr/lib64/python2.7/pickle.py", line 753, in save_global 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler (obj, module, name)) 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session 2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler 2017-06-29 12:59:37.662 76698 INFO nova.metadata.wsgi.server [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] 10.100.0.7,172.17.1.18 "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 229 time: 1.1885862 2017-06-29 12:59:39.408 76685 INFO nova.osapi_compute.wsgi.server [req-42799951-de1a-46d9-bdc4-6d4ecddbb7be e3e5f14ac9604d4c9526da6cac1086ec e8179f4577354ceebc3d09cccffbe3b1 - default default] 172.17.1.18 "POST /v2.1/os-floating-ips HTTP/1.1" status: 200 len: 532 time: 1.8437629 Expected results: tempest tests pass Additional info: