Bug 1466378 - nova metadata handler pickle error
nova metadata handler pickle error
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
urgent Severity urgent
: Upstream M3
: 12.0 (Pike)
Assigned To: Artom Lifshitz
Archit Modi
: AutomationBlocker, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-29 09:47 EDT by Jon Schlueter
Modified: 2017-12-13 16:35 EST (History)
15 users (show)

See Also:
Fixed In Version: openstack-nova-16.0.0-0.20170624031429.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:35:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1694666 None None None 2017-06-29 09:47 EDT
OpenStack gerrit 478991 None None None 2017-06-29 16:47 EDT

  None (edit)
Description Jon Schlueter 2017-06-29 09:47:03 EDT
Description of problem:
Failing Tempest tests on basic director based deployment.


Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:
Tempest Basic Network Ops tests fail

following from nova logs
2017-06-29 12:59:36.508 76698 DEBUG oslo_db.sqlalchemy.engines [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:280
2017-06-29 12:59:37.442 76685 DEBUG nova.osapi_compute.wsgi.server [req-4e29c8c8-f365-4676-b316-6385e86a6e7c - - - - -] (76685) accepted ('172.17.1.18', 42790) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-06-29 12:59:37.443 76685 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x7f3ee50>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-06-29 12:59:37.444 76685 INFO nova.osapi_compute.wsgi.server [-] 172.17.1.18 "OPTIONS / HTTP/1.0" status: 200 len: 499 time: 0.0013390
2017-06-29 12:59:37.563 76685 DEBUG nova.osapi_compute.wsgi.server [req-4e29c8c8-f365-4676-b316-6385e86a6e7c - - - - -] (76685) accepted ('172.17.1.18', 42798) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-06-29 12:59:37.572 76685 DEBUG nova.api.openstack.wsgi [req-42799951-de1a-46d9-bdc4-6d4ecddbb7be e3e5f14ac9604d4c9526da6cac1086ec e8179f4577354ceebc3d09cccffbe3b1 - default default] Action: 'create', calling method: <function version_select at 0xa89b5f0>, body: {"pool": "nova"} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:609
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] Failed to get metadata for instance id: e3fe962b-21de-4bce-95cb-67a48cf28e44: PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler Traceback (most recent call last):
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 285, in _get_meta_by_instance_id
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     remote_address)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 87, in get_metadata_by_instance_id
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self._cache.set(cache_key, data)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return self.region.set(key, value)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 973, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self.backend.set(key, self._value(value))
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     **self.set_arguments
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return getattr(client, __name)(*args, **kwargs)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 656, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return self._set("set", key, val, time, min_compress_len)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 936, in _set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return _unsafe_set()
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 909, in _unsafe_set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     store_info = self._val_to_store_info(val, min_compress_len)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 877, in _val_to_store_info
 
 
 
....
 
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 663, in _batch_setitems
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     save(v)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 331, in save
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self.save_reduce(obj=obj, *rv)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 419, in save_reduce
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     save(state)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 286, in save
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     f(self, obj) # Call unbound method with explicit self
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 649, in save_dict
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self._batch_setitems(obj.iteritems())
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 663, in _batch_setitems
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     save(v)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 286, in save
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     f(self, obj) # Call unbound method with explicit self
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 753, in save_global
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     (obj, module, name))
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler
2017-06-29 12:59:37.662 76698 INFO nova.metadata.wsgi.server [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] 10.100.0.7,172.17.1.18 "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 229 time: 1.1885862
2017-06-29 12:59:39.408 76685 INFO nova.osapi_compute.wsgi.server [req-42799951-de1a-46d9-bdc4-6d4ecddbb7be e3e5f14ac9604d4c9526da6cac1086ec e8179f4577354ceebc3d09cccffbe3b1 - default default] 172.17.1.18 "POST /v2.1/os-floating-ips HTTP/1.1" status: 200 len: 532 time: 1.8437629


Expected results:

tempest tests pass

Additional info:
Comment 7 errata-xmlrpc 2017-12-13 16:35:30 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.