1466378 – nova metadata handler pickle error

Bug 1466378 - nova metadata handler pickle error

Summary: nova metadata handler pickle error

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	Upstream M3
Target Release:	12.0 (Pike)
Assignee:	Artom Lifshitz
QA Contact:	Archit Modi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-06-29 13:47 UTC by Jon Schlueter
Modified:	2019-09-09 17:18 UTC (History)
CC List:	14 users (show)
Fixed In Version:	openstack-nova-16.0.0-0.20170624031429.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-12-13 21:35:30 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1694666	None	None	None	2017-06-29 13:47:32 UTC
OpenStack gerrit	478991	None	None	None	2017-06-29 20:47:57 UTC
Red Hat Product Errata	RHEA-2017:3462	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-16 01:43:25 UTC

Description Jon Schlueter 2017-06-29 13:47:03 UTC

Description of problem:
Failing Tempest tests on basic director based deployment.


Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:
Tempest Basic Network Ops tests fail

following from nova logs
2017-06-29 12:59:36.508 76698 DEBUG oslo_db.sqlalchemy.engines [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:280
2017-06-29 12:59:37.442 76685 DEBUG nova.osapi_compute.wsgi.server [req-4e29c8c8-f365-4676-b316-6385e86a6e7c - - - - -] (76685) accepted ('172.17.1.18', 42790) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-06-29 12:59:37.443 76685 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x7f3ee50>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:612
2017-06-29 12:59:37.444 76685 INFO nova.osapi_compute.wsgi.server [-] 172.17.1.18 "OPTIONS / HTTP/1.0" status: 200 len: 499 time: 0.0013390
2017-06-29 12:59:37.563 76685 DEBUG nova.osapi_compute.wsgi.server [req-4e29c8c8-f365-4676-b316-6385e86a6e7c - - - - -] (76685) accepted ('172.17.1.18', 42798) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-06-29 12:59:37.572 76685 DEBUG nova.api.openstack.wsgi [req-42799951-de1a-46d9-bdc4-6d4ecddbb7be e3e5f14ac9604d4c9526da6cac1086ec e8179f4577354ceebc3d09cccffbe3b1 - default default] Action: 'create', calling method: <function version_select at 0xa89b5f0>, body: {"pool": "nova"} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:609
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] Failed to get metadata for instance id: e3fe962b-21de-4bce-95cb-67a48cf28e44: PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler Traceback (most recent call last):
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 285, in _get_meta_by_instance_id
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     remote_address)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 87, in get_metadata_by_instance_id
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self._cache.set(cache_key, data)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return self.region.set(key, value)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 973, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self.backend.set(key, self._value(value))
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     **self.set_arguments
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return getattr(client, __name)(*args, **kwargs)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 656, in set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return self._set("set", key, val, time, min_compress_len)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 936, in _set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     return _unsafe_set()
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 909, in _unsafe_set
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     store_info = self._val_to_store_info(val, min_compress_len)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib/python2.7/site-packages/memcache.py", line 877, in _val_to_store_info
 
 
 
....
 
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 663, in _batch_setitems
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     save(v)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 331, in save
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self.save_reduce(obj=obj, *rv)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 419, in save_reduce
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     save(state)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 286, in save
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     f(self, obj) # Call unbound method with explicit self
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 649, in save_dict
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     self._batch_setitems(obj.iteritems())
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 663, in _batch_setitems
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     save(v)
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 286, in save
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     f(self, obj) # Call unbound method with explicit self
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler   File "/usr/lib64/python2.7/pickle.py", line 753, in save_global
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler     (obj, module, name))
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
2017-06-29 12:59:37.658 76698 ERROR nova.api.metadata.handler
2017-06-29 12:59:37.662 76698 INFO nova.metadata.wsgi.server [req-4ab78206-2e39-484f-8935-a16fce4a3c22 - - - - -] 10.100.0.7,172.17.1.18 "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 229 time: 1.1885862
2017-06-29 12:59:39.408 76685 INFO nova.osapi_compute.wsgi.server [req-42799951-de1a-46d9-bdc4-6d4ecddbb7be e3e5f14ac9604d4c9526da6cac1086ec e8179f4577354ceebc3d09cccffbe3b1 - default default] 172.17.1.18 "POST /v2.1/os-floating-ips HTTP/1.1" status: 200 len: 532 time: 1.8437629


Expected results:

tempest tests pass

Additional info:

Comment 7 errata-xmlrpc 2017-12-13 21:35:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.