1717375 – [RHOSP15] Cinder's Google backup driver fails to backup a volume

Bug 1717375 - [RHOSP15] Cinder's Google backup driver fails to backup a volume

Summary: [RHOSP15] Cinder's Google backup driver fails to backup a volume

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-cinder
Sub Component:
Version:	15.0 (Stein)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Pablo Caruana
QA Contact:	Tzach Shefi
Docs Contact:	Tana
URL:
Whiteboard:
Depends On:
Blocks:	1809746
TreeView+	depends on / blocked

Reported:	2019-06-05 11:24 UTC by Tzach Shefi
Modified:	2020-03-05 11:54 UTC (History)
CC List:	1 user (show)
Fixed In Version:	openstack-cinder-14.0.2-0.20190927150437.47726fb.el8ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1809746 (view as bug list)
Environment:
Last Closed:	2020-03-05 11:54:36 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Logs (214.93 KB, application/zip) 2019-06-05 11:24 UTC, Tzach Shefi	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1831735	0	None	None	None	2019-06-05 11:41:44 UTC
OpenStack gerrit	676403	0	'None'	MERGED	Google backup: correct string encoding between py 2 and 3	2020-09-21 10:55:34 UTC

Description Tzach Shefi 2019-06-05 11:24:21 UTC

Created attachment 1577535 [details]
Logs

Description of problem: Cinder volume backup using Google as backup backend fails. We suspect this regression was introduced by move to Python3.  


Version-Release number of selected component (if applicable):
rhel8
puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch                                                                                                                                                                                  
openstack-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                               
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch                                                                                                                                                                            
python3-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                                 

python3-google-api-client-1.6.5-3.el8.noarch 


How reproducible:
Every time

Steps to Reproduce:
1. Configure Google backup:
Add  gcs-backup.yaml (attached) to overcloud deploy script. 
You also need to create Google credential file in my case  gbackup.json (can't share)

Deploy the overcloud, Cinder backup service is enabled and up
$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                    | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | controller-0            | nova | enabled | up    | 2019-06-05T11:17:47.000000 | -               |


2. Create a volume 
vol id -> 0336a629-25c4-4e05-be06-de51a75afc9d


3. If you try to backup the volume, backup fails
cinder backup logs reports the trace show below

$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| ID                                   | Volume ID                            | Status | Name | Size | Object Count | Container      |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| d256989b-a88f-4dcf-95d2-1c7afa12be76 | 0336a629-25c4-4e05-be06-de51a75afc9d | error  | -    | 1    | 0            | mycinderbucket |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+


e trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:170
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server [req-4f591f5c-031a-417e-974c-37cdb2ced102 c45b531877804ab097b8f587497031db ac390c9999244f56aa05cb13e181f0bc - default default] Exception during message handling: cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 121, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 312, in close
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     md5 = md5.encode('utf-8')
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server AttributeError: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 406, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self._update_backup_error(backup, six.text_type(err))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 395, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     updates = self._run_backup(context, backup, volume)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 479, in _run_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     backup, tpool.Proxy(device_file))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 625, in backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     object_meta, extra_metadata)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 374, in _backup_chunk
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     writer.write(output_data)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 221, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return proxy_call(self._autowrap, self._obj.__exit__, *exc)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 293, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.close()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 127, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise exception.GCSConnectionFailure(reason=err)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:57.656 21 DEBUG oslo_service.periodic_task [req-dbe4b355-7960-4100-a621-4fba6a5c05ac - - - - -] Running periodic task BackupManager._report_driver_status run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217

 
Actual results:
Cinder backup fails 

Expected results:
Cinder backup should successful complete the backup.
Least this worked on previous openstack release.    

Additional info:

Comment 9 errata-xmlrpc 2020-03-05 11:54:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0712

Note You need to log in before you can comment on or make changes to this bug.