Bug 1809746 - [RHOSP15] Cinder's Google backup driver fails to backup a volume
Summary: [RHOSP15] Cinder's Google backup driver fails to backup a volume
Keywords:
Status: CLOSED EOL
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: z3
: 15.0 (Stein)
Assignee: Pablo Caruana
QA Contact: Tzach Shefi
Chuck Copello
URL:
Whiteboard:
Depends On: 1717375
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-03 19:23 UTC by Luigi Toscano
Modified: 2020-09-23 12:04 UTC (History)
4 users (show)

Fixed In Version: openstack-cinder-14.0.2-0.20190927150437.47726fb.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1717375
Environment:
Last Closed: 2020-09-23 12:04:11 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Luigi Toscano 2020-03-03 19:23:52 UTC
+++ This bug was initially created as a clone of Bug #1717375 +++

Description of problem: Cinder volume backup using Google as backup backend fails. We suspect this regression was introduced by move to Python3.  


Version-Release number of selected component (if applicable):
rhel8
puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch                                                                                                                                                                                  
openstack-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                               
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch                                                                                                                                                                            
python3-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                                 

python3-google-api-client-1.6.5-3.el8.noarch 


How reproducible:
Every time

Steps to Reproduce:
1. Configure Google backup:
Add  gcs-backup.yaml (attached) to overcloud deploy script. 
You also need to create Google credential file in my case  gbackup.json (can't share)

Deploy the overcloud, Cinder backup service is enabled and up
$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                    | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | controller-0            | nova | enabled | up    | 2019-06-05T11:17:47.000000 | -               |


2. Create a volume 
vol id -> 0336a629-25c4-4e05-be06-de51a75afc9d


3. If you try to backup the volume, backup fails
cinder backup logs reports the trace show below

$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| ID                                   | Volume ID                            | Status | Name | Size | Object Count | Container      |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| d256989b-a88f-4dcf-95d2-1c7afa12be76 | 0336a629-25c4-4e05-be06-de51a75afc9d | error  | -    | 1    | 0            | mycinderbucket |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+


e trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:170
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server [req-4f591f5c-031a-417e-974c-37cdb2ced102 c45b531877804ab097b8f587497031db ac390c9999244f56aa05cb13e181f0bc - default default] Exception during message handling: cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 121, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 312, in close
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     md5 = md5.encode('utf-8')
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server AttributeError: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 406, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self._update_backup_error(backup, six.text_type(err))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 395, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     updates = self._run_backup(context, backup, volume)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 479, in _run_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     backup, tpool.Proxy(device_file))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 625, in backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     object_meta, extra_metadata)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 374, in _backup_chunk
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     writer.write(output_data)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 221, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return proxy_call(self._autowrap, self._obj.__exit__, *exc)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 293, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.close()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 127, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise exception.GCSConnectionFailure(reason=err)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:57.656 21 DEBUG oslo_service.periodic_task [req-dbe4b355-7960-4100-a621-4fba6a5c05ac - - - - -] Running periodic task BackupManager._report_driver_status run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217

 
Actual results:
Cinder backup fails 

Expected results:
Cinder backup should successful complete the backup.
Least this worked on previous openstack release.    

Additional info:

Comment 3 Lon Hohberger 2020-03-06 11:38:14 UTC
According to our records, this should be resolved by openstack-cinder-14.0.4-0.20200107100455.a59c01e.el8ost.  This build is available now.

Comment 5 Luigi Toscano 2020-09-23 12:04:11 UTC
OSP15 is EOL, and this case is covered by regression testing on all releases.


Note You need to log in before you can comment on or make changes to this bug.