Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1809746

Summary: [RHOSP15] Cinder's Google backup driver fails to backup a volume
Product: Red Hat OpenStack Reporter: Luigi Toscano <ltoscano>
Component: openstack-cinderAssignee: Pablo Caruana <pcaruana>
Status: CLOSED EOL QA Contact: Tzach Shefi <tshefi>
Severity: high Docs Contact: Chuck Copello <ccopello>
Priority: medium    
Version: 15.0 (Stein)CC: jschluet, pcaruana, tberry, tshefi
Target Milestone: z3Keywords: Regression, TestOnly, Triaged, ZStream
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-14.0.2-0.20190927150437.47726fb.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1717375 Environment:
Last Closed: 2020-09-23 12:04:11 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1717375    
Bug Blocks:    

Description Luigi Toscano 2020-03-03 19:23:52 UTC 
+++ This bug was initially created as a clone of Bug #1717375 +++

Description of problem: Cinder volume backup using Google as backup backend fails. We suspect this regression was introduced by move to Python3.  


Version-Release number of selected component (if applicable):
rhel8
puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch                                                                                                                                                                                  
openstack-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                               
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch                                                                                                                                                                            
python3-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch                                                                                                                                                                                 

python3-google-api-client-1.6.5-3.el8.noarch 


How reproducible:
Every time

Steps to Reproduce:
1. Configure Google backup:
Add  gcs-backup.yaml (attached) to overcloud deploy script. 
You also need to create Google credential file in my case  gbackup.json (can't share)

Deploy the overcloud, Cinder backup service is enabled and up
$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                    | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | controller-0            | nova | enabled | up    | 2019-06-05T11:17:47.000000 | -               |


2. Create a volume 
vol id -> 0336a629-25c4-4e05-be06-de51a75afc9d


3. If you try to backup the volume, backup fails
cinder backup logs reports the trace show below

$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| ID                                   | Volume ID                            | Status | Name | Size | Object Count | Container      |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| d256989b-a88f-4dcf-95d2-1c7afa12be76 | 0336a629-25c4-4e05-be06-de51a75afc9d | error  | -    | 1    | 0            | mycinderbucket |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+


e trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:170
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server [req-4f591f5c-031a-417e-974c-37cdb2ced102 c45b531877804ab097b8f587497031db ac390c9999244f56aa05cb13e181f0bc - default default] Exception during message handling: cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 121, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 312, in close
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     md5 = md5.encode('utf-8')
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server AttributeError: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 406, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self._update_backup_error(backup, six.text_type(err))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 395, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     updates = self._run_backup(context, backup, volume)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 479, in _run_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     backup, tpool.Proxy(device_file))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 625, in backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     object_meta, extra_metadata)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 374, in _backup_chunk
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     writer.write(output_data)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 221, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     return proxy_call(self._autowrap, self._obj.__exit__, *exc)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 293, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     self.close()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 127, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server     raise exception.GCSConnectionFailure(reason=err)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server 
2019-06-05 09:25:57.656 21 DEBUG oslo_service.periodic_task [req-dbe4b355-7960-4100-a621-4fba6a5c05ac - - - - -] Running periodic task BackupManager._report_driver_status run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217

 
Actual results:
Cinder backup fails 

Expected results:
Cinder backup should successful complete the backup.
Least this worked on previous openstack release.    

Additional info:
Comment 3 Lon Hohberger 2020-03-06 11:38:14 UTC 
According to our records, this should be resolved by openstack-cinder-14.0.4-0.20200107100455.a59c01e.el8ost.  This build is available now.
Comment 5 Luigi Toscano 2020-09-23 12:04:11 UTC 
OSP15 is EOL, and this case is covered by regression testing on all releases.