Bug 1488304 - cinder-backup in containerized ipv6 setup unable to create backups
Summary: cinder-backup in containerized ipv6 setup unable to create backups
Keywords:
Status: CLOSED DUPLICATE of bug 1484467
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-05 06:07 UTC by Attila Fazekas
Modified: 2018-08-03 18:32 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-05 13:41:11 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Attila Fazekas 2017-09-05 06:07:56 UTC
Description of problem:

cinder-backup service unable to access the volumes on ipv6 controller.

Version-Release number of selected component (if applicable):
rhosp12/openstack-iscsid-docker:2017-08-31.2 
openstack-tripleo-heat-templates-7.0.0-0.20170821194254.el7ost

How reproducible:
always

Steps to Reproduce:
- ipv6, 3 controller, 1 compute setup, lvm/iscsi
- run tempest backup tests

Additional info:
- ipv4 version works
- non-containerized version works


c-bak log:

packages/oslo_concurrency/processutils.py:355
2017-09-05 00:11:13.222 242976 DEBUG oslo_concurrency.processutils [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] CMD "iscsiadm -m node" returned: 21 in 0.005s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:385
2017-09-05 00:11:13.222 242976 DEBUG oslo.privsep.daemon [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] privsep: reply[110782256]: (4, ('', 'iscsiadm: No records found\n')) loop /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:456
2017-09-05 00:11:13.223 120862 WARNING os_brick.initiator.connectors.iscsi [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found
: VolumeDeviceNotFound: Volume device not found at .
2017-09-05 00:11:13.224 242976 DEBUG oslo.privsep.daemon [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] privsep: request[110782256]: (3, 'os_brick.privileged.rootwrap.execute_root', ('iscsiadm', '-m', 'session'), {'check_exit_code': (0, 1, 21, 255)}) loop /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:443
2017-09-05 00:11:13.224 242976 DEBUG oslo_concurrency.processutils [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] Running cmd (subprocess): iscsiadm -m session execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:355
2017-09-05 00:11:13.230 242976 DEBUG oslo_concurrency.processutils [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] CMD "iscsiadm -m session" returned: 21 in 0.006s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:385
2017-09-05 00:11:13.230 242976 DEBUG oslo.privsep.daemon [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] privsep: reply[110782256]: (4, ('', 'iscsiadm: No active sessions.\n')) loop /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:456
2017-09-05 00:11:13.231 120862 DEBUG os_brick.initiator.connectors.iscsi [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] iscsiadm ('-m', 'session'): stdout= stderr=iscsiadm: No active sessions.
 _run_iscsiadm_bare /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1077
2017-09-05 00:11:13.231 120862 DEBUG os_brick.initiator.connectors.iscsi [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] iscsi session list stdout= stderr=iscsiadm: No active sessions.
 _run_iscsi_session /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1066
2017-09-05 00:11:13.232 120862 WARNING os_brick.initiator.connectors.iscsi [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] Couldn't find iscsi sessions because iscsiadm err: iscsiadm: No active sessions.
: VolumeDeviceNotFound: Volume device not found at .
2017-09-05 00:11:13.232 120862 DEBUG os_brick.initiator.connectors.iscsi [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] Resulting device map defaultdict(<function <lambda> at 0x6a1bde8>, {}) _get_connection_devices /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:812
2017-09-05 00:11:13.232 120862 DEBUG os_brick.initiator.connectors.iscsi [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] Disconnecting from: [] _disconnect_connection /usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py:1054
2017-09-05 00:11:13.233 120862 DEBUG oslo_concurrency.lockutils [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] Lock "connect_volume" released by "os_brick.initiator.connectors.iscsi.connect_volume" :: held 6.479s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-09-05 00:11:13.233 120862 DEBUG os_brick.initiator.connectors.iscsi [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] <== connect_volume: exception (6480ms) VolumeDeviceNotFound(u'Volume device not found at .',) trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:156
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server [req-4fad01c1-20ff-47ee-9065-5a2d4d2c3248 f1991001fc504cb78e1343b326ae4906 b5f0878198214837b753d315cc16ac18 - default default] Exception during message handling: VolumeDeviceNotFound: Volume device not found at .
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 358, in create_backup
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     self._update_backup_error(backup, six.text_type(err))
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 347, in create_backup
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     updates = self._run_backup(context, backup, volume)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 396, in _run_backup
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     backup_device.is_snapshot)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 882, in _attach_device
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     return self._attach_volume(ctxt, backup_device, properties)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 904, in _attach_volume
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     {'volume_id', volume.id})
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 893, in _attach_volume
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     return self._connect_device(conn)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/backup/manager.py", line 934, in _connect_device
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     vol_handle = connector.connect_volume(conn['data'])
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 150, in trace_logging_wrapper
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 506, in connect_volume
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     self._cleanup_connection(connection_properties, force=True)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 500, in connect_volume
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     return self._connect_single_volume(connection_properties)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 61, in _wrapper
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     return r.call(f, *args, **kwargs)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     raise attempt.get()
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 575, in _connect_single_volume
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server     raise exception.VolumeDeviceNotFound(device='')
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server VolumeDeviceNotFound: Volume device not found at .
2017-09-05 00:11:14.941 120862 ERROR oslo_messaging.rpc.server

Comment 2 Gorka Eguileor 2017-09-05 13:41:11 UTC
The problem here is that iscsid is containerized in the controller while Cinder isn't, so they don't share the same iSCSI initiator name and Cinder's authorizes the Cinder backup's initiator name (which is the hosts) but the the iscsid container tries to do the actual connection (with a different initiator name).

*** This bug has been marked as a duplicate of bug 1484467 ***


Note You need to log in before you can comment on or make changes to this bug.