Attachment 1981418 Details for Bug 2228733 – setup, cli output, nova, cinder logs

setup, cli output, nova, cinder logs

bug.txt (text/plain), 18.15 KB, created by jon.hartley@btinternet.com on 2023-08-03 06:46:22 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: jon.hartley@btinternet.com

Created: 2023-08-03 06:46:22 UTC

Size: 18.15 KB

patch

obsolete

>packstack --allinone \
>  --default-password=openstack \
>  --keystone-db-passwd=openstack \
>  --glance-db-passwd=openstack \
>  --cinder-db-passwd=openstack \
>  --nova-db-passwd=openstack \
>  --keystone-admin-passwd='874jvcbhdh847sbjv93jhfkso39' \
>  --cinder-volumes-create=n \
>  --cinder-db-purge-enable=y \
>  --cinder-backend=lvm \
>  --os-swift-install=n \
>  --os-swift-storage-fstype=xfs \
>  --os-swift-storages=/dev/swift/swift \
>  --os-client-install=y \
>  --provision-demo=n \
>  --os-neutron-ovs-bridge-mappings=extnet:br-ex \
>  --os-neutron-ovs-bridge-interfaces=br-ex:bond0 \
>  --os-neutron-ml2-mechanism-drivers=openvswitch \
>  --os-neutron-ml2-tenant-network-types=vxlan \
>  --os-neutron-ml2-type-drivers=vxlan,flat,vlan \
>  --os-neutron-l2-agent=openvswitch \
>  --nova-libvirt-virt-type=kvm \
>  --novasched-cpu-allocation-ratio=2.0 \
>  --novasched-ram-allocation-ratio=2.0 \
>  --nova-manage-flavors=n \
>  --glance-backend=file \
>  --enable-rdo-testing=n \
>  --os-horizon-ssl=y \
>  --amqp-enable-ssl=y \
>  --amqp-enable-auth=y \
>  --ssl-cacert-selfsign=y \
>  --ssl-cert-subject-country=GB \
>  --ssl-cert-subject-state=Yorkshire \
>  --ssl-cert-subject-location=Leeds \
>  --ssl-cert-subject-organization=Home \
>  --ssl-cert-subject-organizational-unit=Jon \
>  --ssl-cert-subject-common-name=openstack.5kg.os \
>  --ssl-cert-subject-email=jon.hartley@btinternet.com \
>  --nova-db-purge-enable=y \
>  --os-aodh-install=n \
>  --os-manila-install=n \
>  --os-ceilometer-install=n \
>  --os-sahara-install=n \
>  --os-heat-install=n \
>  --os-magnum-install=n \
>  --os-trove-install=n \
>  --os-ironic-install=n \
>  --os-heat-cfn-install=n
>
>[root@openstack ~(keystone_jonh)]$ openstack server list
>+--------------------------------------+-------+--------+--------------------------------+--------------------------+---------+
>| ID                                   | Name  | Status | Networks                       | Image                    | Flavor  |
>+--------------------------------------+-------+--------+--------------------------------+--------------------------+---------+
>| d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 | kube1 | ACTIVE | external_network=192.168.3.213 | N/A (booted from volume) | c4p.m20 |
>+--------------------------------------+-------+--------+--------------------------------+--------------------------+---------+
>[root@openstack ~(keystone_jonh)]$ openstack volume list
>+--------------------------------------+-----------------+-----------+------+--------------------------------+
>| ID                                   | Name            | Status    | Size | Attached to                    |
>+--------------------------------------+-----------------+-----------+------+--------------------------------+
>| 825caa38-2d3b-4bff-8e91-9e46e85244fc | kube1_extra_vol | available |   10 |                                |
>| 48ee91cf-1525-445a-94fe-6baf5a81b46c | kube1_vol       | in-use    |   10 | Attached to kube1 on /dev/vda  |
>+--------------------------------------+-----------------+-----------+------+--------------------------------+
>[root@openstack ~(keystone_jonh)]$ openstack server add volume d7e390e7-c1e5-4f6e-b700-18cb6cf411d7  825caa38-2d3b-4bff-8e91-9e46e85244fc
>+-----------------------+--------------------------------------+
>| Field                 | Value                                |
>+-----------------------+--------------------------------------+
>| ID                    | 825caa38-2d3b-4bff-8e91-9e46e85244fc |
>| Server ID             | d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 |
>| Volume ID             | 825caa38-2d3b-4bff-8e91-9e46e85244fc |
>| Device                | /dev/vdb                             |
>| Tag                   | None                                 |
>| Delete On Termination | False                                |
>+-----------------------+--------------------------------------+
>[root@openstack ~(keystone_jonh)]$ openstack volume list
>+--------------------------------------+-----------------+--------+------+--------------------------------+
>| ID                                   | Name            | Status | Size | Attached to                    |
>+--------------------------------------+-----------------+--------+------+--------------------------------+
>| 825caa38-2d3b-4bff-8e91-9e46e85244fc | kube1_extra_vol | in-use |   10 | Attached to kube1 on /dev/vdb  |
>| 48ee91cf-1525-445a-94fe-6baf5a81b46c | kube1_vol       | in-use |   10 | Attached to kube1 on /dev/vda  |
>+--------------------------------------+-----------------+--------+------+--------------------------------+
>[root@openstack ~(keystone_jonh)]$ openstack server remove volume d7e390e7-c1e5-4f6e-b700-18cb6cf411d7  825caa38-2d3b-4bff-8e91-9e46e85244fc
>[root@openstack ~(keystone_jonh)]$ openstack volume list
>+--------------------------------------+-----------------+-----------+------+--------------------------------+
>| ID                                   | Name            | Status    | Size | Attached to                    |
>+--------------------------------------+-----------------+-----------+------+--------------------------------+
>| 825caa38-2d3b-4bff-8e91-9e46e85244fc | kube1_extra_vol | detaching |   10 | Attached to kube1 on /dev/vdb  |
>| 48ee91cf-1525-445a-94fe-6baf5a81b46c | kube1_vol       | in-use    |   10 | Attached to kube1 on /dev/vda  |
>+--------------------------------------+-----------------+-----------+------+--------------------------------+
>
>nova logs for detachment
>
>2023-08-03 06:29:18.359 3104 INFO nova.compute.manager [None req-fb420385-8c15-4fbe-9e8f-4607b0f26d03 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - default default] [instance: d7e390e7-c1e5-4f6e-b700-18cb6cf411d7] Detaching volume 825caa38-2d3b-4bff-8e91-9e46e85244fc
>2023-08-03 06:29:18.442 3104 INFO nova.virt.block_device [None req-fb420385-8c15-4fbe-9e8f-4607b0f26d03 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - default default] [instance: d7e390e7-c1e5-4f6e-b700-18cb6cf411d7] Attempting to driver detach volume 825caa38-2d3b-4bff-8e91-9e46e85244fc from mountpoint /dev/vdb
>2023-08-03 06:29:18.459 3104 INFO nova.virt.libvirt.driver [None req-fb420385-8c15-4fbe-9e8f-4607b0f26d03 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - default default] Successfully detached device vdb from instance d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 from the persistent domain config.
>2023-08-03 06:29:18.596 3104 INFO nova.virt.libvirt.driver [None req-fb420385-8c15-4fbe-9e8f-4607b0f26d03 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - default default] Successfully detached device vdb from instance d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 from the live domain config.
>2023-08-03 06:29:18.598 3104 WARNING os_brick.initiator.connectors.base [None req-fb420385-8c15-4fbe-9e8f-4607b0f26d03 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - default default] Service needs to call os_brick.setup() before connecting volumes, if it doesn't it will break on the next release
>2023-08-03 06:29:20.013 3104 ERROR nova.volume.cinder [None req-fb420385-8c15-4fbe-9e8f-4607b0f26d03 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - default default] Delete attachment failed for attachment 41d083f2-38ca-41c8-aae3-90f6f3133dde. Error: ConflictNovaUsingAttachment: Detach volume from instance d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 using the Compute API (HTTP 409) (Request-ID: req-9234f7b9-ce8e-41e0-88e5-d3c2bac68844) Code: 409: cinderclient.exceptions.ClientException: ConflictNovaUsingAttachment: Detach volume from instance d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 using the Compute API (HTTP 409) (Request-ID: req-9234f7b9-ce8e-41e0-88e5-d3c2bac68844)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server [None req-fb420385-8c15-4fbe-9e8f-4607b0f26d03 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - default default] Exception during message handling: cinderclient.exceptions.ClientException: ConflictNovaUsingAttachment: Detach volume from instance d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 using the Compute API (HTTP 409) (Request-ID: req-9234f7b9-ce8e-41e0-88e5-d3c2bac68844)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/exception_wrapper.py", line 71, in wrapped
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     _emit_versioned_exception_notification(
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     raise self.value
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/exception_wrapper.py", line 63, in wrapped
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/utils.py", line 1439, in decorated_function
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 214, in decorated_function
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     compute_utils.add_instance_fault_from_exc(context,
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     raise self.value
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 203, in decorated_function
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 7603, in detach_volume
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     do_detach_volume(context, volume_id, instance, attachment_id)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 7600, in do_detach_volume
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     self._detach_volume(context, bdm, instance,
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 7551, in _detach_volume
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     driver_bdm.detach(context, instance, self.volume_api, self.driver,
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/virt/block_device.py", line 538, in detach
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     self._do_detach(context, instance, volume_api, virt_driver,
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/virt/block_device.py", line 519, in _do_detach
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 397, in wrapper
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 451, in wrapper
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/retrying.py", line 49, in wrapped_f
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return Retrying(*dargs, **dkw).call(f, *args, **kw)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/retrying.py", line 206, in call
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return attempt.get(self._wrap_exception)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/retrying.py", line 247, in get
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     raise value
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/retrying.py", line 200, in call
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 905, in attachment_delete
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     LOG.error('Delete attachment failed for attachment '
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     raise self.value
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 896, in attachment_delete
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     cinderclient(
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in substitution
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return method.func(obj, *args, **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 45, in delete
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 313, in _delete
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 229, in delete
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 211, in _cs_request
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 197, in request
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: ConflictNovaUsingAttachment: Detach volume from instance d7e390e7-c1e5-4f6e-b700-18cb6cf411d7 using the Compute API (HTTP 409) (Request-ID: req-9234f7b9-ce8e-41e0-88e5-d3c2bac68844)
>2023-08-03 06:29:20.065 3104 ERROR oslo_messaging.rpc.server
>
>
>cinder log for attachment (nothing for attempt to detach ?)
>2023-08-02 20:37:34.533 2694 INFO cinder.volume.manager [None req-88102204-8127-4001-aefe-8dd45dbbb799 - - - - - -] Initializing RPC dependent components of volume driver LVMVolumeDriver (3.0.0)
>2023-08-02 20:37:37.924 2694 INFO cinder.volume.manager [None req-88102204-8127-4001-aefe-8dd45dbbb799 - - - - - -] Driver post RPC initialization completed successfully.
>2023-08-03 06:28:33.237 2694 INFO cinder.volume.targets.lio [req-9a53daa0-7f1d-45e3-b6f4-8611bcb667d0 req-b8dcaa04-2e6f-4d35-aabc-3e7dae2bb885 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - - -] Creating iscsi_target for volume: volume-825caa38-2d3b-4bff-8e91-9e46e85244fc
>2023-08-03 06:28:37.800 2694 INFO cinder.volume.manager [req-9a53daa0-7f1d-45e3-b6f4-8611bcb667d0 req-b8dcaa04-2e6f-4d35-aabc-3e7dae2bb885 c2200aacea5f45a98144e0f15c7e6925 b102f2e227274d0fb7aa3c301d537eb9 - - - -] attachment_update completed successfully.
>

Actions: View

Attachments on bug 2228733: 1981418