Bug 2230602
| Summary: | [RHOS-17.1] Resizing compute with NVMeOF Cinder backend intermittently fails to find volume | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | James Parker <jparker> |
| Component: | python-os-brick | Assignee: | Cinder Bugs List <cinder-bugs> |
| Status: | NEW --- | QA Contact: | Evelina Shames <eshames> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 17.1 (Wallaby) | CC: | apevec, brian.rosmaita, eharney, geguileo, jschluet, lhh, ltoscano |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Marking it low priority; let's keep an eye on this issue. |
Description of problem: Resizing a guest with NVMeOF Cinder backend in intermittently failing in phase3 CI. Test specifically triggering this failure [1]. Full logs, deployment details, and job details will be included in follow up comment. Please advise if the attempted actions are unsupported for the environment. 2023-08-03 02:01:55.660 18 DEBUG placement.objects.research_context [req-37c55e62-a4c1-41d6-be37-e83b9dbe7364 d296552183034fba8b15267e5b19b68d ddccfaba37ea41a4bc85a4a418f8f9e2 - default default] found 2 providers after filtering by previous result get_provider_ids_matching /usr/lib/python3.9/site-packages/placement/objects/research_context.py:580 2023-08-03 02:01:55.661 2 DEBUG os_brick.initiator.connectors.base [req-a9e03d62-5284-4fbf-bdc8-c2bcc78d4703 571ccebe0b70413691998e6cac3046ca e40a3908acf04d27b9e8f4bbbe5005ff - default default] Lock "connect_volume" "released" by "os_brick.initiator.connectors.nvmeof.NVMeOFConnector.connect_volume" :: held 6.036s inner /usr/lib/python3.9/site-packages/os_brick/initiator/connectors/base.py:83 2023-08-03 02:01:55.661 2 DEBUG os_brick.initiator.connectors.nvmeof [req-a9e03d62-5284-4fbf-bdc8-c2bcc78d4703 571ccebe0b70413691998e6cac3046ca e40a3908acf04d27b9e8f4bbbe5005ff - default default] <== connect_volume: exception (6037ms) VolumeDeviceNotFound('Volume device not found at nqn.nvme-subsystem-compute-0.redhat.local.') trace_logging_wrapper /usr/lib/python3.9/site-packages/os_brick/utils.py:176 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [req-a9e03d62-5284-4fbf-bdc8-c2bcc78d4703 571ccebe0b70413691998e6cac3046ca e40a3908acf04d27b9e8f4bbbe5005ff - default default] [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] Setting instance vm_state to ERROR: os_brick.exception.VolumeDeviceNotFound: Volume device not found at nqn.nvme-subsystem-compute-0.redhat.local. 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] Traceback (most recent call last): 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10256, in _error_out_instance_on_exception 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] yield 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5851, in _finish_resize_helper 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] network_info = self._finish_resize(context, instance, migration, ....<REMOVED FOR BREVITY>.... 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] result = fn(*args, **kwargs) 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/nvmeof.py", line 908, in _connect_target 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] raise exception.VolumeDeviceNotFound(device=target.nqn) 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] os_brick.exception.VolumeDeviceNotFound: Volume device not found at nqn.nvme-subsystem-compute-0.redhat.local. 2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20230712.n.1 How reproducible: 50% Steps to Reproduce: 1. Deploy 17.1 with NVMeOF Cinder backend enabled 2. Create a server and volume and attach the volume to the server 3. Resize the server Actual results: Server resize is failing due to attach volume not being found during resize process Expected results: server resize action is successful Additional info: [1] https://github.com/openstack/tempest/blob/34.1.0/tempest/api/compute/servers/test_server_actions.py#L459C1-L459C58