Bug 1964318

Summary:	EMC VMAX volume backend VolumeDeviceNotFound during Find Multipath device
Product:	Red Hat OpenStack	Reporter:	Luigi Tamagnone <ltamagno>
Component:	openstack-cinder	Assignee:	Cinder Bugs List <cinder-bugs>
Status:	CLOSED EOL	QA Contact:	Evelina Shames <eshames>
Severity:	high	Docs Contact:	Andy Stillman <astillma>
Priority:	high
Version:	13.0 (Queens)	CC:	astillma, pcaruana
Target Milestone:	---	Keywords:	OtherQA, Triaged, ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-07-10 19:48:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Luigi Tamagnone 2021-05-25 08:13:45 UTC

Description of problem:
The instance creation fails with "exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8e22b053-ccf0-47dd-b968-93f61d4ee22e" The same volume on other compute nodes works.
This issue happens for every volume from VMAX backend
The issue could be bind with BUG 1942079 and 1964078. But I'm not sure.

Version-Release number of selected component (if applicable):
[redhat-release] Red Hat Enterprise Linux Server release 7.7 (Maipo)
[rhosp-release] Red Hat OpenStack Platform release 13.0.10 (Queens)

How reproducible:
The customer can reproduce the error every time they try to create an instance from a volume.

Steps to Reproduce:
1. create volume in VMAX backend from an image
2. Create an instance from that volume

Actual results:
Instance creation fails

Expected results:
instance creation succeed

Additional info:
From nova.log on cmpc23 we can see log about searching Multipath device:
2021-05-21 15:32:17.741 8 INFO os_brick.initiator.linuxscsi [req-95e3cfc2-7647-4c65-baa8-84ea9345c8a9 ad9a17ce8088485ca950656ba84345b5 ff9c71872c20417e83e4387fccf85434 - default default] Find Multipath device file for volume WWN 360000970000297802096533030373435
2021-05-21 15:32:17.742 8 DEBUG os_brick.initiator.linuxscsi [req-95e3cfc2-7647-4c65-baa8-84ea9345c8a9 ad9a17ce8088485ca950656ba84345b5 ff9c71872c20417e83e4387fccf85434 - default default] /dev/disk/by-id/dm-uuid-mpath-360000970000297802096533030373435 doesn't exists yet. wait_for_path /usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py:337
2021-05-21 15:32:17.742 8 DEBUG os_brick.utils [req-95e3cfc2-7647-4c65-baa8-84ea9345c8a9 ad9a17ce8088485ca950656ba84345b5 ff9c71872c20417e83e4387fccf85434 - default default] Failed attempt 1 _print_stop /usr/lib/python2.7/site-packages/os_brick/utils.py:45
2021-05-21 15:32:29.758 8 WARNING os_brick.initiator.linuxscsi [req-95e3cfc2-7647-4c65-baa8-84ea9345c8a9 ad9a17ce8088485ca950656ba84345b5 ff9c71872c20417e83e4387fccf85434 - default default] couldn't find a valid multipath device path for 360000970000297802096533030373435: VolumeDeviceNotFound: Volume device not found at /dev/mapper/360000970000297802096533030373435.
2021-05-21 15:32:29.826 8 WARNING os_brick.initiator.linuxscsi [req-95e3cfc2-7647-4c65-baa8-84ea9345c8a9 ad9a17ce8088485ca950656ba84345b5 ff9c71872c20417e83e4387fccf85434 - default default] multipath call failed exit 1: ProcessExecutionError: Unexpected error while running command.

and then failed with Failed to execute command multipath -l /dev/sdl
2021-05-21 15:32:29.828 8 ERROR nova.compute.manager [req-95e3cfc2-7647-4c65-baa8-84ea9345c8a9 ad9a17ce8088485ca950656ba84345b5 ff9c71872c20417e83e4387fccf85434 - default default] [instance: 8e22b053-ccf0-47dd-b968-93f61d4ee22e] Instance failed to spawn: CommandExecutionFailed: Failed to execute command multipath -l /dev/sdl

Comment 9 Lon Hohberger 2023-07-10 19:48:08 UTC

OSP13 support officially ended on 27 June 2023