Bug 1629729

Summary: hp 3par fc volume create from image fails due to Duplicate: Volume (osv-xxx-.xxxxxx) already exists on array
Product: Red Hat OpenStack Reporter: Jaison Raju <jraju>
Component: python-os-brickAssignee: Alan Bishop <abishop>
Status: CLOSED ERRATA QA Contact: Avi Avraham <aavraham>
Severity: urgent Docs Contact: Kim Nylander <knylande>
Priority: urgent    
Version: 12.0 (Pike)CC: abishop, amcleod, apevec, astupnik, cinder-bugs, geguileo, jraju, jschluet, lhh, pmannidi, pmorey, rcernin, scohen, srevivo, tshefi
Target Milestone: ---Keywords: OtherQA, Triaged, ZStream
Target Release: 12.0 (Pike)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: python-os-brick-1.15.5-5.el7ost Doc Type: Bug Fix
Doc Text:
Previously, the os-brick code that scanned FibreChannel HBA hosts returned an invalid value under certain circumstances. As a result, services including cinder and nova failed. With this update, the FibreChannel HBA scan code always returns a valid value and cinder and nova no longer fail when scanning FibreChannel HBA hosts.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-05 18:51:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1653058    
Bug Blocks:    

Description Jaison Raju 2018-09-17 12:23:34 UTC
Description of problem:
On cinder.volume.drivers.hpe.hpe_3par_fc.HPE3PARFCDriver , volume creation from image or instance creation with boot from volume always fails.

2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager Traceback (most recent call last):
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     result = task.execute(**arguments)
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1006, in execute
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     **volume_spec)
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 953, in _create_from_image
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     image_service)
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 854, in _create_from_image_cache_or_download
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     image_service
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 685, in _create_from_image_download
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     model_update = self.driver.create_volume(volume) or {}
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 883, in trace_logging_wrapper
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     result = f(*args, **kwargs)
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_fc.py", line 199, in create_volume
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     return common.create_volume(volume)
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2050, in create_volume
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager     raise exception.Duplicate(msg)
2018-09-17 11:57:52.046 753160 ERROR cinder.volume.manager Duplicate: Volume (osv-xxx-.xxxxxx) already exists on array

Version-Release number of selected component (if applicable):
RHOS12
openstack-cinder-11.1.0-14.el7ost.noarch
python-os-brick-1.15.5-2.el7ost.noarch

How reproducible:
Always on customer environment

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 9 Alan Bishop 2018-09-19 18:56:10 UTC
Same as bug #1630842 (but not marking this a duplicate).

Comment 10 Alan Bishop 2018-09-20 17:12:00 UTC
This BZ describes two symptoms that have the same root cause, and that is a
bug in the code that scans FC HBAs. For background, creating a volume from an
image involves several steps, roughly:

1. Download the image from glance to a temporary location
2. Create an empty volume on the storage backend
3. Copy the image to the volume
4. Update the cinder volume database

The first failure can be seen in [1]

[1] https://bugzilla.redhat.com/attachment.cgi?id=1484862

ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 575, in _copy_image_to_volume
ERROR cinder.volume.manager     raise exception.ImageCopyFailure(reason=ex)
ERROR cinder.volume.manager ImageCopyFailure: Failed to copy image to volume: 'NoneType' object is not iterable

This happened during Step 3, which means:
- Step 2 completed, so the volume has been created on the storage backend
- Step 4 did not execute, so the cinder's database was not updated

Then, if the operation is tried again, the second failure will arise:

ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_fc.py", line 199, in create_volume
ERROR cinder.volume.manager     return common.create_volume(volume)
ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_common.py", line 2050, in create_volume
ERROR cinder.volume.manager     raise exception.Duplicate(msg)
ERROR cinder.volume.manager Duplicate: Volume (osv-xxx-.xxxxxx) already exists on array

This failure happened in Step 2, and it happens because the volume already
exists on the storage backend (it was left over from the first failure).

Comment 31 Tzach Shefi 2018-11-25 10:06:50 UTC
Blocked with BM UC deployment issue: 
https://bugzilla.redhat.com/show_bug.cgi?id=1653058

Don't suspect it's related/blocks the BZ per say, just I can't deploy BM to verify.

Comment 33 errata-xmlrpc 2018-12-05 18:51:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3789