Bug 1320414

Summary: Unable to attach volume backed by LVM driver to instance in IPv6 and SSL deployment
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Giulio Fidente <gfidente>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: eharney, gfidente, jslagle, mburns, mcornea, nlevinki, ohochman, rhel-osp-director-maint, scohen, sgotliv, yeylon, yrabl
Target Milestone: gaKeywords: ZStream
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.14-4.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1300853 Environment:
Last Closed: 2016-04-07 21:49:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1300853, 1336226    
Bug Blocks:    
Attachments:
Description Flags
volume.log none

Description Marius Cornea 2016-03-23 07:35:19 UTC
I tested this on IPv6 enabled OSP8 and I got the same result. Attaching the volume.log

+++ This bug was initially created as a clone of Bug #1300853 +++

Description of problem:
Unable to attach volume to instance in IPv6 and SSL deployment

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-0.8.6-110.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy IPv6 and SSL enabled overcloud
2. Spawn an instance
3. Create a volume
4. Attach volume to intance

Actual results:
Volume doesn't get attached.

Expected results:
Volume gets attached.

Additional info:
Attaching the error log in cinder/volume.log

--- Additional comment from Sean Cohen on 2016-01-27 09:40:43 EST ---

(In reply to Marius Cornea from comment #0)
> Created attachment 1117016 [details]
> cinder.log
> 
> Description of problem:
> Unable to attach volume to instance in IPv6 and SSL deployment
> 
> Version-Release number of selected component (if applicable):
> openstack-tripleo-heat-templates-0.8.6-110.el7ost.noarch
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Deploy IPv6 and SSL enabled overcloud
> 2. Spawn an instance
> 3. Create a volume
> 4. Attach volume to intance
> 
> Actual results:
> Volume doesn't get attached.
> 
> Expected results:
> Volume gets attached.
> 
> Additional info:
> Attaching the error log in cinder/volume.log

I understand that this is LVM flow, and not Ceph one (that should be supported)
Since LVM is a reference implementation that should not be deployed in production I don't see this as a high priority for a zstream release.

Please clarify / add customer case.
Sean

--- Additional comment from Marius Cornea on 2016-01-27 09:53:22 EST ---

This issue was encountered during IPv6 testing. I think this will show up when deploying block storage nodes as well which is supported for production as far as I know.

--- Additional comment from Sean Cohen on 2016-01-27 11:40:09 EST ---

(In reply to Marius Cornea from comment #2)
> This issue was encountered during IPv6 testing. I think this will show up
> when deploying block storage nodes as well which is supported for production
> as far as I know.

Understood. 

See: https://access.redhat.com/articles/1535373#RHEL-OpenStack-Platform-Director

The default driver with Cinder is the internal LVM software iSCSI driver. This is the volume back-end which manages local volumes.

However, the Cinder iSCSI LVM driver has significant performance issues. In production environments, with high I/O activity, there are many potential issues which could affect performance or data integrity,

Red Hat strongly recommends using a certified Cinder plug-in provider for storage in a production environment. The software iSCSI LVM driver should be used and is only supported for single node evaluations and proof of concept environments. 

Adding Yogev to comment on the Ceph IPv6 testing (Ceph has IPv6 support since 1.2.3)

--- Additional comment from Sergey Gotliv on 2016-01-28 06:58:58 EST ---

Guys,

This issue is related to the LVM driver only! Unfortunately its really hard to say what exactly happened because Cinder failed to print an error message due to Unicode error, but most probably we failed to instantiate rtslib.NetworkPortal with IPv6 address. Some put that comment [1] in the rtstool.py for the good reason.

Marius,

Can I get an access to that environment to check my suspect?

[1]  # TODO(emh): Binding to IPv6 fails sometimes -- let pass for now.

--- Additional comment from Marius Cornea on 2016-01-28 08:21:44 EST ---

Sure, the undercloud can be accessed by:
ssh stack.44.97 -p 2900
credentials file is overcloudrc

then jump on the controller by:
ssh heat-admin.2.23

--- Additional comment from Lon Hohberger on 2016-02-04 09:55:25 EST ---

Not a 7.0.4 blocker - sgotliv notes that LVM is PoC

Comment 1 Marius Cornea 2016-03-23 07:37:00 UTC
Created attachment 1139365 [details]
volume.log

Comment 2 Sergey Gotliv 2016-03-23 13:56:58 UTC
This is not a blocker for GA since LVM is a POC driver.

Comment 3 Sergey Gotliv 2016-03-27 14:29:43 UTC
IMO, configuring iscsi_ip_address with the brackets for IPv6 is a nice workaround, but this is a workaround. I wouldn't even bother configuring that parameter the default value is perfect [1]. Unfortunately, with default value Cinder will fail to create an iSCSI portal with IPv6 address that's why I believe the fix should be done in Cinder.

[1] cfg.StrOpt('iscsi_ip_address',
               default='$my_ip',
               help='The IP address that the iSCSI daemon is listening on')

Anyway, I am working to fix Cinder in parallel. We have at least one place where we are using ::0 without brackets.

Comment 6 errata-xmlrpc 2016-04-07 21:49:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html