Bug 1119756 - Failed to create Cinder volume on Gluster backend configured by packstack answer file
Summary: Failed to create Cinder volume on Gluster backend configured by packstack ans...
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 5.0 (RHEL 6)
Hardware: x86_64
OS: Linux
Target Milestone: ---
: 5.0 (RHEL 6)
Assignee: Eric Harney
QA Contact: nlevinki
Depends On:
TreeView+ depends on / blocked
Reported: 2014-07-15 12:47 UTC by Tzach Shefi
Modified: 2016-04-26 15:36 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2014-07-17 13:42:15 UTC
Target Upstream Version:

Attachments (Terms of Use)
Cinder volume.log (97.24 KB, application/x-gzip)
2014-07-15 12:47 UTC, Tzach Shefi
no flags Details

Description Tzach Shefi 2014-07-15 12:47:20 UTC
Created attachment 918157 [details]
Cinder volume.log

Description of problem: Cinder create volumes works intermittently, create volume sometime fails, Cinder's backend is a Gluster dist' volume.  

Version-Release number of selected component (if applicable):
rhel 6.5

How reproducible:
Intermittently, some volumes are created while others are not.  

Steps to Reproduce:
1. Configure Cinder to use Gluster volume via packstack
2. Cinder create 1    (or cinder create 1 --display-name  or with image-id xxxx)
3. Some volumes are created (state available) while others fails with state error. 

Gluster volume is a dist' volume, volume options are:
auth.allow           *
server.root-squash   off
server.allow-insecure on
user.cifs            enable
storage.owner-gid    165
storage.owner-uid    165
nfs.disable          off

Both gluster volume servers replay to ping and show mount as ok. 

Restarting volume share didn't help
Restarting AIO setup didn't help. 

Actual results:

Some volumes are created others fail, attached volume log. 

Expected results:
Volumes should be created successfully every time. 

Additional info:

Comment 1 Tzach Shefi 2014-07-16 10:53:09 UTC
Possible Gluster issue rather then RHOS, errors from gluster mnt log:

2014-07-16 05:40:34.598553] E [socket.c:2158:socket_connect_finish] 0-gluster-glance-tshefi-client-0: connection to failed (No route to host)
[2014-07-16 05:40:38.598859] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-gluster-glance-tshefi-client-0: changing port to 49493 (from 0)

Note errors are for Glance's volume, which oddly enough works fine, it did however gave me a hunch. 

As next troubleshooting step, created a new Gluster dist' volume, with 1 brick only on server which doesn't exhibit these "no route to host" problems. Cinder operations now work without a problem. 

Not a blocker as on new volume I can continue Cinder sanity testing.

Re-configuring original volume produces problems again, if this is something we want to check?

Comment 2 Eric Harney 2014-07-16 13:33:28 UTC
This indicates something is not quite right:

WARNING cinder.volume.drivers.glusterfs [-] Exception during mounting GlusterFS share at /var/lib/cinder/mnt/7395c6c5745705b5c77187ca90f13207 is not writable by the Cinder volume service. Snapshot operations will not be supported.

What are the permissions on the mount?

Comment 3 Eric Harney 2014-07-16 14:54:07 UTC
Appears to be caused by network connectivity issues between the Cinder volume server and the GlusterFS server.

Comment 4 Tzach Shefi 2014-07-17 07:50:32 UTC
Yes I agree with assumption, yet uneasy letting it pass just yet. 
I'll try testing another single bricked volume this time on suspected server Orion, as on Tiger server this passed. 

The volume options:
Auth.allow *
server.root-squash  on (tested with off also)
user.cifs enabled
storage.owner-gid 165
storage.owner-uid 165
server.allow-insecure  on 

[root@tiger share]# ll | grep cinder-tshefi
drwxrwxr-x  3  165  165   4096 Jul 17 10:38 gluster2-cinder-tshefi
drwxrwxr-x  3  165  165   4096 Jul 17 10:17 gluster3-cinder-tshefi
drwxrwxr-x  3  165  165     23 Jul 17 10:35 gluster4-cinder-tshefi
drwxrwxr-x  3  165  165   4096 Jul 17 10:07 gluster-cinder-tshefi

Comment 5 Tzach Shefi 2014-07-17 13:42:15 UTC
iptables -F on offending Gluster server fixed problem. 

Cinder operations now work, continuing test run.

Note You need to log in before you can comment on or make changes to this bug.