Bug 1121378 - Attach cinder volume fails using Gluster as backend, virt_use_fusefs not enabled
Summary: Attach cinder volume fails using Gluster as backend, virt_use_fusefs not enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 5.0 (RHEL 6)
Hardware: x86_64
OS: Linux
high
unspecified
Target Milestone: rc
: 5.0 (RHEL 6)
Assignee: Ryan Hallisey
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-20 08:08 UTC by Tzach Shefi
Modified: 2014-09-08 05:20 UTC (History)
5 users (show)

Fixed In Version: openstack-selinux-0.1.4-2.el6ost.src.rpm
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-09-02 17:38:20 UTC


Attachments (Terms of Use)
Nova compute log (1.10 MB, application/x-gzip)
2014-07-20 08:09 UTC, Tzach Shefi
no flags Details
nova compute log and audit log (35.13 KB, application/x-gzip)
2014-07-22 11:07 UTC, Tzach Shefi
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1117 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Bug Fix and Enhancement Advisory 2014-09-02 21:38:01 UTC

Description Tzach Shefi 2014-07-20 08:08:59 UTC
Description of problem: Mounting a cinder volume (Gluster backend) fails from CLI and UI with error on nova compute log: 

/var/log/nova/compute.log:2014-07-20 10:22:16.037 25142 ERROR nova.virt.block_device [req-23d4ae4a-d238-494c-b33b-85b32d50cbfb d122e4455eef4b93b216e4af45a78b18 47a9b96b62e24c6fa2a92c02ce9be27c] [instance: 2f96aec7-eea9-4d32-b2e7-e9f9639c2aa1] Driver failed to attach volume 691f5ebc-71a2-477e-b2fa-6d3926e7910f at /dev/vdb
/var/log/nova/compute.log:2014-07-20 10:22:16.232 25142 DEBUG urllib3.connectionpool [-] "POST /v1/47a9b96b62e24c6fa2a92c02ce9be27c/volumes/691f5ebc-71a2-477e-b2fa-6d3926e7910f/action HTTP/1.1" 202 0 _make_request /usr/lib/python2.6/site-packages/urllib3/connectionpool.py:295


Version-Release number of selected component (if applicable):
RHEL6.5 
openstack-selinux-0.1.4-1.el6ost.noarch
openstack-cinder-2014.1.1-1.el6ost.noarch
python-cinderclient-1.0.9-1.el6ost.noarch
python-cinder-2014.1.1-1.el6ost.noarch
openstack-nova-compute-2014.1.1-2.el6ost.noarch


How reproducible:
Every time

Steps to Reproduce:
1. Configure Cinder to use Gluster as backend (in my case via packstack)
2. Create cinder volume
3. Try to attach volume to a running instance, fails no volume is attached, nothing shows up on CLI or UI. 

4. Checking selinux -> virt_use_fusefs --> off

5. Enabled missing bool:
[root@orange-vdse ~]# setsebool -P virt_use_fusefs=1
Full path required for exclude: net:[4026532276].
Full path required for exclude: net:[4026532332].

6. Rechecking: virt_use_fusefs --> on

7. Cinder volume attach now works, ok, see last volume attachment on log.  

Actual results:
Volume not attached to instance
Error on nova log

Expected results:
Volume should attach successfully

Additional info:

Comment 1 Tzach Shefi 2014-07-20 08:09:50 UTC
Created attachment 919331 [details]
Nova compute log

Comment 2 Ryan Hallisey 2014-07-21 14:54:15 UTC
Can you reproduce in permissive and attach your /var/log/audit/audit.log file?  I just want to see the AVCs before I make any changes.

Comment 3 Tzach Shefi 2014-07-22 11:07:09 UTC
Created attachment 919871 [details]
nova compute log and audit log

Comment 4 Tzach Shefi 2014-07-22 11:07:51 UTC
Hey Ryan, 

Sure thing, reproduced steps below 

1. Installed RHEL6.5
2. Installed RHOS5 
3. # getsebool -a | grep off   (Just to check status before)
virt_use_fusefs --> off

4. Enabled debug logging for nova
5. Created instance based on Cirros 
6. Created empty Cinder volume 1Giga
7. Volume attach failed, right after that on audit / compute log added line look for -> tshefi
8. Enabled virt_use_fuzefs 
9. Now volume attachment works 
10. Added another marker on logs look for -> worked

Ping me back if you need more. 
BTW if you wish to ssh no problem, let me know I'll send details.

Comment 5 Ryan Hallisey 2014-07-22 14:20:50 UTC
There was only one AVC so I will add this bool the newest policy.
# setsebool -P virt_use_fusefs on

Comment 9 Tzach Shefi 2014-08-19 09:04:12 UTC
Technically bug is verified. 
version: openstack-selinux-0.1.5-1.el6ost.noarch
I've tested this out, using packstack and Gluster.

virt_use_fusefs --> on  is indeed enabled, Cinder configuration looks fine. 


I did however still fail to create Cinder volume on Gluster, looking at logs problem caused by Gluster or fuse client bug.  
It's a new Gluster server deployment, might be configuration related error.  
Gluster version:

glusterfs-fuse-3.6.0.27-1.el6rhs.x86_64
glusterfs-server-3.6.0.27-1.el6rhs.x86_64
glusterfs-libs-3.6.0.27-1.el6rhs.x86_64
glusterfs-cli-3.6.0.27-1.el6rhs.x86_64
samba-glusterfs-3.6.9-168.4.el6rhs.x86_64
glusterfs-api-3.6.0.27-1.el6rhs.x86_64
glusterfs-rdma-3.6.0.27-1.el6rhs.x86_64
vdsm-gluster-4.14.7.2-1.el6rhs.noarch
glusterfs-3.6.0.27-1.el6rhs.x86_64

Error on Cinder volume log:
2014-08-18 17:06:45.303 17138 WARNING cinder.volume.drivers.glusterfs [req-d07abe66-37da-422c-9211-590fe85e627c - - - - -] Exception during mounting Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf mount -t glusterfs 10.35.163.51:/tshefi-cinder /var/lib/cinder/mnt/5a8dfee58d4bedfde7d3ede02f2c3278
Exit code: 1
Stdout: 'Mount failed. Please check the log file for more details.\n'
Stderr: ''


Error on Gluster mount log

[2014-08-19 07:30:49.067450] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
[2014-08-19 07:30:49.094869] E [glusterfsd-mgmt.c:1369:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server
[2014-08-19 07:30:49.094959] E [glusterfsd-mgmt.c:1460:mgmt_getspec_cbk] 0-mgmt: Server is operating at an op-version which is not supported


I'll open a new bug and add it's BZ number here for reference.

Comment 11 errata-xmlrpc 2014-09-02 17:38:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1117.html


Note You need to log in before you can comment on or make changes to this bug.