Bug 1095615 (RHEV_CEPH_AS_POSIX_DOMAIN)

Just following up on this BZ to verify that this would fit the following use-case:

Moving away from the traditional Fibre storage model and moving all things into CEPH + OpenStack. Also utilizing RHEV to house other infrastructure components.  Without FC storage and no planned NFS, shared storage for RHEV is missing with the
current plan. 

Looking for something simmilar to the following closed RFE for RHEL 7:

In RHEL 7.1 there is native qemu support for CEPH RBD:
https://bugzilla.redhat.com/show_bug.cgi?id=1140742

Currently supporting CephFS isn't on targeted to any upcoming version, and IMHO, I don't think it should be.

RHEV best practice to handle Ceph is via a Cinder backend (see bug 1135132 for details). CephFS could still work as a POSIX domain, but I don't think it should be treated as a first-class citizen.

Wrt concerns raised:
> Moving away from the traditional Fibre storage model and moving all things
> into CEPH + OpenStack. Also utilizing RHEV to house other infrastructure
> components.  Without FC storage and no planned NFS, shared storage for RHEV
> is missing with the
> current plan. 
Handled by bug 1135132, albeit probably not completely in 3.6.0 (some feature-gaps between Cinder-backed approach and traditional RHEV storage).

Adding PM stakeholders to weigh in on this.

(In reply to Allon Mureinik from comment #3)
> Currently supporting CephFS isn't on targeted to any upcoming version, and
> IMHO, I don't think it should be.
> 
> RHEV best practice to handle Ceph is via a Cinder backend (see bug 1135132
> for details). CephFS could still work as a POSIX domain, but I don't think
> it should be treated as a first-class citizen.
> 
> Wrt concerns raised:
> > Moving away from the traditional Fibre storage model and moving all things
> > into CEPH + OpenStack. Also utilizing RHEV to house other infrastructure
> > components.  Without FC storage and no planned NFS, shared storage for RHEV
> > is missing with the
> > current plan. 
> Handled by bug 1135132, albeit probably not completely in 3.6.0 (some
> feature-gaps between Cinder-backed approach and traditional RHEV storage).
> 
> Adding PM stakeholders to weigh in on this.

We will still have all the storage feature for ISCSI\FC\NFS using LVM and will continue with maintaining that infra. Any Cinder based infra will be complimentary and we will look to expend this support for future releases.

So what's left to be done here?

(In reply to Yaniv Kaul from comment #5)
> So what's left to be done here?
I don't see any good usecase for this RFE.
If you want to use Ceph, you should use Ceph (albeit via Cinder at the moment).

Since CephFS is POSIX compliant, you can consume it as a POSIX domain. I see no reason to have it as a top level citizen in RHEV (a-la GlusterFS).

Yaniv - please see if we can close this one.

Makes sense to me. I'll move this one to ON_QA as test only and let this be tested as a POSIX storage domain.

How is it ON_QA for 4.0 already?

(In reply to Yaniv Kaul from comment #9)
> How is it ON_QA for 4.0 already?

Read comment #8.

Yaniv Kaul/Dary - 

As this will work under POSIX, are we good with testing storage sanity to verify it? 
Do you think we should run something specific ?

(In reply to Aharon Canan from comment #11)
> Yaniv Kaul/Dary - 
> 
> As this will work under POSIX, are we good with testing storage sanity to
> verify it? 
> Do you think we should run something specific ?

Sanity should be good. Allon, anything specific you think is needed?

(In reply to Yaniv Dary from comment #12)
> (In reply to Aharon Canan from comment #11)
> > Yaniv Kaul/Dary - 
> > 
> > As this will work under POSIX, are we good with testing storage sanity to
> > verify it? 
> > Do you think we should run something specific ?
> 
> Sanity should be good. Allon, anything specific you think is needed?

Sanity will be fine, thanks!

Mounting using ceph-fuse with POSIX doesn't seem to be possible since the ceph-fuse syntax is as follows:

# ceph-fuse -k ./ceph.client.admin.keyring -m ceph-1.qa.lab:6789 /mnt/cephfs


Second choice is to use 'mount -t ceph'. In this case, storage domain creation with POSIX seems more likely to work. 
The following syntax works while mounting manually from host:

# mount -t ceph ceph-1.qa.lab:6789:/ /mnt/cephfs/ -o name=admin,secret=<key>

(The key is not hidden in the actual mount command).

Tried to use this way for ceph based POSIX storage domain creation as follows:

=======================================
Path = 10.35.65.18:/

VFS Type = ceph

Mount Options = name=admin,secret=<key>
=======================================

Domain creation doesn't work. Examined vdsm.log, I noticed that if nothing is given after '/' in the path, the '/' is ignored in the mount command that vdsm executes:

jsonrpc.Executor/3::DEBUG::2016-01-26 17:51:04,338::mount::229::Storage.Misc.excCmd::(_runcmd) /usr/bin/taskset --cpu-list 0-1 /usr/bin/sudo -n /usr/bin/mount -t ceph -o name=ad
min,secret=AQC3W1dWhplVLBAARW/zKtQzjafZDKAGfVpWbQ== 10.35.65.18: /rhev/data-center/mnt/10.35.65.18:___ (cwd None)


Here is the failure:

jsonrpc.Executor/3::ERROR::2016-01-26 17:51:04,372::hsm::2473::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2470, in connectStorageServer
    conObj.connect()
  File "/usr/share/vdsm/storage/storageServer.py", line 234, in connect
    six.reraise(t, v, tb)
  File "/usr/share/vdsm/storage/storageServer.py", line 226, in connect
    self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
  File "/usr/share/vdsm/storage/mount.py", line 225, in mount
    return self._runcmd(cmd, timeout)
  File "/usr/share/vdsm/storage/mount.py", line 241, in _runcmd
    raise MountError(rc, ";".join((out, err)))
MountError: (1, 'source mount path was not specified\nfailed to resolve source\n;')



Seems that no matter how many '/' are given, vdsm ignores all if there is nothing after them. (I tried a path with '//' at the end, didn't work.)

This has to be fixed for ceph fs to work as a VFS type for POSIXFS compliant storage domain creation.

Please open a bug (for comment 14) and block this RFE on it.
After applying workaround please test the full usage.

Yaniv, due to BZ #1303550, a cephfs POSIX compliant domain cannot be attached to a storage pool. Therefore, we cannot test the full usage.

Moving to ON_QA since BZ #1303550 was closed.

Tested storage sanity using ceph fs posix domain with 2 hosts. Tested while SELinux was set to permissive due to https://bugzilla.redhat.com/show_bug.cgi?id=1315332. All cases passed.


We will verify the RFE once the SELinux issue will be fixed.


Used:

rhevm-3.6.3.4-0.1.el6.noarch
vdsm-4.17.23-0.el7ev.noarch
ceph-fuse-0.94.6-0.el7.x86_64
ceph-common-0.94.6-0.el7.x86_64
libcephfs1-0.94.6-0.el7.x86_64
ceph-0.94.6-0.el7.x86_64
python-cephfs-0.94.6-0.el7.x86_64
libvirt-daemon-1.2.17-13.el7_2.4.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.8.x86_64
selinux-policy-3.13.1-60.el7_2.3.noarch

Sanlock is able to quire lock and cephfs (POSIXFS compliant) storage domain attachment to pool succeeds. 

2016-09-14 14:30:57+0300 3111497 [726]: s23 lockspace 822249d8-cdfb-4977-a0d3-ff0852f8bc4f:1:/rhev/data-center/mnt/10.35.140.87:6789:_elad/822249d8-cdfb-4977-a0d3-ff0852f8bc4f/dom_md/ids:0



2016-09-14 14:30:28,823 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4259caa4] START, ConnectStorageServerVDSCommand(
HostName = seal09, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='1830e736-4f19-4b89-ae34-371286ed77ba', storagePoolId='00000000-0000-0000-0000-000000000000', stora
geType='POSIXFS', connectionList='[StorageServerConnections:{id='c566e67b-5350-4612-891d-bc2413299d9c', connection='10.35.140.87:6789:/elad', iqn='null', vfsType='ceph', mountOptions='name=a
dmin,secret=AQA9DKtX/2r1ChAAsyOgwLlUieCw2C5dsVAh4g==', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4004419a
2016-09-14 14:30:29,572 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-46) [4259caa4] FINISH, ConnectStorageServerVDSCommand
, return: {c566e67b-5350-4612-891d-bc2413299d9c=0}, log id: 4004419a



2016-09-14 14:31:59,619 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-8) [5749a40c] Correlation ID: 36ada255, Job ID: c505ccbf-f31d-4186-aa54-bc73405ae35e, Call Stack: null, Custom Event ID: -1, Message: Storage Domain ceph1 was attached to Data Center 73 by admin@internal-authz


======================================================================

Used:

Hypervisor:

RHEL7.3
vdsm-4.18.13-1.el7ev.x86_64
selinux-policy-3.13.1-93.el7.noarch
python-cephfs-10.2.2-41.el7cp.x86_64
ceph-selinux-10.2.2-41.el7cp.x86_64
ceph-common-10.2.2-41.el7cp.x86_64
ceph-base-10.2.2-41.el7cp.x86_64
libcephfs1-10.2.2-41.el7cp.x86_64

Ceph server:

ceph-mon-10.2.2-38.el7cp.x86_64
ceph-osd-10.2.2-38.el7cp.x86_64
ceph-ansible-1.0.5-32.el7scon.noarch
python-cephfs-10.2.2-38.el7cp.x86_64
ceph-common-10.2.2-38.el7cp.x86_64
ceph-base-10.2.2-38.el7cp.x86_64
ceph-mds-10.2.2-38.el7cp.x86_64
ceph-radosgw-10.2.2-38.el7cp.x86_64
ceph-selinux-10.2.2-38.el7cp.x86_64
libcephfs1-10.2.2-38.el7cp.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1967.html