Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1509230 - [container][ceph-ansible] container deployment with osd-scenario as lvm fails
[container][ceph-ansible] container deployment with osd-scenario as lvm fails
Status: POST
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: Container (Show other bugs)
3.0
Unspecified Unspecified
high Severity high
: rc
: 3.2
Assigned To: Guillaume Abrioux
mkasturi
Bara Ancincova
:
Depends On:
Blocks: 1553640 1494421
  Show dependency treegraph
 
Reported: 2017-11-03 07:36 EDT by mkasturi
Modified: 2018-11-01 14:42 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
.`lvm` installation scenario does not work when deploying Ceph in containers It is not possible to use the `osd_scenario: lvm` installation method to install a Ceph cluster in containers.
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ceph-ansible playbook log (24.41 KB, text/plain)
2017-11-03 07:36 EDT, mkasturi
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Github ceph/ceph-ansible/pull/2144 None None None 2017-11-07 09:39 EST
Github ceph/ceph-ansible/pull/2866 None None None 2018-07-17 04:46 EDT

  None (edit)
Description mkasturi 2017-11-03 07:36:26 EDT
Created attachment 1347282 [details]
ceph-ansible playbook log

Description of problem:
Deployment of container with osd scenario as "lvm" fails.

Version-Release number of selected component (if applicable):
[admin@magna003 ceph-ansible]$ rpm -qa | grep ansible
ceph-ansible-3.0.8-1.el7cp.noarch
ansible-2.4.0.0-5.el7.noarch
[admin@magna003 ceph-ansible]$ 

How reproducible:
2/2

Steps to Reproduce:
1. created lv cache volume on the osd nodes using the below commands.
a. pvcreate  /dev/sdb1 /dev/sdc1
b. vgcreate data_vg /dev/sdb1 /dev/sdc1
c. lvcreate -L 400G -n slowdisk data_vg /dev/sdb1 
d. lvcreate -L 100G -n cachedisk data_vg /dev/sdc1
e. lvcreate -L 2G -n metadisk data_vg /dev/sdc1
f. lvconvert --type cache-pool /dev/data_vg/cachedisk --poolmetadata /dev/data_vg/metadisk
g. lvconvert --type cache data_vg/slowdisk --cachepool data_vg/cachedisk
2. In osds.yml set the osd_scenario to "lvm"
3. The contianer deployment fails.

p.s. used /dev/sdd1 partition for journal

TASK [ceph-defaults : resolve device link(s)] *********************************************************************************************************************************************************************
fatal: [magna030]: FAILED! => {"failed": true, "msg": "'devices' is undefined"}
Actual results:
The deployment fails 

Expected results:
The deployment should succeed.

Additional info:
ubuntu@magna003 ceph-ansible]$ cat group_vars/osds.yml | egrep -v ^# | grep -v ^$
---
dummy:
osd_scenario: lvm #"{{ 'collocated' if journal_collocation or dmcrytpt_journal_collocation else 'non-collocated' if raw_multi_journal or dmcrypt_dedicated_journal else 'dummy' }}" # backward compatibility with stable-2.2, will disappear in stable 3.1
lvm_volumes:
   - data: slowdisk
     journal: /dev/sdd1
     data_vg: data_vg

=====
[ubuntu@magna003 ceph-ansible]$ cat group_vars/all.yml | egrep -v ^# | grep -v ^$
---
dummy:
fetch_directory: ~/ceph-ansible-keys
ceph_origin: distro
ceph_repository: rhcs
monitor_interface: eno1
public_network: 10.8.128.0/21
ceph_docker_image: "rhcs"
ceph_docker_image_tag: "ceph-3.0-rhel-7-docker-candidate-82532-20171102231218"
ceph_docker_registry: ""brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888"
containerized_deployment: True #"{{ True if mon_containerized_deployment or osd_containerized_deployment or mds_containerized_deployment or rgw_containerized_deployment else False }}" # backward compatibility with stable-2.2, will disappear in stable 3.1
[ubuntu@magna003 ceph-ansible]$ 

=====
[ubuntu@magna003 ceph-ansible]$ cat /etc/ansible/hosts 
[mons]
magna030
magna056
magna084

[osds]
magna030
magna056
magna084

[mgrs]
magna030
magna056
magna084

[ubuntu@magna003 ceph-ansible]$
Comment 3 Alfredo Deza 2017-11-03 13:08:35 EDT
Not sure I can help with the container portion of this. I can assist with the ceph-volume implementation though.
Comment 4 Guillaume Abrioux 2017-11-07 10:06:08 EST
Hi,

there are tasks to set some facts in ceph-defaults role which should be skipped when using osd_scenario: lvm.
it's fixed upstream, waiting for the CI to merge the commit in master.
Comment 5 Guillaume Abrioux 2017-11-07 10:50:09 EST
In any case, even if the error mentioned here is fixed, you are going to get an issue like no osd up, indeed, as far as I know, lvm scenario is not supposed to work with containerized deployments yet.
Comment 11 Ben England 2018-02-07 07:33:49 EST
Since RHOSP 13 now depends on RHCS 3.0z2, can we get this support into that z-stream build?  

Background: we need ceph-volume support in Ceph containers, particularly for bluestore.  Bluestore is critical to RHCS performance improvement, and ceph-volume seems critical for supporting deployment of bluestore, explained here by upstream documentation.  

http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/

This is particularly true for increasingly common all-flash configurations, where you need multiple OSDs/NVM device (or lose >= 40% throughput).  Only osd_scenario=lvm allows this functionality today.

https://mojo.redhat.com/groups/product-performance-scale-community-of-practice/blog/2018/01/31/bluestore-on-all-nvm-rados-bench
Comment 15 Guillaume Abrioux 2018-07-17 05:06:57 EDT
Hi John,

This patch has been backported in 3.0 since 3.0.11 and v3.1.0beta2 for 3.1.

Note that these backports are not going to add ceph-volume support in containers.
Comment 16 Ken Dreyer (Red Hat) 2018-10-16 14:54:35 EDT
Guillaume, what additional changes in ceph-ansible and ceph-container are needed for this BZ?

Note You need to log in before you can comment on or make changes to this bug.