Description of problem: During initial RHHI-V 1.6 deployment, while configuring cache, ansible fails with: ~~~ TASK [gluster.infra/roles/backend_setup : Change attributes of LV] ************* failed: [node3.example.com] (item={u'vgname': u'gluster_vg_sdb', u'cachethinpoolname': u'gluster_thinpool_gluster_vg_sdb', u'cachelvname': u'cachelv_gluster_thinpool_gluster_vg_sdb', u'cachedisk': u'/dev/sdc', u'cachemetalvname': u'cache_gluster_thinpool_gluster_vg_sdb', u'cachemode': u'writethrough', u'cachemetalvsize': u'20G', u'cachelvsize': u'180G'}) => {"changed": false, "item": {"cachedisk": "/dev/sdc", "cachelvname": "cachelv_gluster_thinpool_gluster_vg_sdb", "cachelvsize": "180G", "cachemetalvname": "cache_gluster_thinpool_gluster_vg_sdb", "cachemetalvsize": "20G", "cachemode": "writethrough", "cachethinpoolname": "gluster_thinpool_gluster_vg_sdb", "vgname": "gluster_vg_sdb"}, "msg": "No size given."} failed: [node2.example.com] (item={u'vgname': u'gluster_vg_sdb', u'cachethinpoolname': u'gluster_thinpool_gluster_vg_sdb', u'cachelvname': u'cachelv_gluster_thinpool_gluster_vg_sdb', u'cachedisk': u'/dev/sdc', u'cachemetalvname': u'cache_gluster_thinpool_gluster_vg_sdb', u'cachemode': u'writethrough', u'cachemetalvsize': u'20G', u'cachelvsize': u'180G'}) => {"changed": false, "item": {"cachedisk": "/dev/sdc", "cachelvname": "cachelv_gluster_thinpool_gluster_vg_sdb", "cachelvsize": "180G", "cachemetalvname": "cache_gluster_thinpool_gluster_vg_sdb", "cachemetalvsize": "20G", "cachemode": "writethrough", "cachethinpoolname": "gluster_thinpool_gluster_vg_sdb", "vgname": "gluster_vg_sdb"}, "msg": "No size given."} failed: [node1.example.com] (item={u'vgname': u'gluster_vg_sdb', u'cachethinpoolname': u'gluster_thinpool_gluster_vg_sdb', u'cachelvname': u'cachelv_gluster_thinpool_gluster_vg_sdb', u'cachedisk': u'/dev/sdc', u'cachemetalvname': u'cache_gluster_thinpool_gluster_vg_sdb', u'cachemode': u'writethrough', u'cachemetalvsize': u'20G', u'cachelvsize': u'180G'}) => {"changed": false, "item": {"cachedisk": "/dev/sdc", "cachelvname": "cachelv_gluster_thinpool_gluster_vg_sdb", "cachelvsize": "180G", "cachemetalvname": "cache_gluster_thinpool_gluster_vg_sdb", "cachemetalvsize": "20G", "cachemode": "writethrough", "cachethinpoolname": "gluster_thinpool_gluster_vg_sdb", "vgname": "gluster_vg_sdb"}, "msg": "No size given."} ~~~ Although, the /etc/ansible/hc_wizard_inventory.yml, for each node, has: ~~~ gluster_infra_cache_vars: - vgname: gluster_vg_sdb cachedisk: /dev/sdc cachelvname: cachelv_gluster_thinpool_gluster_vg_sdb cachethinpoolname: gluster_thinpool_gluster_vg_sdb cachelvsize: 270G cachemetalvsize: 30G cachemetalvname: cache_gluster_thinpool_gluster_vg_sdb cachemode: writethrough ~~~ Version-Release number of selected component (if applicable): gluster-ansible-cluster-1.0-1.el7rhgs.noarch Tue Apr 9 20:36:45 2019 gluster-ansible-features-1.0.4-5.el7rhgs.noarch Tue Apr 9 20:36:46 2019 gluster-ansible-infra-1.0.3-3.el7rhgs.noarch Tue Apr 9 20:36:46 2019 gluster-ansible-maintenance-1.0.1-1.el7rhgs.noarch Tue Apr 9 20:36:45 2019 gluster-ansible-repositories-1.0-1.el7rhgs.noarch Tue Apr 9 20:36:46 2019 gluster-ansible-roles-1.0.4-4.el7rhgs.noarch Tue Apr 9 20:36:47 2019 kmod-kvdo-6.1.1.125-5.el7.x86_64 Tue Apr 9 20:33:58 2019 vdo-6.1.1.125-3.el7.x86_64 Tue Apr 9 20:36:31 2019 How reproducible: 100 % in user environment Steps to Reproduce: 1. Start RHHI 1.6 deploy and configure LV cache with a PV and a size Actual results: Ansible fails with >> "msg": "No size given." on the three nodes. Message is not clear, all values were passed Expected results: Deploy should proceed with the information already provided by the user
Javier Coscia, when I went through the variable file (hc_wizard_inventory.yml) you have attached. It does not contain an entry for gluster_infra_thinpools at all. If no valid lv name is given, the module thinks that a new lv has to be created and expects size parameter. @gobinda, what could be the reason for this erroneous variable file to be generated?
I think I kind of found out the problem: There are 2 disks used: sdb, sdd 'sdb' is enabled with data-deduplication and compression, so thick LVs are created on 'sdb'. This is because VDO doesn't work well with LVM thinpool ( discard problem ) 'sdd' on the other hand has 'vmstore' brick and its thinpool LVM cache, when enabled tries to attach the LVM cache to thinpool on sdb, which is not there and its failing. I feel this is the reason. Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1608268 The area to look in to are: - But why did 'gluster-ansible' not provide proper errors ? - I think cache-meta-lv name & size are not required. Need to check on this
(In reply to SATHEESARAN from comment #4) > I think I kind of found out the problem: > > There are 2 disks used: sdb, sdd > > 'sdb' is enabled with data-deduplication and compression, so thick LVs are > created on 'sdb'. > This is because VDO doesn't work well with LVM thinpool ( discard problem ) > > 'sdd' on the other hand has 'vmstore' brick and its thinpool > If I get this right, user selected a different SD device, sdd, for the 3rd brick which is vmstore and un-checked dedup & comp on it, leaving checked it on `engine` and `data` bricks ? > LVM cache, when enabled tries to attach the LVM cache to thinpool on sdb, > which is not there > and its failing. I feel this is the reason. > Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1608268 > > The area to look in to are: > - But why did 'gluster-ansible' not provide proper errors ? > - I think cache-meta-lv name & size are not required. Need to check on this Is there a workaround to share for the user ? If my assumption is correct, checking the dedup&comp check box for `vmstore` brick will work ?
(In reply to Sachidananda Urs from comment #3) > Javier Coscia, when I went through the variable file > (hc_wizard_inventory.yml) you have attached. > It does not contain an entry for gluster_infra_thinpools at all. If no > valid lv name is given, the module thinks that a new lv has to be created > and expects size parameter. > > @gobinda, what could be the reason for this erroneous variable file to be > generated? As of now vdo does not support thin pool from cockpit side (discard problem) and sdb becomes thick. I think this is a bug, eventhough it's thick but lvm cache trying to use as thinpool which is causing issue. I will file a bug and work on it. Anyways we are going to disable Lvm cache option if VDO used because lvm cache does not perform well in case of VDO. The workarround is to create thinpool from non-vdo volume and attach to lvmcache like: ---- gluster_infra_thinpools: - vgname: gluster_vg_sde thinpoolname: gluster_thinpool_gluster_vg_sde poolmetadatasize: 3G gluster_infra_cache_vars: - vgname: gluster_vg_sde cachedisk: /dev/sdc cachelvname: cachelv_gluster_thinpool_gluster_vg_sde cachethinpoolname: gluster_thinpool_gluster_vg_sde cachelvsize: 270G cachemetalvsize: 30G cachemetalvname: cache_gluster_thinpool_gluster_vg_sde cachemode: writethrough ---- You can edit inventory file from cockpit and save.
(In reply to Gobinda Das from comment #6) > (In reply to Sachidananda Urs from comment #3) > > Javier Coscia, when I went through the variable file > > (hc_wizard_inventory.yml) you have attached. > > It does not contain an entry for gluster_infra_thinpools at all. If no > > valid lv name is given, the module thinks that a new lv has to be created > > and expects size parameter. > > > > @gobinda, what could be the reason for this erroneous variable file to be > > generated? > > As of now vdo does not support thin pool from cockpit side (discard problem) > and sdb becomes thick. > I think this is a bug, eventhough it's thick but lvm cache trying to use as > thinpool which is causing issue. > I will file a bug and work on it. > Anyways we are going to disable Lvm cache option if VDO used because lvm > cache does not perform well in case of VDO. > The workarround is to create thinpool from non-vdo volume and attach to > lvmcache like: > > ---- > gluster_infra_thinpools: > - vgname: gluster_vg_sde > thinpoolname: gluster_thinpool_gluster_vg_sde > poolmetadatasize: 3G > > gluster_infra_cache_vars: > - vgname: gluster_vg_sde > cachedisk: /dev/sdc > cachelvname: cachelv_gluster_thinpool_gluster_vg_sde > cachethinpoolname: gluster_thinpool_gluster_vg_sde > cachelvsize: 270G > cachemetalvsize: 30G > cachemetalvname: cache_gluster_thinpool_gluster_vg_sde > cachemode: writethrough > ---- > You can edit inventory file from cockpit and save. Right now we don't support LVM Cache for VDO. I know there is a bug in cockpit which is showing the LVM Cache option in case of VDO.
FYI, user was able to complete the RHHI-V 1.6 deployment after unchecking `Configure LV Cache` in the `Bricks` step
VDO doesn't work good with thinpool() To avoid this, when VDO volume is used, thick LV will be used. and with gluster-ansible roles, lvmcache can only be attached to thinpool https://bugzilla.redhat.com/show_bug.cgi?id=1608268
Javier Coscia, closing this a duplicate of BZ 1721451 Please re-open if you think otherwise. *** This bug has been marked as a duplicate of bug 1721451 ***
As far as I understood, the last fix needed to make VDO behave with thinpool was changing the vdo.service file as described here : https://bugzilla.redhat.com/show_bug.cgi?id=1693657 https://bugzilla.redhat.com/show_bug.cgi?id=1634551 https://bugzilla.redhat.com/show_bug.cgi?id=1690606 That fix consists of the following one liner : ExecStartPre=/bin/sh -c "echo 4096 > /sys/kvdo/max_discard_sectors" If that's indeed the only thing left to have lvcache on VDO, I do not understand why such back pedaling here to remove thinpool option for VDO enabled devices instead of properly fixing the feature.
Hi Marina, The issue was VDO had discard problems and this is solved from VDo side. Now I think all problem solved and vdo with thinp works, as per https://bugzilla.redhat.com/show_bug.cgi?id=1693653 So based on this I am closing this bug.
(In reply to Gobinda Das from comment #25) > Hi Marina, > The issue was VDO had discard problems and this is solved from VDo side. > Now I think all problem solved and vdo with thinp works, as per > https://bugzilla.redhat.com/show_bug.cgi?id=1693653 > > So based on this I am closing this bug. Hi Gobinda, Sas, Please help me out here. I am trying to understand if this KCS needs to be updated: https://access.redhat.com/solutions/4170661 I understand, we still cannot support lv cache on VDO volumes, right? (I also went to the docs bz#1715568 and didn't find the answer I was looking for). Thank you!
Hi Marina, Now we have introduced new field to attach specific thinpool to lvmcache in bz#1721451 . And as per our earlier perf test lvm cache does not perform well with VDO but deployment should work without any issue.