Bug 1548157

Summary: [GSS] heketi db keeps trying to remove thin pool which is already removed
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Olek Toborek <atoborek>
Component: heketiAssignee: Raghavendra Talur <rtalur>
Status: CLOSED ERRATA QA Contact: Rachael <rgeorge>
Severity: medium Docs Contact:
Priority: unspecified    
Version: cns-3.6CC: akrishna, hchiramm, jmulligan, madam, pprakash, rhs-bugs, rtalur, storage-qa-internal, tcarlin, vinug
Target Milestone: ---   
Target Release: CNS 3.10   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, Heketi could not delete volumes that were fully or partially removed from the underlying storage system. Running Heketi commands such as 'heketi-cli volume delete <ID>' on a volume that was deleted from gluster failed due to synchronization issues. With this fix, Heketi volume delete functionality can incrementally delete a volume, skipping over items that have already been deleted.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 09:22:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1568861, 1573420    

Description Olek Toborek 2018-02-22 20:57:59 UTC 
Description of problem:

Thin pool is already gone but heketi doesn't know it and keeps trying to remove it. Is there any way to update heketi db with the latest topology?

Version-Release number of selected component (if applicable):

N/A

How reproducible:

N/A

Steps to Reproduce:
1.
2.
3.

Actual results:

heketi keeps trying to remove the thin pool

Expected results:

updated heketi db with latest information about topology


Additional info:

Thin pool in question: tp_74223a190c197a64d0f92729b7f7241e


========================================================================


[negroni] Started DELETE /volumes/53ef9e5d676bf5233e5c8920279e76e9
[negroni] Completed 202 Accepted in 370.331µs
[asynchttp] INFO 2018/02/13 17:51:45 Started job da24e7adb82a87dbe63571adaa0d8c9c
[heketi] INFO 2018/02/13 17:51:45 Destroying volume 53ef9e5d676bf5233e5c8920279e76e9
[negroni] Started GET /queue/da24e7adb82a87dbe63571adaa0d8c9c
[negroni] Completed 200 OK in 25.56µs
[kubeexec] DEBUG 2018/02/13 17:51:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: edclosid117.bcbsfl.com Pod: glusterfs-9xpzq Command: gluster --mode=script snapshot list vol_53ef9e5d676bf5233e5c8920279e76e9 --xml
Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
  <opRet>-1</opRet>
  <opErrno>30806</opErrno>
  <opErrstr>Volume (vol_53ef9e5d676bf5233e5c8920279e76e9) does not exist</opErrstr>
</cliOutput>
[kubeexec] DEBUG 2018/02/13 17:51:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: edclosid115.bcbsfl.com Pod: glusterfs-q2gc9 Command: lvs --options=lv_name,thin_count --separator=:
Result:   LV:#Thins
  docker-pool:0
  home:
  opt:
  opt-bmc:
  opt-cvt:
  opt-nimsoft:
  root:
  tmp:
  usr:
  usr-local:
  var:
  var:
  var:
  var:
  var:
  var:
  var-log:
  var-log:
  var-log-audit:
  brick_69629d4438e16d45227b2e6a84374d0b:
  brick_6aaacb3795dbbf4a5821d80c25b2c43a:
  brick_c0c1e0316093ab248e3375a4e9bb3bf3:
  brick_c35b85fd10e3ec825d6e1d808a3287b7:
  brick_ef4c65add03ab9caa8b530c62a4b5413:
  brick_fad582bf40e8e3ac8a89caa119a20476:
  tp_69629d4438e16d45227b2e6a84374d0b:1
  tp_6aaacb3795dbbf4a5821d80c25b2c43a:1
  tp_c0c1e0316093ab248e3375a4e9bb3bf3:1
  tp_c35b85fd10e3ec825d6e1d808a3287b7:1
  tp_ef4c65add03ab9caa8b530c62a4b5413:1
  tp_fad582bf40e8e3ac8a89caa119a20476:1
  brick_407b39ef7a3dfd23f50536d5064f6360:
  brick_51005307ee9c3988ccdaecbdd5651282:
  brick_7a9645f49829407bdc6e17711cde25dc:
  brick_c031af051eec0105bee9c17b02d24927:
  brick_f8ecc7f462fc59bfca6e9803f2c1151a:
  brick_f9f8af21f3b25c473cb8ed20710bab07:
  tp_407b39ef7a3dfd23f50536d5064f6360:1
  tp_51005307ee9c3988ccdaecbdd5651282:1
  tp_7a9645f49829407bdc6e17711cde25dc:1
  tp_c031af051eec0105bee9c17b02d24927:1
  tp_f8ecc7f462fc59bfca6e9803f2c1151a:1
  tp_f9f8af21f3b25c473cb8ed20710bab07:1
[kubeexec] DEBUG 2018/02/13 17:51:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: edclosid117.bcbsfl.com Pod: glusterfs-9xpzq Command: lvs --options=lv_name,thin_count --separator=:
Result:   LV:#Thins
  docker-pool:0
  home:
  opt:
  opt-bmc:
  opt-cvt:
  opt-nimsoft:
  root:
  tmp:
  usr:
  usr-local:
  var:
  var:
  var:
  var:
  var:
  var:
  var-log:
  var-log:
  var-log-audit:
  brick_025f5a721051d6b897bfc658f0311319:
  brick_8c99f2defb1419adfc79677b5dce8b69:
  brick_a612d85b353aadc7ab6cc92be1875ee4:
  brick_a7c2ffc8fa44d0652fa5e5c546b774b6:
  brick_e2595d8174c350d7d37be697d269d636:
  brick_fb6a8b10989840380df4cbb62cb53f91:
  tp_025f5a721051d6b897bfc658f0311319:1
  tp_8c99f2defb1419adfc79677b5dce8b69:1
  tp_a612d85b353aadc7ab6cc92be1875ee4:1
  tp_a7c2ffc8fa44d0652fa5e5c546b774b6:1
  tp_e2595d8174c350d7d37be697d269d636:1
  tp_fb6a8b10989840380df4cbb62cb53f91:1
  brick_1627d4f6996e1f9b79b9bb4c2b31b871:
  brick_2d3c641f97daa67dca8d4c5e2dc81020:
  brick_2dee7499f805f2631d554f7c2432f4d3:
  brick_4d874f689c98afc338fb6f13c23a2a0c:
  brick_54e18068adce69989bed726673b3beec:
  brick_eaa2796aa0d2a1ba9981ed5cfdaaa640:
  tp_1627d4f6996e1f9b79b9bb4c2b31b871:1
  tp_2d3c641f97daa67dca8d4c5e2dc81020:1
  tp_2dee7499f805f2631d554f7c2432f4d3:1
  tp_4d874f689c98afc338fb6f13c23a2a0c:1
  tp_54e18068adce69989bed726673b3beec:1
  tp_eaa2796aa0d2a1ba9981ed5cfdaaa640:1
[kubeexec] DEBUG 2018/02/13 17:51:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: edclosid116.bcbsfl.com Pod: glusterfs-0v0r8 Command: lvs --options=lv_name,thin_count --separator=:
Result:   LV:#Thins
  docker-pool:0
  home:
  opt:
  opt-bmc:
  opt-cvt:
  opt-nimsoft:
  root:
  tmp:
  usr:
  usr-local:
  var:
  var:
  var:
  var:
  var:
  var:
  var-log:
  var-log:
  var-log-audit:
  brick_20e909cc4cdb0c5602098d9f0d238504:
  brick_32469e855410e048890257f7a4320088:
  brick_7e1d616a0f74483b461bc4248adcab9d:
  brick_b671e21cb828488def725137220a573f:
  tp_20e909cc4cdb0c5602098d9f0d238504:1
  tp_32469e855410e048890257f7a4320088:1
  tp_7e1d616a0f74483b461bc4248adcab9d:1
  tp_b671e21cb828488def725137220a573f:1
  brick_13d83958f394998eecf255465dff0e7d:
  brick_19959e33de77e0e21eac2bcaeb3b38f4:
  brick_1b9e0b9421561bd0e0f82972344d3b14:
  brick_45c1b69a6e0ae301dd2e1663dc1668f9:
  brick_4e7024b39223076f94745387fb984c0e:
  brick_5d56e3fd09c00d0ec80ed7d90c64b940:
  brick_eeaa39704b1c667e4d4aaa88ed57d90b:
  brick_f0cceab9b03024274912be94ef8ea70c:
  tp_13d83958f394998eecf255465dff0e7d:1
  tp_19959e33de77e0e21eac2bcaeb3b38f4:1
  tp_1b9e0b9421561bd0e0f82972344d3b14:1
  tp_45c1b69a6e0ae301dd2e1663dc1668f9:1
  tp_4e7024b39223076f94745387fb984c0e:1
  tp_5d56e3fd09c00d0ec80ed7d90c64b940:1
  tp_eeaa39704b1c667e4d4aaa88ed57d90b:1
  tp_f0cceab9b03024274912be94ef8ea70c:1
[asynchttp] INFO 2018/02/13 17:51:45 Completed job da24e7adb82a87dbe63571adaa0d8c9c in 331.529285ms
[heketi] ERROR 2018/02/13 17:51:45 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:584: Cannot delete thin pool tp_74223a190c197a64d0f92729b7f7241e on edclosid116.bcbsfl.com because it is used by [-1] snapshot(s) or cloned volume(s)
[heketi] ERROR 2018/02/13 17:51:45 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:429: Cannot delete thin pool tp_74223a190c197a64d0f92729b7f7241e on edclosid116.bcbsfl.com because it is used by [-1] snapshot(s) or cloned volume(s)
[heketi] ERROR 2018/02/13 17:51:45 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:278: Failed to delete volume 53ef9e5d676bf5233e5c8920279e76e9: Cannot delete thin pool tp_74223a190c197a64d0f92729b7f7241e on edclosid116.bcbsfl.com because it is used by [-1] snapshot(s) or cloned volume(s)
[negroni] Started GET /queue/da24e7adb82a87dbe63571adaa0d8c9c
[negroni] Completed 500 Internal Server Error in 72.294µs


======================================================================


root@edclosmd111 /root:>export  HEKETI_CLI_SERVER=http://heketi-gluster-storage.ose-dev.bcbsfl.com
root@edclosmd111 /root:>heketi-cli topology info

Cluster Id: 6aa026659670f2e265ee3b7e56e75729

    Volumes:

        Name: vol_47340003d1be9d8ebfa6fd28787aa203
        Size: 2
        Id: 47340003d1be9d8ebfa6fd28787aa203
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_47340003d1be9d8ebfa6fd28787aa203
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 54e18068adce69989bed726673b3beec
                        Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_54e18068adce69989bed726673b3beec/brick
                        Size (GiB): 2
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: f013ae5aea3e5c9cc6290aea55dfc24d

                        Id: 7e1d616a0f74483b461bc4248adcab9d
                        Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_7e1d616a0f74483b461bc4248adcab9d/brick
                        Size (GiB): 2
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: 6aa696b595543258c93bcfdcfdddcd66

                        Id: fad582bf40e8e3ac8a89caa119a20476
                        Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_fad582bf40e8e3ac8a89caa119a20476/brick
                        Size (GiB): 2
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 04dec09775c2c880b6fb0c9fb45ed045


        Name: vol_53ef9e5d676bf5233e5c8920279e76e9
        Size: 1
        Id: 53ef9e5d676bf5233e5c8920279e76e9
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_53ef9e5d676bf5233e5c8920279e76e9
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 0ae4564f42771f56b3a0e32e57557134
                        Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_0ae4564f42771f56b3a0e32e57557134/brick
                        Size (GiB): 1
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: f013ae5aea3e5c9cc6290aea55dfc24d

                        Id: 74223a190c197a64d0f92729b7f7241e
                        Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_74223a190c197a64d0f92729b7f7241e/brick
                        Size (GiB): 1
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: 6aa696b595543258c93bcfdcfdddcd66

                        Id: e061dfd76f047fe86b3f9efeb7a87c68
                        Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_e061dfd76f047fe86b3f9efeb7a87c68/brick
                        Size (GiB): 1
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 04dec09775c2c880b6fb0c9fb45ed045


        Name: vol_5ff37a64e96a5a643f3ab671c1831fe0
        Size: 1
        Id: 5ff37a64e96a5a643f3ab671c1831fe0
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_5ff37a64e96a5a643f3ab671c1831fe0
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 407b39ef7a3dfd23f50536d5064f6360
                        Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_407b39ef7a3dfd23f50536d5064f6360/brick
                        Size (GiB): 1
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 7ba0222da7c46dc6bad701d0c3bafa7b

                        Id: 45c1b69a6e0ae301dd2e1663dc1668f9
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_45c1b69a6e0ae301dd2e1663dc1668f9/brick
                        Size (GiB): 1
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787

                        Id: 4d874f689c98afc338fb6f13c23a2a0c
                        Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_4d874f689c98afc338fb6f13c23a2a0c/brick
                        Size (GiB): 1
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: f013ae5aea3e5c9cc6290aea55dfc24d


        Name: vol_65003318d2db592c39da38cd533228f3
        Size: 1
        Id: 65003318d2db592c39da38cd533228f3
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_65003318d2db592c39da38cd533228f3
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 13d83958f394998eecf255465dff0e7d
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_13d83958f394998eecf255465dff0e7d/brick
                        Size (GiB): 1
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787

                        Id: a7c2ffc8fa44d0652fa5e5c546b774b6
                        Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_a7c2ffc8fa44d0652fa5e5c546b774b6/brick
                        Size (GiB): 1
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: 475f1ad2bc9e10b9462d012b5e4c172e

                        Id: f9f8af21f3b25c473cb8ed20710bab07
                        Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_f9f8af21f3b25c473cb8ed20710bab07/brick
                        Size (GiB): 1
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 7ba0222da7c46dc6bad701d0c3bafa7b


        Name: vol_66e791788a564f3d17690a64014fa385
        Size: 1
        Id: 66e791788a564f3d17690a64014fa385
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_66e791788a564f3d17690a64014fa385
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 2dee7499f805f2631d554f7c2432f4d3
                        Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_2dee7499f805f2631d554f7c2432f4d3/brick
                        Size (GiB): 1
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: f013ae5aea3e5c9cc6290aea55dfc24d

                        Id: 32469e855410e048890257f7a4320088
                        Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_32469e855410e048890257f7a4320088/brick
                        Size (GiB): 1
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: 6aa696b595543258c93bcfdcfdddcd66

                        Id: 69629d4438e16d45227b2e6a84374d0b
                        Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_69629d4438e16d45227b2e6a84374d0b/brick
                        Size (GiB): 1
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 04dec09775c2c880b6fb0c9fb45ed045


        Name: vol_733424755cd524bdf0d6faf4770ba266
        Size: 40
        Id: 733424755cd524bdf0d6faf4770ba266
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_733424755cd524bdf0d6faf4770ba266
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 20e909cc4cdb0c5602098d9f0d238504
                        Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_20e909cc4cdb0c5602098d9f0d238504/brick
                        Size (GiB): 40
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: 6aa696b595543258c93bcfdcfdddcd66

                        Id: c0c1e0316093ab248e3375a4e9bb3bf3
                        Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_c0c1e0316093ab248e3375a4e9bb3bf3/brick
                        Size (GiB): 40
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 04dec09775c2c880b6fb0c9fb45ed045

                        Id: eaa2796aa0d2a1ba9981ed5cfdaaa640
                        Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_eaa2796aa0d2a1ba9981ed5cfdaaa640/brick
                        Size (GiB): 40
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: f013ae5aea3e5c9cc6290aea55dfc24d


        Name: vol_850c6752ac3a596fc90a3d9d18cdbb1a
        Size: 1
        Id: 850c6752ac3a596fc90a3d9d18cdbb1a
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_850c6752ac3a596fc90a3d9d18cdbb1a
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 025f5a721051d6b897bfc658f0311319
                        Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_025f5a721051d6b897bfc658f0311319/brick
                        Size (GiB): 1
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: 475f1ad2bc9e10b9462d012b5e4c172e

                        Id: 4e7024b39223076f94745387fb984c0e
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_4e7024b39223076f94745387fb984c0e/brick
                        Size (GiB): 1
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787

                        Id: 51005307ee9c3988ccdaecbdd5651282
                        Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_51005307ee9c3988ccdaecbdd5651282/brick
                        Size (GiB): 1
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 7ba0222da7c46dc6bad701d0c3bafa7b


        Name: vol_8c79aec1b42e88bd16cac011de31deaf
        Size: 1
        Id: 8c79aec1b42e88bd16cac011de31deaf
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_8c79aec1b42e88bd16cac011de31deaf
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 1b9e0b9421561bd0e0f82972344d3b14
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_1b9e0b9421561bd0e0f82972344d3b14/brick
                        Size (GiB): 1
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787

                        Id: 7a9645f49829407bdc6e17711cde25dc
                        Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_7a9645f49829407bdc6e17711cde25dc/brick
                        Size (GiB): 1
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 7ba0222da7c46dc6bad701d0c3bafa7b

                        Id: e2595d8174c350d7d37be697d269d636
                        Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_e2595d8174c350d7d37be697d269d636/brick
                        Size (GiB): 1
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: 475f1ad2bc9e10b9462d012b5e4c172e


        Name: vol_a1a87dcdafa7e464d0b175ea35b28652
        Size: 1
        Id: a1a87dcdafa7e464d0b175ea35b28652
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_a1a87dcdafa7e464d0b175ea35b28652
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 19959e33de77e0e21eac2bcaeb3b38f4
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_19959e33de77e0e21eac2bcaeb3b38f4/brick
                        Size (GiB): 1
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787

                        Id: 8c99f2defb1419adfc79677b5dce8b69
                        Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_8c99f2defb1419adfc79677b5dce8b69/brick
                        Size (GiB): 1
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: 475f1ad2bc9e10b9462d012b5e4c172e

                        Id: c031af051eec0105bee9c17b02d24927
                        Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_c031af051eec0105bee9c17b02d24927/brick
                        Size (GiB): 1
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 7ba0222da7c46dc6bad701d0c3bafa7b


        Name: vol_bb9edd5eb9c20a09829d73010886d1f2
        Size: 10
        Id: bb9edd5eb9c20a09829d73010886d1f2
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_bb9edd5eb9c20a09829d73010886d1f2
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: eeaa39704b1c667e4d4aaa88ed57d90b
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_eeaa39704b1c667e4d4aaa88ed57d90b/brick
                        Size (GiB): 10
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787

                        Id: f8ecc7f462fc59bfca6e9803f2c1151a
                        Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_f8ecc7f462fc59bfca6e9803f2c1151a/brick
                        Size (GiB): 10
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 7ba0222da7c46dc6bad701d0c3bafa7b

                        Id: fb6a8b10989840380df4cbb62cb53f91
                        Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_fb6a8b10989840380df4cbb62cb53f91/brick
                        Size (GiB): 10
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: 475f1ad2bc9e10b9462d012b5e4c172e


        Name: vol_d0213a8153bfd6b7395dcd3f5204b5a6
        Size: 10
        Id: d0213a8153bfd6b7395dcd3f5204b5a6
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_d0213a8153bfd6b7395dcd3f5204b5a6
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 2d3c641f97daa67dca8d4c5e2dc81020
                        Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_2d3c641f97daa67dca8d4c5e2dc81020/brick
                        Size (GiB): 10
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: f013ae5aea3e5c9cc6290aea55dfc24d

                        Id: 5d56e3fd09c00d0ec80ed7d90c64b940
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_5d56e3fd09c00d0ec80ed7d90c64b940/brick
                        Size (GiB): 10
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787

                        Id: 6aaacb3795dbbf4a5821d80c25b2c43a
                        Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_6aaacb3795dbbf4a5821d80c25b2c43a/brick
                        Size (GiB): 10
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 04dec09775c2c880b6fb0c9fb45ed045


        Name: vol_d905f769500fda044d79b8e492a6b5bb
        Size: 10
        Id: d905f769500fda044d79b8e492a6b5bb
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:vol_d905f769500fda044d79b8e492a6b5bb
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: 1627d4f6996e1f9b79b9bb4c2b31b871
                        Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_1627d4f6996e1f9b79b9bb4c2b31b871/brick
                        Size (GiB): 10
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: f013ae5aea3e5c9cc6290aea55dfc24d

                        Id: b671e21cb828488def725137220a573f
                        Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_b671e21cb828488def725137220a573f/brick
                        Size (GiB): 10
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: 6aa696b595543258c93bcfdcfdddcd66

                        Id: c35b85fd10e3ec825d6e1d808a3287b7
                        Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_c35b85fd10e3ec825d6e1d808a3287b7/brick
                        Size (GiB): 10
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 04dec09775c2c880b6fb0c9fb45ed045


        Name: heketidbstorage
        Size: 2
        Id: fccf7677bcd704f2103826bf22378e19
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Mount: 172.18.37.168:heketidbstorage
        Mount Options: backup-volfile-servers=172.18.37.170,172.18.37.189
        Durability Type: replicate
        Replica: 3
        Snapshot: Disabled

                Bricks:
                        Id: a612d85b353aadc7ab6cc92be1875ee4
                        Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_a612d85b353aadc7ab6cc92be1875ee4/brick
                        Size (GiB): 2
                        Node: 5ba92e3146229cc6f3ff7268c1f6a1df
                        Device: 475f1ad2bc9e10b9462d012b5e4c172e

                        Id: ef4c65add03ab9caa8b530c62a4b5413
                        Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_ef4c65add03ab9caa8b530c62a4b5413/brick
                        Size (GiB): 2
                        Node: e3e90dd0c30fbb90f0c8fb677fd0a8ec
                        Device: 04dec09775c2c880b6fb0c9fb45ed045

                        Id: f0cceab9b03024274912be94ef8ea70c
                        Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_f0cceab9b03024274912be94ef8ea70c/brick
                        Size (GiB): 2
                        Node: d63fc31e842053f29fc1d24066ccc1c6
                        Device: f2828ffd79a7aa5948e45ea4ddc98787


    Nodes:

        Node Id: 5ba92e3146229cc6f3ff7268c1f6a1df
        State: online
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Zone: 1
        Management Hostname: edclosid117.bcbsfl.com
        Storage Hostname: 172.18.37.189
        Devices:
                Id:475f1ad2bc9e10b9462d012b5e4c172e   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):16      Free (GiB):83    
                        Bricks:
                                Id:025f5a721051d6b897bfc658f0311319   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_025f5a721051d6b897bfc658f0311319/brick
                                Id:8c99f2defb1419adfc79677b5dce8b69   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_8c99f2defb1419adfc79677b5dce8b69/brick
                                Id:a612d85b353aadc7ab6cc92be1875ee4   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_a612d85b353aadc7ab6cc92be1875ee4/brick
                                Id:a7c2ffc8fa44d0652fa5e5c546b774b6   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_a7c2ffc8fa44d0652fa5e5c546b774b6/brick
                                Id:e2595d8174c350d7d37be697d269d636   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_e2595d8174c350d7d37be697d269d636/brick
                                Id:fb6a8b10989840380df4cbb62cb53f91   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_475f1ad2bc9e10b9462d012b5e4c172e/brick_fb6a8b10989840380df4cbb62cb53f91/brick
                Id:f013ae5aea3e5c9cc6290aea55dfc24d   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):65      Free (GiB):34    
                        Bricks:
                                Id:0ae4564f42771f56b3a0e32e57557134   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_0ae4564f42771f56b3a0e32e57557134/brick
                                Id:1627d4f6996e1f9b79b9bb4c2b31b871   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_1627d4f6996e1f9b79b9bb4c2b31b871/brick
                                Id:2d3c641f97daa67dca8d4c5e2dc81020   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_2d3c641f97daa67dca8d4c5e2dc81020/brick
                                Id:2dee7499f805f2631d554f7c2432f4d3   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_2dee7499f805f2631d554f7c2432f4d3/brick
                                Id:4d874f689c98afc338fb6f13c23a2a0c   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_4d874f689c98afc338fb6f13c23a2a0c/brick
                                Id:54e18068adce69989bed726673b3beec   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_54e18068adce69989bed726673b3beec/brick
                                Id:eaa2796aa0d2a1ba9981ed5cfdaaa640   Size (GiB):40      Path: /var/lib/heketi/mounts/vg_f013ae5aea3e5c9cc6290aea55dfc24d/brick_eaa2796aa0d2a1ba9981ed5cfdaaa640/brick

        Node Id: d63fc31e842053f29fc1d24066ccc1c6
        State: online
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Zone: 1
        Management Hostname: edclosid116.bcbsfl.com
        Storage Hostname: 172.18.37.170
        Devices:
                Id:6aa696b595543258c93bcfdcfdddcd66   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):54      Free (GiB):45    
                        Bricks:
                                Id:20e909cc4cdb0c5602098d9f0d238504   Size (GiB):40      Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_20e909cc4cdb0c5602098d9f0d238504/brick
                                Id:32469e855410e048890257f7a4320088   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_32469e855410e048890257f7a4320088/brick
                                Id:74223a190c197a64d0f92729b7f7241e   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_74223a190c197a64d0f92729b7f7241e/brick
                                Id:7e1d616a0f74483b461bc4248adcab9d   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_7e1d616a0f74483b461bc4248adcab9d/brick
                                Id:b671e21cb828488def725137220a573f   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_6aa696b595543258c93bcfdcfdddcd66/brick_b671e21cb828488def725137220a573f/brick
                Id:f2828ffd79a7aa5948e45ea4ddc98787   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):27      Free (GiB):72    
                        Bricks:
                                Id:13d83958f394998eecf255465dff0e7d   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_13d83958f394998eecf255465dff0e7d/brick
                                Id:19959e33de77e0e21eac2bcaeb3b38f4   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_19959e33de77e0e21eac2bcaeb3b38f4/brick
                                Id:1b9e0b9421561bd0e0f82972344d3b14   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_1b9e0b9421561bd0e0f82972344d3b14/brick
                                Id:45c1b69a6e0ae301dd2e1663dc1668f9   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_45c1b69a6e0ae301dd2e1663dc1668f9/brick
                                Id:4e7024b39223076f94745387fb984c0e   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_4e7024b39223076f94745387fb984c0e/brick
                                Id:5d56e3fd09c00d0ec80ed7d90c64b940   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_5d56e3fd09c00d0ec80ed7d90c64b940/brick
                                Id:eeaa39704b1c667e4d4aaa88ed57d90b   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_eeaa39704b1c667e4d4aaa88ed57d90b/brick
                                Id:f0cceab9b03024274912be94ef8ea70c   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_f2828ffd79a7aa5948e45ea4ddc98787/brick_f0cceab9b03024274912be94ef8ea70c/brick

        Node Id: e3e90dd0c30fbb90f0c8fb677fd0a8ec
        State: online
        Cluster Id: 6aa026659670f2e265ee3b7e56e75729
        Zone: 1
        Management Hostname: edclosid115.bcbsfl.com
        Storage Hostname: 172.18.37.168
        Devices:
                Id:04dec09775c2c880b6fb0c9fb45ed045   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):66      Free (GiB):33    
                        Bricks:
                                Id:69629d4438e16d45227b2e6a84374d0b   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_69629d4438e16d45227b2e6a84374d0b/brick
                                Id:6aaacb3795dbbf4a5821d80c25b2c43a   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_6aaacb3795dbbf4a5821d80c25b2c43a/brick
                                Id:c0c1e0316093ab248e3375a4e9bb3bf3   Size (GiB):40      Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_c0c1e0316093ab248e3375a4e9bb3bf3/brick
                                Id:c35b85fd10e3ec825d6e1d808a3287b7   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_c35b85fd10e3ec825d6e1d808a3287b7/brick
                                Id:e061dfd76f047fe86b3f9efeb7a87c68   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_e061dfd76f047fe86b3f9efeb7a87c68/brick
                                Id:ef4c65add03ab9caa8b530c62a4b5413   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_ef4c65add03ab9caa8b530c62a4b5413/brick
                                Id:fad582bf40e8e3ac8a89caa119a20476   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_04dec09775c2c880b6fb0c9fb45ed045/brick_fad582bf40e8e3ac8a89caa119a20476/brick
                Id:7ba0222da7c46dc6bad701d0c3bafa7b   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):15      Free (GiB):84    
                        Bricks:
                                Id:407b39ef7a3dfd23f50536d5064f6360   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_407b39ef7a3dfd23f50536d5064f6360/brick
                                Id:51005307ee9c3988ccdaecbdd5651282   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_51005307ee9c3988ccdaecbdd5651282/brick
                                Id:7a9645f49829407bdc6e17711cde25dc   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_7a9645f49829407bdc6e17711cde25dc/brick
                                Id:c031af051eec0105bee9c17b02d24927   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_c031af051eec0105bee9c17b02d24927/brick
                                Id:f8ecc7f462fc59bfca6e9803f2c1151a   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_f8ecc7f462fc59bfca6e9803f2c1151a/brick
                                Id:f9f8af21f3b25c473cb8ed20710bab07   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_7ba0222da7c46dc6bad701d0c3bafa7b/brick_f9f8af21f3b25c473cb8ed20710bab07/brick
Comment 4 Olek Toborek 2018-03-15 18:22:42 UTC 
Hello.

That is really good news that in 3.9 it will be available, but what we can advise customer for the time being? He needs to clean up the thin pool situation, and he is looking for relief.

- Olek.
Comment 20 Raghavendra Talur 2018-07-06 20:03:05 UTC 
A possible test case for this would be:

1. create a volume using heketi
2. delete the volume using gluster
3. delete the volume using heketi

Compare the user experience in a previous build and in fixed-in-version build.
Comment 22 Anjana KD 2018-08-30 14:20:30 UTC 
Updated doc text in the Doc Text field. Please review for technical accuracy.
Comment 23 John Mulligan 2018-08-30 17:15:09 UTC 
Doc Text LGTM
Comment 25 errata-xmlrpc 2018-09-12 09:22:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2686