Created attachment 1513018 [details] heketi_pod_logs.txt Description of problem: If we install OCP3.10, create standalone Gluster cluster (CRS) and then deploy containerized heketi using "openshift-ansible" library, then we are not able to create block-based PVCs. It is able to create file volumes, but not block ones. According to the logs, there is some problem in autocreation of block hosting volume. Version-Release number of selected component (if applicable): Heketi docker image (GA): - registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7:v3.10 - docker-pullable://registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7@sha256:2e9ac0e979e63e9683e7d9c722c0b8413f131e3afbc83d61a952ddb06781eae8 OC Version: - oc v3.10.72 - kubernetes v1.10.0+b81c8f8 - openshift v3.10.72 - kubernetes v1.10.0+b81c8f8 Openshift Ansible lib version: 3.10.73 (GA) How reproducible: 100% Steps to Reproduce: 1. Create standalone Gluster cluster 2. Deploy OCP and containerized heketi with enabled autocreation of blockhosting volume. 3. Try to create block hosting volume. Actual results: Creation of block-based PVC fails. Expected results: Creation of block-based PVC should be successful. Additional info: 1) Failed PVC info: $ oc describe pvc pvc-block Name: pvc-block Namespace: default StorageClass: glusterfs-crs-block Status: Pending Volume: Labels: <none> Annotations: control-plane.alpha.kubernetes.io/leader={"holderIdentity":"5f028468-fc5f-11e8-8073-0a58ac10020c","leaseDurationSeconds":15,"acquireTime":"2018-12-10T09:51:42Z","renewTime":"2018-12-10T09:52:15Z","lea... volume.beta.kubernetes.io/storage-class=glusterfs-crs-block volume.beta.kubernetes.io/storage-provisioner=gluster.org/glusterblock Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 12s gluster.org/glusterblock 5f028468-fc5f-11e8-8073-0a58ac10020c Failed to provision volume with StorageClass "glusterfs-crs-block": failed to create volume: heketi block volume creation failed: [heketi] failed to create volume: WARNING: This metadata update is NOT backed up. lvcreate: metadata/pv_map.c:198: consume_pv_area: Assertion `to_go <= pva->count' failed. Normal ExternalProvisioning 1s (x20 over 34s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "gluster.org/glusterblock" or manually created by system administrator Normal Provisioning 1s (x6 over 34s) gluster.org/glusterblock 5f028468-fc5f-11e8-8073-0a58ac10020c External provisioner is provisioning volume for claim "default/pvc-block" Warning ProvisioningFailed 1s (x5 over 10s) gluster.org/glusterblock 5f028468-fc5f-11e8-8073-0a58ac10020c Failed to provision volume with StorageClass "glusterfs-crs-block": failed to create volume: heketi block volume creation failed: [heketi] failed to create volume: Server busy. Retry operation later.
Created attachment 1513019 [details] heketi_pod_info.txt
Created attachment 1513020 [details] heketi_json_config.txt
Created attachment 1513021 [details] heketi_db.json
This appears to be an lvm issue (at the very least it should be giving us a proper error message if we fed it a bad command). We may need an LVM expert to look at it. Can you grab the outputs for 'lvs', 'vgs' and 'pvs' for the nodes? Thanks.
[root@dhcp47-97 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp47-42 lvm2 a-- <83.00g 0 /dev/sdd vg_4a1b2772b8a3d2be7b22e878686cfda0 lvm2 a-- <2.00t <1.89t /dev/sde vg_af665090fe3e5d7a253caa4d5f523da1 lvm2 a-- <2.00t <2.00t [root@dhcp46-70 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp47-42 lvm2 a-- <83.00g 0 /dev/sdd vg_416c1e0a6d83780628cb46ac67c58ad1 lvm2 a-- <2.00t 1.90t /dev/sde vg_aa776359e194b7d4a6c75dfac26a8b9e lvm2 a-- <2.00t <1.99t [root@dhcp46-152 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp47-42 lvm2 a-- <83.00g 0 /dev/sdd vg_ad6bda58895fa8a2c0148d228a04626c lvm2 a-- <2.00t 1.89t /dev/sde vg_8de5c63759427f59adc7e07119fb73f2 lvm2 a-- <2.00t <2.00t -------------------------------------------------------------------------- [root@dhcp47-97 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp47-42 1 2 0 wz--n- <83.00g 0 vg_4a1b2772b8a3d2be7b22e878686cfda0 1 8 0 wz--n- <2.00t <1.89t vg_af665090fe3e5d7a253caa4d5f523da1 1 0 0 wz--n- <2.00t <2.00t [root@dhcp46-70 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp47-42 1 2 0 wz--n- <83.00g 0 vg_416c1e0a6d83780628cb46ac67c58ad1 1 2 0 wz--n- <2.00t 1.90t vg_aa776359e194b7d4a6c75dfac26a8b9e 1 6 0 wz--n- <2.00t <1.99t [root@dhcp46-152 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp47-42 1 2 0 wz--n- <83.00g 0 vg_8de5c63759427f59adc7e07119fb73f2 1 2 0 wz--n- <2.00t <2.00t vg_ad6bda58895fa8a2c0148d228a04626c 1 6 0 wz--n- <2.00t 1.89t -------------------------------------------------------------------------- [root@dhcp47-97 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel_dhcp47-42 -wi-ao---- <33.00g root rhel_dhcp47-42 -wi-ao---- 50.00g brick_c44e5da892a30260febc143e4ab58789 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 100.00g tp_c44e5da892a30260febc143e4ab58789 0.06 brick_d04ea81ba88123242dbbcd91c146a5ea vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 2.00g tp_d04ea81ba88123242dbbcd91c146a5ea 0.70 brick_f474f535ad3edea0f9514ec13d724ae9 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 10.00g tp_212277d8ff5c82a9d155fbfe8525d835 0.17 brick_f9eab198a86731eb09ff3a3398b7f6a8 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 1.00g tp_f9eab198a86731eb09ff3a3398b7f6a8 1.39 tp_212277d8ff5c82a9d155fbfe8525d835 vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 10.00g 0.17 10.09 tp_c44e5da892a30260febc143e4ab58789 vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 100.00g 0.06 3.16 tp_d04ea81ba88123242dbbcd91c146a5ea vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 2.00g 0.70 10.32 tp_f9eab198a86731eb09ff3a3398b7f6a8 vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 1.00g 1.39 10.45 [root@dhcp46-70 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel_dhcp47-42 -wi-ao---- <33.00g root rhel_dhcp47-42 -wi-ao---- 50.00g brick_ad889820a490e42021ab3da578cbcc30 vg_416c1e0a6d83780628cb46ac67c58ad1 Vwi-aotz-- 100.00g tp_255db5b8b9bb07472e83b5418f0c0984 0.06 tp_255db5b8b9bb07472e83b5418f0c0984 vg_416c1e0a6d83780628cb46ac67c58ad1 twi-aotz-- 100.00g 0.06 3.16 brick_355c5c27dd914c84c952b685803e3d11 vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz-- 2.00g tp_741b50fd7998673649a284cdb31a1246 0.70 brick_3eb382fa8bab1d1a3dfbf041d680c40a vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz-- 1.00g tp_3eb382fa8bab1d1a3dfbf041d680c40a 1.39 brick_71c76198b95e9d84d1feb55c7e0e50c0 vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz-- 10.00g tp_71c76198b95e9d84d1feb55c7e0e50c0 0.17 tp_3eb382fa8bab1d1a3dfbf041d680c40a vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz-- 1.00g 1.39 10.45 tp_71c76198b95e9d84d1feb55c7e0e50c0 vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz-- 10.00g 0.17 10.09 tp_741b50fd7998673649a284cdb31a1246 vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz-- 2.00g 0.70 10.32 [root@dhcp46-152 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel_dhcp47-42 -wi-ao---- <33.00g root rhel_dhcp47-42 -wi-ao---- 50.00g brick_b476bde69e90f2dc0b4d1259bc5c73a3 vg_8de5c63759427f59adc7e07119fb73f2 Vwi-aotz-- 2.00g tp_b476bde69e90f2dc0b4d1259bc5c73a3 0.70 tp_b476bde69e90f2dc0b4d1259bc5c73a3 vg_8de5c63759427f59adc7e07119fb73f2 twi-aotz-- 2.00g 0.70 10.32 brick_2534e2d7f5825eefa5e719cbfad7a0ef vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz-- 100.00g tp_2534e2d7f5825eefa5e719cbfad7a0ef 0.06 brick_a60d8334f78c3ab9270e84f051d9f9f0 vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz-- 1.00g tp_f1b6384cf8b3c5f0c40310cef0480f32 1.39 brick_d0d9d96b76e89ef597eb1e0015bb3fc1 vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz-- 10.00g tp_d0d9d96b76e89ef597eb1e0015bb3fc1 0.17 tp_2534e2d7f5825eefa5e719cbfad7a0ef vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz-- 100.00g 0.06 3.16 tp_d0d9d96b76e89ef597eb1e0015bb3fc1 vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz-- 10.00g 0.17 10.09 tp_f1b6384cf8b3c5f0c40310cef0480f32 vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz-- 1.00g 1.39 10.45 ----------------------------------------------------------------------------- [root@dhcp46-236 ~]# heketi-cli topology info Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 File: true Block: true Volumes: Name: ms-file Size: 10 Id: 7bb9330bb39b794ee94bc8e92cdd9b99 Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Mount: 10.70.46.152:ms-file Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 71c76198b95e9d84d1feb55c7e0e50c0 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick Size (GiB): 10 Node: 284a68e0c3628b8d0c6d684e92fb908c Device: aa776359e194b7d4a6c75dfac26a8b9e Id: d0d9d96b76e89ef597eb1e0015bb3fc1 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick Size (GiB): 10 Node: 080daebb1e6fb61158ad527948470491 Device: ad6bda58895fa8a2c0148d228a04626c Id: f474f535ad3edea0f9514ec13d724ae9 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick Size (GiB): 10 Node: fbb42c14f89246d1ec799413ec1be9e9 Device: 4a1b2772b8a3d2be7b22e878686cfda0 Name: heketidbstorage Size: 2 Id: 808731c9656b6d2e3817b601259dae41 Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Mount: 10.70.46.152:heketidbstorage Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 355c5c27dd914c84c952b685803e3d11 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick Size (GiB): 2 Node: 284a68e0c3628b8d0c6d684e92fb908c Device: aa776359e194b7d4a6c75dfac26a8b9e Id: b476bde69e90f2dc0b4d1259bc5c73a3 Path: /var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick Size (GiB): 2 Node: 080daebb1e6fb61158ad527948470491 Device: 8de5c63759427f59adc7e07119fb73f2 Id: d04ea81ba88123242dbbcd91c146a5ea Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick Size (GiB): 2 Node: fbb42c14f89246d1ec799413ec1be9e9 Device: 4a1b2772b8a3d2be7b22e878686cfda0 Name: vol_b896ada19d8fee0f3009cb9deb861bdb Size: 1 Id: b896ada19d8fee0f3009cb9deb861bdb Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Mount: 10.70.46.152:vol_b896ada19d8fee0f3009cb9deb861bdb Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 3eb382fa8bab1d1a3dfbf041d680c40a Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick Size (GiB): 1 Node: 284a68e0c3628b8d0c6d684e92fb908c Device: aa776359e194b7d4a6c75dfac26a8b9e Id: a60d8334f78c3ab9270e84f051d9f9f0 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick Size (GiB): 1 Node: 080daebb1e6fb61158ad527948470491 Device: ad6bda58895fa8a2c0148d228a04626c Id: f9eab198a86731eb09ff3a3398b7f6a8 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick Size (GiB): 1 Node: fbb42c14f89246d1ec799413ec1be9e9 Device: 4a1b2772b8a3d2be7b22e878686cfda0 Nodes: Node Id: 080daebb1e6fb61158ad527948470491 State: online Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Zone: 3 Management Hostnames: dhcp46-152.lab.eng.blr.redhat.com Storage Hostnames: 10.70.46.152 Devices: Id:8de5c63759427f59adc7e07119fb73f2 Name:/dev/sde State:online Size (GiB):2047 Used (GiB):2 Free (GiB):2045 Bricks: Id:b476bde69e90f2dc0b4d1259bc5c73a3 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick Id:ad6bda58895fa8a2c0148d228a04626c Name:/dev/sdd State:online Size (GiB):2047 Used (GiB):111 Free (GiB):1936 Bricks: Id:2534e2d7f5825eefa5e719cbfad7a0ef Size (GiB):100 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_2534e2d7f5825eefa5e719cbfad7a0ef/brick Id:a60d8334f78c3ab9270e84f051d9f9f0 Size (GiB):1 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick Id:d0d9d96b76e89ef597eb1e0015bb3fc1 Size (GiB):10 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick Node Id: 284a68e0c3628b8d0c6d684e92fb908c State: online Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Zone: 2 Management Hostnames: dhcp46-70.lab.eng.blr.redhat.com Storage Hostnames: 10.70.46.70 Devices: Id:416c1e0a6d83780628cb46ac67c58ad1 Name:/dev/sdd State:online Size (GiB):2047 Used (GiB):100 Free (GiB):1947 Bricks: Id:ad889820a490e42021ab3da578cbcc30 Size (GiB):100 Path: /var/lib/heketi/mounts/vg_416c1e0a6d83780628cb46ac67c58ad1/brick_ad889820a490e42021ab3da578cbcc30/brick Id:aa776359e194b7d4a6c75dfac26a8b9e Name:/dev/sde State:online Size (GiB):2047 Used (GiB):13 Free (GiB):2034 Bricks: Id:355c5c27dd914c84c952b685803e3d11 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick Id:3eb382fa8bab1d1a3dfbf041d680c40a Size (GiB):1 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick Id:71c76198b95e9d84d1feb55c7e0e50c0 Size (GiB):10 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick Node Id: fbb42c14f89246d1ec799413ec1be9e9 State: online Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Zone: 1 Management Hostnames: dhcp47-97.lab.eng.blr.redhat.com Storage Hostnames: 10.70.47.97 Devices: Id:4a1b2772b8a3d2be7b22e878686cfda0 Name:/dev/sdd State:online Size (GiB):2047 Used (GiB):113 Free (GiB):1934 Bricks: Id:c44e5da892a30260febc143e4ab58789 Size (GiB):100 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_c44e5da892a30260febc143e4ab58789/brick Id:d04ea81ba88123242dbbcd91c146a5ea Size (GiB):2 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick Id:f474f535ad3edea0f9514ec13d724ae9 Size (GiB):10 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick Id:f9eab198a86731eb09ff3a3398b7f6a8 Size (GiB):1 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick Id:af665090fe3e5d7a253caa4d5f523da1 Name:/dev/sde State:online Size (GiB):2047 Used (GiB):0 Free (GiB):2047 Bricks:
Providing the output from the CRS nodes (the same nodes from the bug report) for "pvs", "vgs" and "lvs" commands: ====================================== [root@vp-ansible-v310-ga8-crs-0 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp46-210 lvm2 a-- <39.00g 0 /dev/sdd vg_337efc571e4f47a03381596e1f85ef3a lvm2 a-- 99.87g <99.38g /dev/sde vg_5e6dc1c01eab77c489313acb5f99bb69 lvm2 a-- 199.87g <197.85g [root@vp-ansible-v310-ga8-crs-0 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp46-210 1 2 0 wz--n- <39.00g 0 vg_337efc571e4f47a03381596e1f85ef3a 1 1 0 wz--n- 99.87g <99.38g vg_5e6dc1c01eab77c489313acb5f99bb69 1 2 0 wz--n- 199.87g <197.85g [root@vp-ansible-v310-ga8-crs-0 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel_dhcp46-210 -wi-ao---- <35.00g swap rhel_dhcp46-210 -wi-ao---- 4.00g lvol0 vg_337efc571e4f47a03381596e1f85ef3a -wi------- 508.00m brick_4ff316665628811fab824af27bda33b4 vg_5e6dc1c01eab77c489313acb5f99bb69 Vwi-aotz-- 2.00g tp_b66f04be0f45b59eced07fb9957ddf6d 0.70 tp_b66f04be0f45b59eced07fb9957ddf6d vg_5e6dc1c01eab77c489313acb5f99bb69 twi-aotz-- 2.00g 0.70 10.32 ====================================== [root@vp-ansible-v310-ga8-crs-1 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp46-210 lvm2 a-- <39.00g 0 /dev/sdd vg_f80c8a68e60735d509a1b095a685825a lvm2 a-- 99.87g <99.38g /dev/sde vg_7977025748559045ba194a73d217e2a9 lvm2 a-- 199.87g <197.85g [root@vp-ansible-v310-ga8-crs-1 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp46-210 1 2 0 wz--n- <39.00g 0 vg_7977025748559045ba194a73d217e2a9 1 2 0 wz--n- 199.87g <197.85g vg_f80c8a68e60735d509a1b095a685825a 1 1 0 wz--n- 99.87g <99.38g [root@vp-ansible-v310-ga8-crs-1 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel_dhcp46-210 -wi-ao---- <35.00g swap rhel_dhcp46-210 -wi-ao---- 4.00g brick_cb656003dfa34bb4fa678e37a6d1b9ee vg_7977025748559045ba194a73d217e2a9 Vwi-aotz-- 2.00g tp_cb656003dfa34bb4fa678e37a6d1b9ee 0.70 tp_cb656003dfa34bb4fa678e37a6d1b9ee vg_7977025748559045ba194a73d217e2a9 twi-aotz-- 2.00g 0.70 10.32 lvol0 vg_f80c8a68e60735d509a1b095a685825a -wi------- 508.00m ====================================== [root@vp-ansible-v310-ga8-crs-2 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp46-210 lvm2 a-- <39.00g 0 /dev/sdd vg_f477b4c896bffe376846d2aa58d76b4f lvm2 a-- 99.87g <97.85g /dev/sde vg_f90aa2b7ffaadbf2e135fa676612e9ba lvm2 a-- 199.87g 199.87g [root@vp-ansible-v310-ga8-crs-2 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp46-210 1 2 0 wz--n- <39.00g 0 vg_f477b4c896bffe376846d2aa58d76b4f 1 2 0 wz--n- 99.87g <97.85g vg_f90aa2b7ffaadbf2e135fa676612e9ba 1 0 0 wz--n- 199.87g 199.87g [root@vp-ansible-v310-ga8-crs-2 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel_dhcp46-210 -wi-ao---- <35.00g swap rhel_dhcp46-210 -wi-ao---- 4.00g brick_29a5513340bae6e65301cbb11f98ae7b vg_f477b4c896bffe376846d2aa58d76b4f Vwi-aotz-- 2.00g tp_29a5513340bae6e65301cbb11f98ae7b 0.70 tp_29a5513340bae6e65301cbb11f98ae7b vg_f477b4c896bffe376846d2aa58d76b4f twi-aotz-- 2.00g 0.70 10.32 ======================================
Seeing this issue with OCP 3.11.1 (live)+ OCS 3.11(RHGS BU2) CRS setup as well (1658166 - Block hosting volume and block volume creation failing in Independent mode setup If block services are not running initially). Output provided for pvs, vgs and lvs in comment #7 is respective to bz-1658166.
@rtalur, it is completely unclear what exactly you want me to provide. Please, write message writing list of things you need.
(In reply to Valerii Ponomarov from comment #11) > @rtalur, it is completely unclear what exactly you want me to provide. > Please, write message writing list of things you need. Let's start with the following and we can add to the list if needed: On heketi pod: rpm -q heketi From within the gluster pods: rpm -qa | grep lvm2 Directly on the nodes (the nodes currently hosting the pods): rpm -qa | grep lvm2
John, this bug is about CRS setup. So, providing following info: ### Heketi POD info $ rpm -q heketi heketi-7.0.0-11.el7rhgs.x86_64 ### CRS node 'vp-ansible-v310-ga8-crs-0' $ rpm -qa | grep lvm2 lvm2-libs-2.02.180-10.el7_6.2.x86_64 lvm2-2.02.180-10.el7_6.2.x86_64 ### CRS node 'vp-ansible-v310-ga8-crs-1' $ rpm -qa | grep lvm2 lvm2-libs-2.02.180-10.el7_6.2.x86_64 lvm2-2.02.180-10.el7_6.2.x86_64 ### CRS node 'vp-ansible-v310-ga8-crs-2' $ rpm -qa | grep lvm2 lvm2-libs-2.02.180-10.el7_6.2.x86_64 lvm2-2.02.180-10.el7_6.2.x86_64
Karthick, it is unclear for me, what kind of an information I am requested to provide. In addition to the already existing.
(In reply to John Mulligan from comment #16) > FWIW we were doing (other unrelated) volume creations on a independent mode > set up with heketi master. This problem does not appear to appear on all > independent mode clusters. > > It sounds like this problem is only existing on one node on one cluster. > Please correct this statement if I am wrong. This is the question: Is it correct that you are seeing this problem only on one node of one specific cluster? (background is: we have not been able to reproduce)
Ok, now I see, that there is lots of private comments which are not visible for me and I only saw "needinfo" requests. So, I going to start one more deployment with CRS and OCP3.10 using GA packages. Then will try to reproduce the bug, and, provide the results here. In case of failure I can share the setup with anyone from dev team.
Reporting the results: With the latest following GA packages bug haven't been reproduced: Version of the "openshift-ansible" lib: - openshift-ansible-3.10.89-1 Version of the Kubernetes: - kubernetes v1.10.0+b81c8f8 Heketi POD: - heketi-7.0.0-11.el7rhgs.x86_64 - heketi-client-7.0.0-11.el7rhgs.x86_64 - Storage release version: Red Hat Gluster Volume Manager 3.4.0 (Container) Gluster nodes storage release version: - Red Hat Gluster Storage Server 3.4 Gluster nodes packages: - glusterfs-3.12.2-32.el7rhgs.x86_64 - gluster-block-0.2.1-28.el7rhgs.x86_64 - targetcli-2.1.fb46-7.el7.noarch - libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.3.x86_64 - tcmu-runner-1.2.0-26.el7rhgs.x86_64 - vdsm-gluster-4.19.43-2.3.el7rhgs.noarch So, this bug is not actual anymore.
Removing the blocker flag based on comment#22.
Even after adding me to the "qa" and "redhat" groups, I do not see private comments here. So, I still have no clue what I am asked about with "needinfo".
(In reply to Valerii Ponomarov from comment #26) > Even after adding me to the "qa" and "redhat" groups, I do not see private > comments here. So, I still have no clue what I am asked about with > "needinfo". Sorry to hear that you're hitting permissions issues. It happened to me as well and took a while to sort out. The question was in reply to your comment "So, this bug is not actual anymore.": "Based on this result, can we close this bug ?"
John, I haven't been able to reproduce the bug with the recent GA packages for OCP3.10. So, yes, we can close it.
OK, thanks. Closing.