Description of problem: ======================= Created setup with OCP 3.11.1 + OCS 3.11(RHGS BU2) in Independent mode. Blockd services are initially inactive. after deploying OCP and OCS, started blockd services. Observed that block hosting volume are not getting created. topology info shows size of 100Gi has been assigned to bricks, I see no block hosting volumes from the list. Later tried creating block volumes manually through heketi-cli. where the block volumes are failed to create saying "Error: Server busy. Retry operation later". Later Created block hosting volume manually through heketi-cli, then block hosting volume got created. But the block volumes creation errors out. Version-Release number of selected component (if applicable): ============================================================= glusterfs-api-3.12.2-31.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-31.el7rhgs.x86_64 glusterfs-3.12.2-31.el7rhgs.x86_64 glusterfs-cli-3.12.2-31.el7rhgs.x86_64 glusterfs-rdma-3.12.2-31.el7rhgs.x86_64 gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64 python2-gluster-3.12.2-31.el7rhgs.x86_64 glusterfs-server-3.12.2-31.el7rhgs.x86_64 gluster-block-0.2.1-28.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-31.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch vdsm-gluster-4.19.43-2.3.el7rhgs.noarch glusterfs-libs-3.12.2-31.el7rhgs.x86_64 glusterfs-fuse-3.12.2-31.el7rhgs.x86_64 libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.3.x86_64 libblockdev-2.18-3.el7.x86_64 libblockdev-utils-2.18-3.el7.x86_64 gluster-block-0.2.1-28.el7rhgs.x86_64 How reproducible: ================= 1/1 Steps to Reproduce: =================== 1. Setup with OCP 3.11.1 + OCS 3.11(RHGS BU2) in Independent mode 2. Blockd service was initially inactive. 3. After deploying OCP and OCS, start block services from gluster nodes. 4. Check volume list throught heketi Actual results: =============== Block hosting volume is in pending state and block volumes are not getting created. Expected results: ================= Block hosting volumes and block volumes should be created. Additional info: ================ [root@dhcp47-97 ~]# systemctl status gluster-blockd ● gluster-blockd.service - Gluster block storage utility Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; disabled; vendor preset: disabled) Active: inactive (dead) [root@dhcp47-97 ~]# systemctl start gluster-blockd [root@dhcp47-97 ~]# systemctl status gluster-blockd ● gluster-blockd.service - Gluster block storage utility Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2018-12-11 12:17:17 IST; 14s ago Main PID: 4020 (gluster-blockd) Tasks: 4 CGroup: /system.slice/gluster-blockd.service └─4020 /usr/sbin/gluster-bloc... Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ... Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ... Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ... Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ... Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ... Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ... Hint: Some lines were ellipsized, use -l to show in full. [root@dhcp47-97 ~]# gluster v info Volume Name: heketidbstorage Type: Replicate Volume ID: ed7f7031-23cd-412e-9861-116a55293d2f Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.46.70:/var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick Brick2: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick Brick3: 10.70.46.152:/var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick Options Reconfigured: server.tcp-user-timeout: 42 transport.address-family: inet nfs.disable: on performance.client-io-threads: off Volume Name: ms-file Type: Replicate Volume ID: 075ef8fa-6ebf-431c-aee5-e876d2358114 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick Brick2: 10.70.46.152:/var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick Brick3: 10.70.46.70:/var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick Options Reconfigured: server.tcp-user-timeout: 42 transport.address-family: inet nfs.disable: on performance.client-io-threads: off Volume Name: vol_b896ada19d8fee0f3009cb9deb861bdb Type: Replicate Volume ID: 272e2b30-8fb5-4e06-97c9-51ba952f84c4 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.46.152:/var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick Brick2: 10.70.46.70:/var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick Brick3: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick Options Reconfigured: server.tcp-user-timeout: 42 transport.address-family: inet nfs.disable: on performance.client-io-threads: off Volume Name: vol_c1e42a8cec401f067c9a387275a112de Type: Replicate Volume ID: 3143c004-1836-4cc6-a283-efe774b954f6 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.46.70:/var/lib/heketi/mounts/vg_416c1e0a6d83780628cb46ac67c58ad1/brick_ad889820a490e42021ab3da578cbcc30/brick Brick2: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_c44e5da892a30260febc143e4ab58789/brick Brick3: 10.70.46.152:/var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_2534e2d7f5825eefa5e719cbfad7a0ef/brick Options Reconfigured: cluster.choose-local: off server.allow-insecure: on user.cifs: off features.shard-block-size: 64MB features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: disable performance.strict-o-direct: on performance.readdir-ahead: off performance.open-behind: off performance.stat-prefetch: off performance.io-cache: off performance.read-ahead: off performance.quick-read: off server.tcp-user-timeout: 42 transport.address-family: inet nfs.disable: on performance.client-io-threads: off [root@dhcp47-97 ~]# systemctl status glusterd ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-12-10 18:38:01 IST; 21h ago Process: 7246 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 7269 (glusterd) Tasks: 96 CGroup: /system.slice/glusterd.service ├─3322 /usr/sbin/glusterfsd -... ├─3615 /usr/sbin/glusterfsd -... ├─4501 /usr/sbin/glusterfsd -... ├─4522 /usr/sbin/glusterfs -s... ├─7269 /usr/sbin/glusterd -p ... └─9577 /usr/sbin/glusterfsd -... Dec 10 18:37:58 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ... Dec 10 18:38:01 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ... Hint: Some lines were ellipsized, use -l to show in full. [root@dhcp47-97 ~]# systemctl status gluster-block-targetd Unit gluster-block-targetd.service could not be found. [root@dhcp47-97 ~]# systemctl status gluster-block-target ● gluster-block-target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/gluster-block-target.service; disabled; vendor preset: disabled) Active: active (exited) since Tue 2018-12-11 12:17:17 IST; 4h 13min ago Process: 4008 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 4008 (code=exited, status=0/SUCCESS) Tasks: 0 CGroup: /system.slice/gluster-block-target.service Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ... Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ... Hint: Some lines were ellipsized, use -l to show in full. [root@dhcp47-97 ~]# systemctl status tcmu-runner ● tcmu-runner.service - LIO Userspace-passthrough daemon Loaded: loaded (/usr/lib/systemd/system/tcmu-runner.service; static; vendor preset: disabled) Active: active (running) since Tue 2018-12-11 12:17:16 IST; 4h 13min ago Process: 3827 ExecStartPre=/usr/libexec/gluster-block/upgrade_activities.sh (code=exited, status=0/SUCCESS) Main PID: 4001 (tcmu-runner) Tasks: 5 CGroup: /system.slice/tcmu-runner.service └─4001 /usr/bin/tcmu-runner -... Dec 11 12:17:14 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ... Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com upgrade_activities.sh[3827]: ... Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com tcmu-runner[4001]: ... Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com tcmu-runner[4001]: ... Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ... Hint: Some lines were ellipsized, use -l to show in full. heketi-cli blockvolume create --size=1 Error: Server busy. Retry operation later. [root@dhcp46-236 ~]# heketi-cli server operations info Operation Counts: Total: 1 In-Flight: 0 New: 1 Stale: 0 [root@dhcp46-236 ~]# heketi-cli server operations info Operation Counts: Total: 1 In-Flight: 0 New: 1 Stale: 0 ----------------------xx----------------------------- Heketi logs: ============ [negroni] Started POST /blockvolumes [heketi] WARNING 2018/12/11 06:47:24 temporarily rejecting block volume request: pending block-hosting-volume found [heketi] ERROR 2018/12/11 06:47:24 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations [negroni] Completed 429 Too Many Requests in 4.56255ms ----------------------xx----------------------------- [root@dhcp46-236 ~]# heketi-cli topology info Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 File: true Block: true Volumes: Name: ms-file Size: 10 Id: 7bb9330bb39b794ee94bc8e92cdd9b99 Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Mount: 10.70.46.152:ms-file Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 71c76198b95e9d84d1feb55c7e0e50c0 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick Size (GiB): 10 Node: 284a68e0c3628b8d0c6d684e92fb908c Device: aa776359e194b7d4a6c75dfac26a8b9e Id: d0d9d96b76e89ef597eb1e0015bb3fc1 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick Size (GiB): 10 Node: 080daebb1e6fb61158ad527948470491 Device: ad6bda58895fa8a2c0148d228a04626c Id: f474f535ad3edea0f9514ec13d724ae9 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick Size (GiB): 10 Node: fbb42c14f89246d1ec799413ec1be9e9 Device: 4a1b2772b8a3d2be7b22e878686cfda0 Name: heketidbstorage Size: 2 Id: 808731c9656b6d2e3817b601259dae41 Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Mount: 10.70.46.152:heketidbstorage Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 355c5c27dd914c84c952b685803e3d11 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick Size (GiB): 2 Node: 284a68e0c3628b8d0c6d684e92fb908c Device: aa776359e194b7d4a6c75dfac26a8b9e Id: b476bde69e90f2dc0b4d1259bc5c73a3 Path: /var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick Size (GiB): 2 Node: 080daebb1e6fb61158ad527948470491 Device: 8de5c63759427f59adc7e07119fb73f2 Id: d04ea81ba88123242dbbcd91c146a5ea Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick Size (GiB): 2 Node: fbb42c14f89246d1ec799413ec1be9e9 Device: 4a1b2772b8a3d2be7b22e878686cfda0 Name: vol_b896ada19d8fee0f3009cb9deb861bdb Size: 1 Id: b896ada19d8fee0f3009cb9deb861bdb Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Mount: 10.70.46.152:vol_b896ada19d8fee0f3009cb9deb861bdb Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97 Durability Type: replicate Replica: 3 Snapshot: Disabled Bricks: Id: 3eb382fa8bab1d1a3dfbf041d680c40a Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick Size (GiB): 1 Node: 284a68e0c3628b8d0c6d684e92fb908c Device: aa776359e194b7d4a6c75dfac26a8b9e Id: a60d8334f78c3ab9270e84f051d9f9f0 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick Size (GiB): 1 Node: 080daebb1e6fb61158ad527948470491 Device: ad6bda58895fa8a2c0148d228a04626c Id: f9eab198a86731eb09ff3a3398b7f6a8 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick Size (GiB): 1 Node: fbb42c14f89246d1ec799413ec1be9e9 Device: 4a1b2772b8a3d2be7b22e878686cfda0 Nodes: Node Id: 080daebb1e6fb61158ad527948470491 State: online Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Zone: 3 Management Hostnames: dhcp46-152.lab.eng.blr.redhat.com Storage Hostnames: 10.70.46.152 Devices: Id:8de5c63759427f59adc7e07119fb73f2 Name:/dev/sde State:online Size (GiB):2047 Used (GiB):2 Free (GiB):2045 Bricks: Id:b476bde69e90f2dc0b4d1259bc5c73a3 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick Id:ad6bda58895fa8a2c0148d228a04626c Name:/dev/sdd State:online Size (GiB):2047 Used (GiB):111 Free (GiB):1936 Bricks: Id:2534e2d7f5825eefa5e719cbfad7a0ef Size (GiB):100 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_2534e2d7f5825eefa5e719cbfad7a0ef/brick Id:a60d8334f78c3ab9270e84f051d9f9f0 Size (GiB):1 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick Id:d0d9d96b76e89ef597eb1e0015bb3fc1 Size (GiB):10 Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick Node Id: 284a68e0c3628b8d0c6d684e92fb908c State: online Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Zone: 2 Management Hostnames: dhcp46-70.lab.eng.blr.redhat.com Storage Hostnames: 10.70.46.70 Devices: Id:416c1e0a6d83780628cb46ac67c58ad1 Name:/dev/sdd State:online Size (GiB):2047 Used (GiB):100 Free (GiB):1947 Bricks: Id:ad889820a490e42021ab3da578cbcc30 Size (GiB):100 Path: /var/lib/heketi/mounts/vg_416c1e0a6d83780628cb46ac67c58ad1/brick_ad889820a490e42021ab3da578cbcc30/brick Id:aa776359e194b7d4a6c75dfac26a8b9e Name:/dev/sde State:online Size (GiB):2047 Used (GiB):13 Free (GiB):2034 Bricks: Id:355c5c27dd914c84c952b685803e3d11 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick Id:3eb382fa8bab1d1a3dfbf041d680c40a Size (GiB):1 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick Id:71c76198b95e9d84d1feb55c7e0e50c0 Size (GiB):10 Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick Node Id: fbb42c14f89246d1ec799413ec1be9e9 State: online Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Zone: 1 Management Hostnames: dhcp47-97.lab.eng.blr.redhat.com Storage Hostnames: 10.70.47.97 Devices: Id:4a1b2772b8a3d2be7b22e878686cfda0 Name:/dev/sdd State:online Size (GiB):2047 Used (GiB):113 Free (GiB):1934 Bricks: Id:c44e5da892a30260febc143e4ab58789 Size (GiB):100 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_c44e5da892a30260febc143e4ab58789/brick Id:d04ea81ba88123242dbbcd91c146a5ea Size (GiB):2 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick Id:f474f535ad3edea0f9514ec13d724ae9 Size (GiB):10 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick Id:f9eab198a86731eb09ff3a3398b7f6a8 Size (GiB):1 Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick Id:af665090fe3e5d7a253caa4d5f523da1 Name:/dev/sde State:online Size (GiB):2047 Used (GiB):0 Free (GiB):2047 Bricks: Tried manually Creating block hosting volume and block volume: ============================================================== # heketi-cli volume create --size=100 --block Name: vol_adf620440109d4758c231d4c5a4dec2b Size: 100 Volume Id: adf620440109d4758c231d4c5a4dec2b Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4 Mount: 10.70.46.152:vol_adf620440109d4758c231d4c5a4dec2b Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97 Block: true Free Size: 98 Reserved Size: 2 Block Hosting Restriction: (none) Block Volumes: [] Durability Type: replicate Distributed+Replica: 3 # heketi-cli volume list Id:7bb9330bb39b794ee94bc8e92cdd9b99 Cluster:5dc64d3efe0e4d1173acb9539670e6a4 Name:ms-file Id:808731c9656b6d2e3817b601259dae41 Cluster:5dc64d3efe0e4d1173acb9539670e6a4 Name:heketidbstorage Id:adf620440109d4758c231d4c5a4dec2b Cluster:5dc64d3efe0e4d1173acb9539670e6a4 Name:vol_adf620440109d4758c231d4c5a4dec2b [block] Id:b896ada19d8fee0f3009cb9deb861bdb Cluster:5dc64d3efe0e4d1173acb9539670e6a4 Name:vol_b896ada19d8fee0f3009cb9deb861bdb # heketi-cli blockvolume create --size=5 Error: server did not provide a message (status 500: Internal Server Error)
Adding output of pvs, vgs and lvs: ================================== [root@dhcp47-97 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp47-42 lvm2 a-- <83.00g 0 /dev/sdd vg_4a1b2772b8a3d2be7b22e878686cfda0 lvm2 a-- <2.00t <1.89t /dev/sde vg_af665090fe3e5d7a253caa4d5f523da1 lvm2 a-- <2.00t <2.00t [root@dhcp46-70 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp47-42 lvm2 a-- <83.00g 0 /dev/sdd vg_416c1e0a6d83780628cb46ac67c58ad1 lvm2 a-- <2.00t 1.90t /dev/sde vg_aa776359e194b7d4a6c75dfac26a8b9e lvm2 a-- <2.00t <1.99t [root@dhcp46-152 ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 rhel_dhcp47-42 lvm2 a-- <83.00g 0 /dev/sdd vg_ad6bda58895fa8a2c0148d228a04626c lvm2 a-- <2.00t 1.89t /dev/sde vg_8de5c63759427f59adc7e07119fb73f2 lvm2 a-- <2.00t <2.00t -------------------------------------------------------------------------- [root@dhcp47-97 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp47-42 1 2 0 wz--n- <83.00g 0 vg_4a1b2772b8a3d2be7b22e878686cfda0 1 8 0 wz--n- <2.00t <1.89t vg_af665090fe3e5d7a253caa4d5f523da1 1 0 0 wz--n- <2.00t <2.00t [root@dhcp46-70 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp47-42 1 2 0 wz--n- <83.00g 0 vg_416c1e0a6d83780628cb46ac67c58ad1 1 2 0 wz--n- <2.00t 1.90t vg_aa776359e194b7d4a6c75dfac26a8b9e 1 6 0 wz--n- <2.00t <1.99t [root@dhcp46-152 ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel_dhcp47-42 1 2 0 wz--n- <83.00g 0 vg_8de5c63759427f59adc7e07119fb73f2 1 2 0 wz--n- <2.00t <2.00t vg_ad6bda58895fa8a2c0148d228a04626c 1 6 0 wz--n- <2.00t 1.89t -------------------------------------------------------------------------- [root@dhcp47-97 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel_dhcp47-42 -wi-ao---- <33.00g root rhel_dhcp47-42 -wi-ao---- 50.00g brick_c44e5da892a30260febc143e4ab58789 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 100.00g tp_c44e5da892a30260febc143e4ab58789 0.06 brick_d04ea81ba88123242dbbcd91c146a5ea vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 2.00g tp_d04ea81ba88123242dbbcd91c146a5ea 0.70 brick_f474f535ad3edea0f9514ec13d724ae9 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 10.00g tp_212277d8ff5c82a9d155fbfe8525d835 0.17 brick_f9eab198a86731eb09ff3a3398b7f6a8 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 1.00g tp_f9eab198a86731eb09ff3a3398b7f6a8 1.39 tp_212277d8ff5c82a9d155fbfe8525d835 vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 10.00g 0.17 10.09 tp_c44e5da892a30260febc143e4ab58789 vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 100.00g 0.06 3.16 tp_d04ea81ba88123242dbbcd91c146a5ea vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 2.00g 0.70 10.32 tp_f9eab198a86731eb09ff3a3398b7f6a8 vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 1.00g 1.39 10.45 [root@dhcp46-70 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel_dhcp47-42 -wi-ao---- <33.00g root rhel_dhcp47-42 -wi-ao---- 50.00g brick_ad889820a490e42021ab3da578cbcc30 vg_416c1e0a6d83780628cb46ac67c58ad1 Vwi-aotz-- 100.00g tp_255db5b8b9bb07472e83b5418f0c0984 0.06 tp_255db5b8b9bb07472e83b5418f0c0984 vg_416c1e0a6d83780628cb46ac67c58ad1 twi-aotz-- 100.00g 0.06 3.16 brick_355c5c27dd914c84c952b685803e3d11 vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz-- 2.00g tp_741b50fd7998673649a284cdb31a1246 0.70 brick_3eb382fa8bab1d1a3dfbf041d680c40a vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz-- 1.00g tp_3eb382fa8bab1d1a3dfbf041d680c40a 1.39 brick_71c76198b95e9d84d1feb55c7e0e50c0 vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz-- 10.00g tp_71c76198b95e9d84d1feb55c7e0e50c0 0.17 tp_3eb382fa8bab1d1a3dfbf041d680c40a vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz-- 1.00g 1.39 10.45 tp_71c76198b95e9d84d1feb55c7e0e50c0 vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz-- 10.00g 0.17 10.09 tp_741b50fd7998673649a284cdb31a1246 vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz-- 2.00g 0.70 10.32 [root@dhcp46-152 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home rhel_dhcp47-42 -wi-ao---- <33.00g root rhel_dhcp47-42 -wi-ao---- 50.00g brick_b476bde69e90f2dc0b4d1259bc5c73a3 vg_8de5c63759427f59adc7e07119fb73f2 Vwi-aotz-- 2.00g tp_b476bde69e90f2dc0b4d1259bc5c73a3 0.70 tp_b476bde69e90f2dc0b4d1259bc5c73a3 vg_8de5c63759427f59adc7e07119fb73f2 twi-aotz-- 2.00g 0.70 10.32 brick_2534e2d7f5825eefa5e719cbfad7a0ef vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz-- 100.00g tp_2534e2d7f5825eefa5e719cbfad7a0ef 0.06 brick_a60d8334f78c3ab9270e84f051d9f9f0 vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz-- 1.00g tp_f1b6384cf8b3c5f0c40310cef0480f32 1.39 brick_d0d9d96b76e89ef597eb1e0015bb3fc1 vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz-- 10.00g tp_d0d9d96b76e89ef597eb1e0015bb3fc1 0.17 tp_2534e2d7f5825eefa5e719cbfad7a0ef vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz-- 100.00g 0.06 3.16 tp_d0d9d96b76e89ef597eb1e0015bb3fc1 vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz-- 10.00g 0.17 10.09 tp_f1b6384cf8b3c5f0c40310cef0480f32 vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz-- 1.00g 1.39 10.45 -----------------------------------------------------------------------------
Retried the same with