Bug 1658166 - Block hosting volume and block volume creation failing in Independent mode setup with brownfield deployment
Summary: Block hosting volume and block volume creation failing in Independent mode se...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: ocs-3.11
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: John Mulligan
QA Contact: Prasanth
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-11 12:30 UTC by Sri Vignesh Selvan
Modified: 2019-06-17 19:08 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-12 21:05:23 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Sri Vignesh Selvan 2018-12-11 12:30:26 UTC
Description of problem:
=======================
Created setup with OCP 3.11.1 + OCS 3.11(RHGS BU2) in Independent mode. Blockd services are initially inactive. after deploying OCP and OCS, started blockd services. Observed that block hosting volume are not getting created. topology info shows size of 100Gi has been assigned to bricks, I see no block hosting volumes from the list. Later tried creating block volumes manually through heketi-cli. where the block volumes are failed to create saying "Error: Server busy. Retry operation later". Later Created block hosting volume manually through heketi-cli, then block hosting volume got created. But the block volumes creation errors out.

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-api-3.12.2-31.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-31.el7rhgs.x86_64
glusterfs-3.12.2-31.el7rhgs.x86_64
glusterfs-cli-3.12.2-31.el7rhgs.x86_64
glusterfs-rdma-3.12.2-31.el7rhgs.x86_64
gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
python2-gluster-3.12.2-31.el7rhgs.x86_64
glusterfs-server-3.12.2-31.el7rhgs.x86_64
gluster-block-0.2.1-28.el7rhgs.x86_64
glusterfs-client-xlators-3.12.2-31.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
vdsm-gluster-4.19.43-2.3.el7rhgs.noarch
glusterfs-libs-3.12.2-31.el7rhgs.x86_64
glusterfs-fuse-3.12.2-31.el7rhgs.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.3.x86_64
libblockdev-2.18-3.el7.x86_64
libblockdev-utils-2.18-3.el7.x86_64
gluster-block-0.2.1-28.el7rhgs.x86_64

How reproducible:
=================
1/1

Steps to Reproduce:
===================
1. Setup with OCP 3.11.1 + OCS 3.11(RHGS BU2) in Independent mode
2. Blockd service was initially inactive.
3. After deploying OCP and OCS, start block services from gluster nodes.
4. Check volume list throught heketi

Actual results:
===============
Block hosting volume is in pending state and block volumes are not getting created.

Expected results:
=================
Block hosting volumes and block volumes should be created.

Additional info:
================

[root@dhcp47-97 ~]# systemctl status gluster-blockd
● gluster-blockd.service - Gluster block storage utility
   Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@dhcp47-97 ~]# systemctl start gluster-blockd
[root@dhcp47-97 ~]# systemctl status gluster-blockd
● gluster-blockd.service - Gluster block storage utility
   Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-12-11 12:17:17 IST; 14s ago
 Main PID: 4020 (gluster-blockd)
    Tasks: 4
   CGroup: /system.slice/gluster-blockd.service
           └─4020 /usr/sbin/gluster-bloc...

Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ...
Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ...
Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ...
Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ...
Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ...
Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com gluster-blockd[4020]: ...
Hint: Some lines were ellipsized, use -l to show in full.
[root@dhcp47-97 ~]# gluster v info
 
Volume Name: heketidbstorage
Type: Replicate
Volume ID: ed7f7031-23cd-412e-9861-116a55293d2f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.46.70:/var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick
Brick2: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick
Brick3: 10.70.46.152:/var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick
Options Reconfigured:
server.tcp-user-timeout: 42
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
 
Volume Name: ms-file
Type: Replicate
Volume ID: 075ef8fa-6ebf-431c-aee5-e876d2358114
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick
Brick2: 10.70.46.152:/var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick
Brick3: 10.70.46.70:/var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick
Options Reconfigured:
server.tcp-user-timeout: 42
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
 
Volume Name: vol_b896ada19d8fee0f3009cb9deb861bdb
Type: Replicate
Volume ID: 272e2b30-8fb5-4e06-97c9-51ba952f84c4
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.46.152:/var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick
Brick2: 10.70.46.70:/var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick
Brick3: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick
Options Reconfigured:
server.tcp-user-timeout: 42
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
 
Volume Name: vol_c1e42a8cec401f067c9a387275a112de
Type: Replicate
Volume ID: 3143c004-1836-4cc6-a283-efe774b954f6
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.46.70:/var/lib/heketi/mounts/vg_416c1e0a6d83780628cb46ac67c58ad1/brick_ad889820a490e42021ab3da578cbcc30/brick
Brick2: 10.70.47.97:/var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_c44e5da892a30260febc143e4ab58789/brick
Brick3: 10.70.46.152:/var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_2534e2d7f5825eefa5e719cbfad7a0ef/brick
Options Reconfigured:
cluster.choose-local: off
server.allow-insecure: on
user.cifs: off
features.shard-block-size: 64MB
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: disable
performance.strict-o-direct: on
performance.readdir-ahead: off
performance.open-behind: off
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
server.tcp-user-timeout: 42
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

[root@dhcp47-97 ~]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-12-10 18:38:01 IST; 21h ago
  Process: 7246 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 7269 (glusterd)
    Tasks: 96
   CGroup: /system.slice/glusterd.service
           ├─3322 /usr/sbin/glusterfsd -...
           ├─3615 /usr/sbin/glusterfsd -...
           ├─4501 /usr/sbin/glusterfsd -...
           ├─4522 /usr/sbin/glusterfs -s...
           ├─7269 /usr/sbin/glusterd -p ...
           └─9577 /usr/sbin/glusterfsd -...

Dec 10 18:37:58 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ...
Dec 10 18:38:01 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ...
Hint: Some lines were ellipsized, use -l to show in full.
[root@dhcp47-97 ~]# systemctl status gluster-block-targetd
Unit gluster-block-targetd.service could not be found.
[root@dhcp47-97 ~]# systemctl status gluster-block-target
● gluster-block-target.service - Restore LIO kernel target configuration
   Loaded: loaded (/usr/lib/systemd/system/gluster-block-target.service; disabled; vendor preset: disabled)
   Active: active (exited) since Tue 2018-12-11 12:17:17 IST; 4h 13min ago
  Process: 4008 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS)
 Main PID: 4008 (code=exited, status=0/SUCCESS)
    Tasks: 0
   CGroup: /system.slice/gluster-block-target.service

Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ...
Dec 11 12:17:17 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ...
Hint: Some lines were ellipsized, use -l to show in full.
[root@dhcp47-97 ~]# systemctl status tcmu-runner
● tcmu-runner.service - LIO Userspace-passthrough daemon
   Loaded: loaded (/usr/lib/systemd/system/tcmu-runner.service; static; vendor preset: disabled)
   Active: active (running) since Tue 2018-12-11 12:17:16 IST; 4h 13min ago
  Process: 3827 ExecStartPre=/usr/libexec/gluster-block/upgrade_activities.sh (code=exited, status=0/SUCCESS)
 Main PID: 4001 (tcmu-runner)
    Tasks: 5
   CGroup: /system.slice/tcmu-runner.service
           └─4001 /usr/bin/tcmu-runner -...

Dec 11 12:17:14 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ...
Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com upgrade_activities.sh[3827]: ...
Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com tcmu-runner[4001]: ...
Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com tcmu-runner[4001]: ...
Dec 11 12:17:16 dhcp47-97.lab.eng.blr.redhat.com systemd[1]: ...
Hint: Some lines were ellipsized, use -l to show in full.


 heketi-cli blockvolume create --size=1
Error: Server busy. Retry operation later.
[root@dhcp46-236 ~]# heketi-cli server operations info
Operation Counts:
  Total: 1
  In-Flight: 0
  New: 1
  Stale: 0
[root@dhcp46-236 ~]# heketi-cli server operations info
Operation Counts:
  Total: 1
  In-Flight: 0
  New: 1
  Stale: 0

----------------------xx-----------------------------
Heketi logs:
============


[negroni] Started POST /blockvolumes
[heketi] WARNING 2018/12/11 06:47:24 temporarily rejecting block volume request: pending block-hosting-volume found
[heketi] ERROR 2018/12/11 06:47:24 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations
[negroni] Completed 429 Too Many Requests in 4.56255ms

----------------------xx-----------------------------

[root@dhcp46-236 ~]# heketi-cli topology info

Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4

    File:  true
    Block: true

    Volumes:

	Name: ms-file
	Size: 10
	Id: 7bb9330bb39b794ee94bc8e92cdd9b99
	Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4
	Mount: 10.70.46.152:ms-file
	Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97
	Durability Type: replicate
	Replica: 3
	Snapshot: Disabled

		Bricks:
			Id: 71c76198b95e9d84d1feb55c7e0e50c0
			Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick
			Size (GiB): 10
			Node: 284a68e0c3628b8d0c6d684e92fb908c
			Device: aa776359e194b7d4a6c75dfac26a8b9e

			Id: d0d9d96b76e89ef597eb1e0015bb3fc1
			Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick
			Size (GiB): 10
			Node: 080daebb1e6fb61158ad527948470491
			Device: ad6bda58895fa8a2c0148d228a04626c

			Id: f474f535ad3edea0f9514ec13d724ae9
			Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick
			Size (GiB): 10
			Node: fbb42c14f89246d1ec799413ec1be9e9
			Device: 4a1b2772b8a3d2be7b22e878686cfda0


	Name: heketidbstorage
	Size: 2
	Id: 808731c9656b6d2e3817b601259dae41
	Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4
	Mount: 10.70.46.152:heketidbstorage
	Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97
	Durability Type: replicate
	Replica: 3
	Snapshot: Disabled

		Bricks:
			Id: 355c5c27dd914c84c952b685803e3d11
			Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick
			Size (GiB): 2
			Node: 284a68e0c3628b8d0c6d684e92fb908c
			Device: aa776359e194b7d4a6c75dfac26a8b9e

			Id: b476bde69e90f2dc0b4d1259bc5c73a3
			Path: /var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick
			Size (GiB): 2
			Node: 080daebb1e6fb61158ad527948470491
			Device: 8de5c63759427f59adc7e07119fb73f2

			Id: d04ea81ba88123242dbbcd91c146a5ea
			Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick
			Size (GiB): 2
			Node: fbb42c14f89246d1ec799413ec1be9e9
			Device: 4a1b2772b8a3d2be7b22e878686cfda0


	Name: vol_b896ada19d8fee0f3009cb9deb861bdb
	Size: 1
	Id: b896ada19d8fee0f3009cb9deb861bdb
	Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4
	Mount: 10.70.46.152:vol_b896ada19d8fee0f3009cb9deb861bdb
	Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97
	Durability Type: replicate
	Replica: 3
	Snapshot: Disabled

		Bricks:
			Id: 3eb382fa8bab1d1a3dfbf041d680c40a
			Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick
			Size (GiB): 1
			Node: 284a68e0c3628b8d0c6d684e92fb908c
			Device: aa776359e194b7d4a6c75dfac26a8b9e

			Id: a60d8334f78c3ab9270e84f051d9f9f0
			Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick
			Size (GiB): 1
			Node: 080daebb1e6fb61158ad527948470491
			Device: ad6bda58895fa8a2c0148d228a04626c

			Id: f9eab198a86731eb09ff3a3398b7f6a8
			Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick
			Size (GiB): 1
			Node: fbb42c14f89246d1ec799413ec1be9e9
			Device: 4a1b2772b8a3d2be7b22e878686cfda0


    Nodes:

	Node Id: 080daebb1e6fb61158ad527948470491
	State: online
	Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4
	Zone: 3
	Management Hostnames: dhcp46-152.lab.eng.blr.redhat.com
	Storage Hostnames: 10.70.46.152
	Devices:
		Id:8de5c63759427f59adc7e07119fb73f2   Name:/dev/sde            State:online    Size (GiB):2047    Used (GiB):2       Free (GiB):2045    
			Bricks:
				Id:b476bde69e90f2dc0b4d1259bc5c73a3   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_8de5c63759427f59adc7e07119fb73f2/brick_b476bde69e90f2dc0b4d1259bc5c73a3/brick
		Id:ad6bda58895fa8a2c0148d228a04626c   Name:/dev/sdd            State:online    Size (GiB):2047    Used (GiB):111     Free (GiB):1936    
			Bricks:
				Id:2534e2d7f5825eefa5e719cbfad7a0ef   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_2534e2d7f5825eefa5e719cbfad7a0ef/brick
				Id:a60d8334f78c3ab9270e84f051d9f9f0   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_a60d8334f78c3ab9270e84f051d9f9f0/brick
				Id:d0d9d96b76e89ef597eb1e0015bb3fc1   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_ad6bda58895fa8a2c0148d228a04626c/brick_d0d9d96b76e89ef597eb1e0015bb3fc1/brick

	Node Id: 284a68e0c3628b8d0c6d684e92fb908c
	State: online
	Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4
	Zone: 2
	Management Hostnames: dhcp46-70.lab.eng.blr.redhat.com
	Storage Hostnames: 10.70.46.70
	Devices:
		Id:416c1e0a6d83780628cb46ac67c58ad1   Name:/dev/sdd            State:online    Size (GiB):2047    Used (GiB):100     Free (GiB):1947    
			Bricks:
				Id:ad889820a490e42021ab3da578cbcc30   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_416c1e0a6d83780628cb46ac67c58ad1/brick_ad889820a490e42021ab3da578cbcc30/brick
		Id:aa776359e194b7d4a6c75dfac26a8b9e   Name:/dev/sde            State:online    Size (GiB):2047    Used (GiB):13      Free (GiB):2034    
			Bricks:
				Id:355c5c27dd914c84c952b685803e3d11   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_355c5c27dd914c84c952b685803e3d11/brick
				Id:3eb382fa8bab1d1a3dfbf041d680c40a   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_3eb382fa8bab1d1a3dfbf041d680c40a/brick
				Id:71c76198b95e9d84d1feb55c7e0e50c0   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_aa776359e194b7d4a6c75dfac26a8b9e/brick_71c76198b95e9d84d1feb55c7e0e50c0/brick

	Node Id: fbb42c14f89246d1ec799413ec1be9e9
	State: online
	Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4
	Zone: 1
	Management Hostnames: dhcp47-97.lab.eng.blr.redhat.com
	Storage Hostnames: 10.70.47.97
	Devices:
		Id:4a1b2772b8a3d2be7b22e878686cfda0   Name:/dev/sdd            State:online    Size (GiB):2047    Used (GiB):113     Free (GiB):1934    
			Bricks:
				Id:c44e5da892a30260febc143e4ab58789   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_c44e5da892a30260febc143e4ab58789/brick
				Id:d04ea81ba88123242dbbcd91c146a5ea   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_d04ea81ba88123242dbbcd91c146a5ea/brick
				Id:f474f535ad3edea0f9514ec13d724ae9   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f474f535ad3edea0f9514ec13d724ae9/brick
				Id:f9eab198a86731eb09ff3a3398b7f6a8   Size (GiB):1       Path: /var/lib/heketi/mounts/vg_4a1b2772b8a3d2be7b22e878686cfda0/brick_f9eab198a86731eb09ff3a3398b7f6a8/brick
		Id:af665090fe3e5d7a253caa4d5f523da1   Name:/dev/sde            State:online    Size (GiB):2047    Used (GiB):0       Free (GiB):2047    
			Bricks:


Tried manually Creating block hosting volume and block volume:
==============================================================
# heketi-cli volume create --size=100 --block
Name: vol_adf620440109d4758c231d4c5a4dec2b
Size: 100
Volume Id: adf620440109d4758c231d4c5a4dec2b
Cluster Id: 5dc64d3efe0e4d1173acb9539670e6a4
Mount: 10.70.46.152:vol_adf620440109d4758c231d4c5a4dec2b
Mount Options: backup-volfile-servers=10.70.46.70,10.70.47.97
Block: true
Free Size: 98
Reserved Size: 2
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3


# heketi-cli volume list 
Id:7bb9330bb39b794ee94bc8e92cdd9b99    Cluster:5dc64d3efe0e4d1173acb9539670e6a4    Name:ms-file
Id:808731c9656b6d2e3817b601259dae41    Cluster:5dc64d3efe0e4d1173acb9539670e6a4    Name:heketidbstorage
Id:adf620440109d4758c231d4c5a4dec2b    Cluster:5dc64d3efe0e4d1173acb9539670e6a4    Name:vol_adf620440109d4758c231d4c5a4dec2b [block]
Id:b896ada19d8fee0f3009cb9deb861bdb    Cluster:5dc64d3efe0e4d1173acb9539670e6a4    Name:vol_b896ada19d8fee0f3009cb9deb861bdb


# heketi-cli blockvolume create --size=5
Error: server did not provide a message (status 500: Internal Server Error)

Comment 4 Sri Vignesh Selvan 2018-12-12 09:37:34 UTC
Adding output of pvs, vgs and lvs:
==================================


[root@dhcp47-97 ~]# pvs
  PV         VG                                  Fmt  Attr PSize   PFree 
  /dev/sda2  rhel_dhcp47-42                      lvm2 a--  <83.00g     0 
  /dev/sdd   vg_4a1b2772b8a3d2be7b22e878686cfda0 lvm2 a--   <2.00t <1.89t
  /dev/sde   vg_af665090fe3e5d7a253caa4d5f523da1 lvm2 a--   <2.00t <2.00t

[root@dhcp46-70 ~]# pvs
  PV         VG                                  Fmt  Attr PSize   PFree 
  /dev/sda2  rhel_dhcp47-42                      lvm2 a--  <83.00g     0 
  /dev/sdd   vg_416c1e0a6d83780628cb46ac67c58ad1 lvm2 a--   <2.00t  1.90t
  /dev/sde   vg_aa776359e194b7d4a6c75dfac26a8b9e lvm2 a--   <2.00t <1.99t

[root@dhcp46-152 ~]# pvs
  PV         VG                                  Fmt  Attr PSize   PFree 
  /dev/sda2  rhel_dhcp47-42                      lvm2 a--  <83.00g     0 
  /dev/sdd   vg_ad6bda58895fa8a2c0148d228a04626c lvm2 a--   <2.00t  1.89t
  /dev/sde   vg_8de5c63759427f59adc7e07119fb73f2 lvm2 a--   <2.00t <2.00t

--------------------------------------------------------------------------

[root@dhcp47-97 ~]# vgs
  VG                                  #PV #LV #SN Attr   VSize   VFree 
  rhel_dhcp47-42                        1   2   0 wz--n- <83.00g     0 
  vg_4a1b2772b8a3d2be7b22e878686cfda0   1   8   0 wz--n-  <2.00t <1.89t
  vg_af665090fe3e5d7a253caa4d5f523da1   1   0   0 wz--n-  <2.00t <2.00t

[root@dhcp46-70 ~]# vgs
  VG                                  #PV #LV #SN Attr   VSize   VFree 
  rhel_dhcp47-42                        1   2   0 wz--n- <83.00g     0 
  vg_416c1e0a6d83780628cb46ac67c58ad1   1   2   0 wz--n-  <2.00t  1.90t
  vg_aa776359e194b7d4a6c75dfac26a8b9e   1   6   0 wz--n-  <2.00t <1.99t

[root@dhcp46-152 ~]# vgs
  VG                                  #PV #LV #SN Attr   VSize   VFree 
  rhel_dhcp47-42                        1   2   0 wz--n- <83.00g     0 
  vg_8de5c63759427f59adc7e07119fb73f2   1   2   0 wz--n-  <2.00t <2.00t
  vg_ad6bda58895fa8a2c0148d228a04626c   1   6   0 wz--n-  <2.00t  1.89t

--------------------------------------------------------------------------

[root@dhcp47-97 ~]# lvs
  LV                                     VG                                  Attr       LSize   Pool                                Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home                                   rhel_dhcp47-42                      -wi-ao---- <33.00g                                                                                   
  root                                   rhel_dhcp47-42                      -wi-ao----  50.00g                                                                                   
  brick_c44e5da892a30260febc143e4ab58789 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz-- 100.00g tp_c44e5da892a30260febc143e4ab58789        0.06                                   
  brick_d04ea81ba88123242dbbcd91c146a5ea vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz--   2.00g tp_d04ea81ba88123242dbbcd91c146a5ea        0.70                                   
  brick_f474f535ad3edea0f9514ec13d724ae9 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz--  10.00g tp_212277d8ff5c82a9d155fbfe8525d835        0.17                                   
  brick_f9eab198a86731eb09ff3a3398b7f6a8 vg_4a1b2772b8a3d2be7b22e878686cfda0 Vwi-aotz--   1.00g tp_f9eab198a86731eb09ff3a3398b7f6a8        1.39                                   
  tp_212277d8ff5c82a9d155fbfe8525d835    vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz--  10.00g                                            0.17   10.09                           
  tp_c44e5da892a30260febc143e4ab58789    vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz-- 100.00g                                            0.06   3.16                            
  tp_d04ea81ba88123242dbbcd91c146a5ea    vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz--   2.00g                                            0.70   10.32                           
  tp_f9eab198a86731eb09ff3a3398b7f6a8    vg_4a1b2772b8a3d2be7b22e878686cfda0 twi-aotz--   1.00g                                            1.39   10.45 

[root@dhcp46-70 ~]# lvs
  LV                                     VG                                  Attr       LSize   Pool                                Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home                                   rhel_dhcp47-42                      -wi-ao---- <33.00g                                                                                   
  root                                   rhel_dhcp47-42                      -wi-ao----  50.00g                                                                                   
  brick_ad889820a490e42021ab3da578cbcc30 vg_416c1e0a6d83780628cb46ac67c58ad1 Vwi-aotz-- 100.00g tp_255db5b8b9bb07472e83b5418f0c0984        0.06                                   
  tp_255db5b8b9bb07472e83b5418f0c0984    vg_416c1e0a6d83780628cb46ac67c58ad1 twi-aotz-- 100.00g                                            0.06   3.16                            
  brick_355c5c27dd914c84c952b685803e3d11 vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz--   2.00g tp_741b50fd7998673649a284cdb31a1246        0.70                                   
  brick_3eb382fa8bab1d1a3dfbf041d680c40a vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz--   1.00g tp_3eb382fa8bab1d1a3dfbf041d680c40a        1.39                                   
  brick_71c76198b95e9d84d1feb55c7e0e50c0 vg_aa776359e194b7d4a6c75dfac26a8b9e Vwi-aotz--  10.00g tp_71c76198b95e9d84d1feb55c7e0e50c0        0.17                                   
  tp_3eb382fa8bab1d1a3dfbf041d680c40a    vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz--   1.00g                                            1.39   10.45                           
  tp_71c76198b95e9d84d1feb55c7e0e50c0    vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz--  10.00g                                            0.17   10.09                           
  tp_741b50fd7998673649a284cdb31a1246    vg_aa776359e194b7d4a6c75dfac26a8b9e twi-aotz--   2.00g                                            0.70   10.32   

[root@dhcp46-152 ~]# lvs
  LV                                     VG                                  Attr       LSize   Pool                                Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home                                   rhel_dhcp47-42                      -wi-ao---- <33.00g                                                                                   
  root                                   rhel_dhcp47-42                      -wi-ao----  50.00g                                                                                   
  brick_b476bde69e90f2dc0b4d1259bc5c73a3 vg_8de5c63759427f59adc7e07119fb73f2 Vwi-aotz--   2.00g tp_b476bde69e90f2dc0b4d1259bc5c73a3        0.70                                   
  tp_b476bde69e90f2dc0b4d1259bc5c73a3    vg_8de5c63759427f59adc7e07119fb73f2 twi-aotz--   2.00g                                            0.70   10.32                           
  brick_2534e2d7f5825eefa5e719cbfad7a0ef vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz-- 100.00g tp_2534e2d7f5825eefa5e719cbfad7a0ef        0.06                                   
  brick_a60d8334f78c3ab9270e84f051d9f9f0 vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz--   1.00g tp_f1b6384cf8b3c5f0c40310cef0480f32        1.39                                   
  brick_d0d9d96b76e89ef597eb1e0015bb3fc1 vg_ad6bda58895fa8a2c0148d228a04626c Vwi-aotz--  10.00g tp_d0d9d96b76e89ef597eb1e0015bb3fc1        0.17                                   
  tp_2534e2d7f5825eefa5e719cbfad7a0ef    vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz-- 100.00g                                            0.06   3.16                            
  tp_d0d9d96b76e89ef597eb1e0015bb3fc1    vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz--  10.00g                                            0.17   10.09                           
  tp_f1b6384cf8b3c5f0c40310cef0480f32    vg_ad6bda58895fa8a2c0148d228a04626c twi-aotz--   1.00g                                            1.39   10.45  

-----------------------------------------------------------------------------

Comment 5 Sri Vignesh Selvan 2018-12-13 05:59:45 UTC
Retried the same with


Note You need to log in before you can comment on or make changes to this bug.