1435530 – [device remove]: heketi volume info doesn't reflect new volfile server when device remove has replaced a device from a new node

Bug 1435530 - [device remove]: heketi volume info doesn't reflect new volfile server when device remove has replaced a device from a new node

Summary: [device remove]: heketi volume info doesn't reflect new volfile server when d...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	heketi
Sub Component:
Version:	cns-3.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	CNS 3.6
Assignee:	Mohamed Ashiq
QA Contact:	krishnaram Karthick
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1445447 1468091
TreeView+	depends on / blocked

Reported:	2017-03-24 06:05 UTC by krishnaram Karthick
Modified:	2017-10-26 07:58 UTC (History)
CC List:	12 users (show)
Fixed In Version:	heketi-5.0.0-8 rhgs-volmanager-docker-5.0.0-10
Doc Type:	If docs needed, set a value
Doc Text:	Previously, the volfile server list would contain just gluster pod IPs which host one or more bricks for a particular volume. With this update, all the servers present in a trusted pool are added to the volfile server of the volume.
Clone Of:
Environment:
Last Closed:	2017-10-11 07:07:22 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2017:2879	0	normal	SHIPPED_LIVE	heketi bug fix and enhancement update	2017-10-11 11:07:06 UTC

Description krishnaram Karthick 2017-03-24 06:05:29 UTC

Description of problem:

When device remove command is run on a device and if the replaced device is coming from a different node other than that of the existing three nodes, volfile servers list shown in volume info doesn't reflect the updated servers.

For ex, if node{1,2,3} are the servers from which a volume is created, when remove device is run on say node 1 device 1, it is possible that the new device comes from node 4 device 1. Now volume info should show node{2,3,4}.  

[root@dhcp46-202 ~]# heketi-cli volume info c3d14cd23a44241701bbff20ac112e37
Name: vol_c3d14cd23a44241701bbff20ac112e37
Size: 9
Volume Id: c3d14cd23a44241701bbff20ac112e37
Cluster Id: ee0be395eee24de0af625fb70b598342
Mount: 10.70.46.165:vol_c3d14cd23a44241701bbff20ac112e37       <---------
Mount Options: backup-volfile-servers=10.70.47.180,10.70.47.51 <---------
Durability Type: replicate
Distributed+Replica: 3
[root@dhcp46-202 ~]# 
[root@dhcp46-202 ~]# 
[root@dhcp46-202 ~]# 
[root@dhcp46-202 ~]# 
[root@dhcp46-202 ~]# 
[root@dhcp46-202 ~]# 
[root@dhcp46-202 ~]# heketi-cli node list
Id:0caf00da1c9dd2cfa275589eee5a3e2c	Cluster:ee0be395eee24de0af625fb70b598342
Id:1bf58eba8401828a90223c45f753b607	Cluster:ee0be395eee24de0af625fb70b598342
Id:21438725a596e7a26203244a73c93e41	Cluster:ee0be395eee24de0af625fb70b598342
Id:76c04cd33916422802b3d14e6088ef75	Cluster:ee0be395eee24de0af625fb70b598342
Id:b5bb6e7ca6a8b74e2ccf776b79d121a8	Cluster:ee0be395eee24de0af625fb70b598342
[root@dhcp46-202 ~]# heketi-cli node info 1bf58eba8401828a90223c45f753b607
Node Id: 1bf58eba8401828a90223c45f753b607
State: online
Cluster Id: ee0be395eee24de0af625fb70b598342
Zone: 2
Management Hostname: dhcp46-165.lab.eng.blr.redhat.com
Storage Hostname: 10.70.46.165
Devices:
Id:270598b26db357c991bc2f820ec2da53   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:8b3ab766c0146956f00d5d928c60fa50   Name:/dev/sdg            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:99e64bf21f879e76dabc80ac250bbd90   Name:/dev/sdi            State:offline   Size (GiB):99      Used (GiB):8       Free (GiB):91      
Id:bebf6c7f3788dde484356e3d3600b776   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:e2207d839a809a6906c3d01ac17b8334   Name:/dev/sdf            State:offline   Size (GiB):99      Used (GiB):3       Free (GiB):96      
Id:f0565469addfecd5ec737e86bdda2950   Name:/dev/sdh            State:online    Size (GiB):99      Used (GiB):9       Free (GiB):90      
[root@dhcp46-202 ~]# heketi-cli device disable f0565469addfecd5ec737e86bdda2950
Device f0565469addfecd5ec737e86bdda2950 is now offline
[root@dhcp46-202 ~]# heketi-cli device remove f0565469addfecd5ec737e86bdda2950
Device f0565469addfecd5ec737e86bdda2950 is now removed
[root@dhcp46-202 ~]# heketi-cli volume info c3d14cd23a44241701bbff20ac112e37
Name: vol_c3d14cd23a44241701bbff20ac112e37
Size: 9
Volume Id: c3d14cd23a44241701bbff20ac112e37
Cluster Id: ee0be395eee24de0af625fb70b598342
Mount: 10.70.46.165:vol_c3d14cd23a44241701bbff20ac112e37        <---------
Mount Options: backup-volfile-servers=10.70.47.180,10.70.47.51  <---------
Durability Type: replicate
Distributed+Replica: 3

Device is replaced from a different node -> dhcp47-180.lab.eng.blr.redhat.com


[root@dhcp46-202 ~]# heketi-cli node info 1bf58eba8401828a90223c45f753b607
Node Id: 1bf58eba8401828a90223c45f753b607
State: online
Cluster Id: ee0be395eee24de0af625fb70b598342
Zone: 2
Management Hostname: dhcp46-165.lab.eng.blr.redhat.com
Storage Hostname: 10.70.46.165
Devices:
Id:270598b26db357c991bc2f820ec2da53   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:8b3ab766c0146956f00d5d928c60fa50   Name:/dev/sdg            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:99e64bf21f879e76dabc80ac250bbd90   Name:/dev/sdi            State:offline   Size (GiB):99      Used (GiB):8       Free (GiB):91      
Id:bebf6c7f3788dde484356e3d3600b776   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:e2207d839a809a6906c3d01ac17b8334   Name:/dev/sdf            State:offline   Size (GiB):99      Used (GiB):3       Free (GiB):96      
Id:f0565469addfecd5ec737e86bdda2950   Name:/dev/sdh            State:failed    Size (GiB):99      Used (GiB):0       Free (GiB):99      

[root@dhcp46-202 ~]# heketi-cli node info 0caf00da1c9dd2cfa275589eee5a3e2c
Node Id: 0caf00da1c9dd2cfa275589eee5a3e2c
State: online
Cluster Id: ee0be395eee24de0af625fb70b598342
Zone: 1
Management Hostname: dhcp47-180.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.180
Devices:
Id:47670cd38fad5d4211b162c480e6e62c   Name:/dev/sdd            State:online    Size (GiB):199     Used (GiB):9       Free (GiB):190     


Version-Release number of selected component (if applicable):
heketi-client-4.0.0-3.el7rhgs.x86_64

How reproducible:
Always

Comment 2 Humble Chirammal 2017-03-24 06:14:41 UTC

Backup volfile servers shouldnt be brick peers, it is wrong. It should be TSP peers. We need to correct this and it will eventually fix this issue.

Comment 6 Mohamed Ashiq 2017-08-03 10:05:38 UTC

Upstream PR:

https://github.com/heketi/heketi/pull/826

Comment 7 Mohamed Ashiq 2017-08-16 07:00:20 UTC

(In reply to Mohamed Ashiq from comment #6)
> Upstream PR:
> 
> https://github.com/heketi/heketi/pull/826

Merged upstream

Comment 8 Apeksha 2017-09-18 13:30:41 UTC

[root@dhcp46-156 ~]# heketi-cli node info 5d41d4b57f8383c37c0346d8ce367b30
Node Id: 5d41d4b57f8383c37c0346d8ce367b30
State: online
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Zone: 1
Management Hostname: dhcp47-184.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.184
Devices:
Id:880250b86d7f0210fff9d86c8910141d   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):50      Free (GiB):49      
Id:bfa7fc86dbf251205b415e883a099081   Name:/dev/sdf            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:f7128c706fb6df5eadd0fd0bcf9ff486   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):22      Free (GiB):77      
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# heketi-cli device disable f7128c706fb6df5eadd0fd0bcf9ff486
Device f7128c706fb6df5eadd0fd0bcf9ff486 is now offline
[root@dhcp46-156 ~]# heketi-cli device remove f7128c706fb6df5eadd0fd0bcf9ff486
Device f7128c706fb6df5eadd0fd0bcf9ff486 is now removed
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# heketi-cli node info 5d41d4b57f8383c37c0346d8ce367b30
Node Id: 5d41d4b57f8383c37c0346d8ce367b30
State: online
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Zone: 1
Management Hostname: dhcp47-184.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.184
Devices:
Id:880250b86d7f0210fff9d86c8910141d   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):50      Free (GiB):49      
Id:bfa7fc86dbf251205b415e883a099081   Name:/dev/sdf            State:online    Size (GiB):99      Used (GiB):20      Free (GiB):79      
Id:f7128c706fb6df5eadd0fd0bcf9ff486   Name:/dev/sdd            State:failed    Size (GiB):99      Used (GiB):0       Free (GiB):99      
[root@dhcp46-156 ~]# heketi-cli node info ^C
[root@dhcp46-156 ~]# heketi-cli node info 775bac341eb672f0afcb075b72e7f41c
Node Id: 775bac341eb672f0afcb075b72e7f41c
State: online
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Zone: 1
Management Hostname: dhcp47-127.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.127
Devices:
Id:2c6bfd73168a368f08a06dfa34d2c0af   Name:/dev/sdf            State:online    Size (GiB):99      Used (GiB):2       Free (GiB):97      
Id:77bd6a1bc66b7b888457d41f6462f8e0   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:92dc7c78f27bd974e68af8bca1a502dc   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# heketi-cli volume info fa783d403a6def7a9c8dfd647079793a
Name: vol_fa783d403a6def7a9c8dfd647079793a
Size: 20
Volume Id: fa783d403a6def7a9c8dfd647079793a
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Mount: 10.70.46.14:vol_fa783d403a6def7a9c8dfd647079793a
Mount Options: backup-volfile-servers=10.70.46.223,10.70.47.184
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
[root@dhcp46-156 ~]# heketi-cli device disable 880250b86d7f0210fff9d86c8910141d
Device 880250b86d7f0210fff9d86c8910141d is now offline
[root@dhcp46-156 ~]# heketi-cli device remove 880250b86d7f0210fff9d86c8910141d
Device 880250b86d7f0210fff9d86c8910141d is now removed
[root@dhcp46-156 ~]# heketi-cli node info 775bac341eb672f0afcb075b72e7f41c
Node Id: 775bac341eb672f0afcb075b72e7f41c
State: online
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Zone: 1
Management Hostname: dhcp47-127.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.127
Devices:
Id:2c6bfd73168a368f08a06dfa34d2c0af   Name:/dev/sdf            State:online    Size (GiB):99      Used (GiB):52      Free (GiB):47      
Id:77bd6a1bc66b7b888457d41f6462f8e0   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:92dc7c78f27bd974e68af8bca1a502dc   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# heketi-cli volume info fa783d403a6def7a9c8dfd647079793a
Name: vol_fa783d403a6def7a9c8dfd647079793a
Size: 20
Volume Id: fa783d403a6def7a9c8dfd647079793a
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Mount: 10.70.46.14:vol_fa783d403a6def7a9c8dfd647079793a
Mount Options: backup-volfile-servers=10.70.46.223,10.70.47.184
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# heketi-cli node info 5d41d4b57f8383c37c0346d8ce367b30
Node Id: 5d41d4b57f8383c37c0346d8ce367b30
State: online
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Zone: 1
Management Hostname: dhcp47-184.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.184
Devices:
Id:880250b86d7f0210fff9d86c8910141d   Name:/dev/sde            State:failed    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:bfa7fc86dbf251205b415e883a099081   Name:/dev/sdf            State:online    Size (GiB):99      Used (GiB):20      Free (GiB):79      
Id:f7128c706fb6df5eadd0fd0bcf9ff486   Name:/dev/sdd            State:failed    Size (GiB):99      Used (GiB):0       Free (GiB):99      
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# 
[root@dhcp46-156 ~]# heketi-cli volume info c0f65102c58470e170e554b1a242121e
Name: vol_c0f65102c58470e170e554b1a242121e
Size: 50
Volume Id: c0f65102c58470e170e554b1a242121e
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Mount: 10.70.46.14:vol_c0f65102c58470e170e554b1a242121e
Mount Options: backup-volfile-servers=10.70.46.223,10.70.47.184
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

Comment 9 Michael Adam 2017-09-19 08:57:26 UTC

(In reply to Apeksha from comment #8)
> [root@dhcp46-156 ~]# heketi-cli node info 5d41d4b57f8383c37c0346d8ce367b30
> Node Id: 5d41d4b57f8383c37c0346d8ce367b30
> State: online
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Zone: 1
> Management Hostname: dhcp47-184.lab.eng.blr.redhat.com
> Storage Hostname: 10.70.47.184
> Devices:
> Id:880250b86d7f0210fff9d86c8910141d   Name:/dev/sde            State:online 
> Size (GiB):99      Used (GiB):50      Free (GiB):49      
> Id:bfa7fc86dbf251205b415e883a099081   Name:/dev/sdf            State:online 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> Id:f7128c706fb6df5eadd0fd0bcf9ff486   Name:/dev/sdd            State:online 
> Size (GiB):99      Used (GiB):22      Free (GiB):77      
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# heketi-cli device disable
> f7128c706fb6df5eadd0fd0bcf9ff486
> Device f7128c706fb6df5eadd0fd0bcf9ff486 is now offline
> [root@dhcp46-156 ~]# heketi-cli device remove
> f7128c706fb6df5eadd0fd0bcf9ff486
> Device f7128c706fb6df5eadd0fd0bcf9ff486 is now removed
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# heketi-cli node info 5d41d4b57f8383c37c0346d8ce367b30
> Node Id: 5d41d4b57f8383c37c0346d8ce367b30
> State: online
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Zone: 1
> Management Hostname: dhcp47-184.lab.eng.blr.redhat.com
> Storage Hostname: 10.70.47.184
> Devices:
> Id:880250b86d7f0210fff9d86c8910141d   Name:/dev/sde            State:online 
> Size (GiB):99      Used (GiB):50      Free (GiB):49      
> Id:bfa7fc86dbf251205b415e883a099081   Name:/dev/sdf            State:online 
> Size (GiB):99      Used (GiB):20      Free (GiB):79      
> Id:f7128c706fb6df5eadd0fd0bcf9ff486   Name:/dev/sdd            State:failed 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> [root@dhcp46-156 ~]# heketi-cli node info ^C
> [root@dhcp46-156 ~]# heketi-cli node info 775bac341eb672f0afcb075b72e7f41c
> Node Id: 775bac341eb672f0afcb075b72e7f41c
> State: online
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Zone: 1
> Management Hostname: dhcp47-127.lab.eng.blr.redhat.com
> Storage Hostname: 10.70.47.127
> Devices:
> Id:2c6bfd73168a368f08a06dfa34d2c0af   Name:/dev/sdf            State:online 
> Size (GiB):99      Used (GiB):2       Free (GiB):97      
> Id:77bd6a1bc66b7b888457d41f6462f8e0   Name:/dev/sdd            State:online 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> Id:92dc7c78f27bd974e68af8bca1a502dc   Name:/dev/sde            State:online 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# heketi-cli volume info fa783d403a6def7a9c8dfd647079793a
> Name: vol_fa783d403a6def7a9c8dfd647079793a
> Size: 20
> Volume Id: fa783d403a6def7a9c8dfd647079793a
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Mount: 10.70.46.14:vol_fa783d403a6def7a9c8dfd647079793a
> Mount Options: backup-volfile-servers=10.70.46.223,10.70.47.184
> Block: false
> Free Size: 0
> Block Volumes: []
> Durability Type: replicate
> Distributed+Replica: 3
> [root@dhcp46-156 ~]# heketi-cli device disable
> 880250b86d7f0210fff9d86c8910141d
> Device 880250b86d7f0210fff9d86c8910141d is now offline
> [root@dhcp46-156 ~]# heketi-cli device remove
> 880250b86d7f0210fff9d86c8910141d
> Device 880250b86d7f0210fff9d86c8910141d is now removed
> [root@dhcp46-156 ~]# heketi-cli node info 775bac341eb672f0afcb075b72e7f41c
> Node Id: 775bac341eb672f0afcb075b72e7f41c
> State: online
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Zone: 1
> Management Hostname: dhcp47-127.lab.eng.blr.redhat.com
> Storage Hostname: 10.70.47.127
> Devices:
> Id:2c6bfd73168a368f08a06dfa34d2c0af   Name:/dev/sdf            State:online 
> Size (GiB):99      Used (GiB):52      Free (GiB):47      
> Id:77bd6a1bc66b7b888457d41f6462f8e0   Name:/dev/sdd            State:online 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> Id:92dc7c78f27bd974e68af8bca1a502dc   Name:/dev/sde            State:online 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# heketi-cli volume info fa783d403a6def7a9c8dfd647079793a
> Name: vol_fa783d403a6def7a9c8dfd647079793a
> Size: 20
> Volume Id: fa783d403a6def7a9c8dfd647079793a
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Mount: 10.70.46.14:vol_fa783d403a6def7a9c8dfd647079793a
> Mount Options: backup-volfile-servers=10.70.46.223,10.70.47.184
> Block: false
> Free Size: 0
> Block Volumes: []
> Durability Type: replicate
> Distributed+Replica: 3
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# heketi-cli node info 5d41d4b57f8383c37c0346d8ce367b30
> Node Id: 5d41d4b57f8383c37c0346d8ce367b30
> State: online
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Zone: 1
> Management Hostname: dhcp47-184.lab.eng.blr.redhat.com
> Storage Hostname: 10.70.47.184
> Devices:
> Id:880250b86d7f0210fff9d86c8910141d   Name:/dev/sde            State:failed 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> Id:bfa7fc86dbf251205b415e883a099081   Name:/dev/sdf            State:online 
> Size (GiB):99      Used (GiB):20      Free (GiB):79      
> Id:f7128c706fb6df5eadd0fd0bcf9ff486   Name:/dev/sdd            State:failed 
> Size (GiB):99      Used (GiB):0       Free (GiB):99      
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# 
> [root@dhcp46-156 ~]# heketi-cli volume info c0f65102c58470e170e554b1a242121e
> Name: vol_c0f65102c58470e170e554b1a242121e
> Size: 50
> Volume Id: c0f65102c58470e170e554b1a242121e
> Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
> Mount: 10.70.46.14:vol_c0f65102c58470e170e554b1a242121e
> Mount Options: backup-volfile-servers=10.70.46.223,10.70.47.184
> Block: false
> Free Size: 0
> Block Volumes: []
> Durability Type: replicate
> Distributed+Replica: 3


This is just an output of 100+ lines.
Without explanation what is wrong.

From looking at the output, I think this works as designed now.
We now always have all online gluster nodes as volfile
servers, which is exactly how it should be.

Please explain why this justifies putting on failed-qa.

Comment 10 Apeksha 2017-09-19 09:12:51 UTC

1. removed 2 devices from node 10.70.47.184:

[root@dhcp46-156 ~]# heketi-cli node info 5d41d4b57f8383c37c0346d8ce367b30
Node Id: 5d41d4b57f8383c37c0346d8ce367b30
State: online
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Zone: 1
Management Hostname: dhcp47-184.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.184
Devices:
Id:880250b86d7f0210fff9d86c8910141d   Name:/dev/sde        State:failed  <--- Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:bfa7fc86dbf251205b415e883a099081   Name:/dev/sdf        State:online    Size (GiB):99      Used (GiB):20      Free (GiB):79      
Id:f7128c706fb6df5eadd0fd0bcf9ff486   Name:/dev/sdd        State:failed  <---  Size (GiB):99      Used (GiB):0       Free (GiB):99 

2. device got replaced from 10.70.47.127

[root@dhcp46-156 ~]# heketi-cli node info 775bac341eb672f0afcb075b72e7f41c
Node Id: 775bac341eb672f0afcb075b72e7f41c
State: online
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Zone: 1
Management Hostname: dhcp47-127.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.127
Devices:
Id:2c6bfd73168a368f08a06dfa34d2c0af   Name:/dev/sdf            State:online    Size (GiB):99      Used (GiB):52      Free (GiB):47      
Id:77bd6a1bc66b7b888457d41f6462f8e0   Name:/dev/sdd            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99      
Id:92dc7c78f27bd974e68af8bca1a502dc   Name:/dev/sde            State:online    Size (GiB):99      Used (GiB):0       Free (GiB):99  


3. still the backup-volfile server points to 10.70.47.184 and not 47.127
[root@dhcp46-156 ~]# heketi-cli volume info c0f65102c58470e170e554b1a242121e
Name: vol_c0f65102c58470e170e554b1a242121e
Size: 50
Volume Id: c0f65102c58470e170e554b1a242121e
Cluster Id: dbdaa75a2da75a7bd6a5fe368725d92c
Mount: 10.70.46.14:vol_c0f65102c58470e170e554b1a242121e
Mount Options: backup-volfile-servers=10.70.46.223,10.70.47.184
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

4. FYI gluster vol info of that volume shows brick om 47.127
Volume Name: vol_c0f65102c58470e170e554b1a242121e
Type: Replicate
Volume ID: b2c95422-4c43-4848-bf18-b2e3ea3b8363
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.47.127:/var/lib/heketi/mounts/vg_2c6bfd73168a368f08a06dfa34d2c0af/brick_f755a60c008b1f0b7a7a0818f6447399/brick
Brick2: 10.70.46.223:/var/lib/heketi/mounts/vg_81dcc5577dcf0147315ed54c847ceb5d/brick_7c877ec1a4def1d1cde28812bf9fe41f/brick
Brick3: 10.70.46.14:/var/lib/heketi/mounts/vg_066d6d9f855bd9d14776cf898265eaf0/brick_2c2eee4318a2b9fc132a36768cf46a1d/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: on

Comment 11 krishnaram Karthick 2017-09-19 09:34:16 UTC

The bug was raised with the intention that backup volfile servers should be updated appropriately when a device remove is done, which doesn't seem to happen as per comment 10. 

Also per comment # 2, replacing brick peers with TSP peers should eventually solve the issue reported. This doesn't seem to happen and hence FailedQA

Comment 12 Michael Adam 2017-09-19 10:05:43 UTC

(In reply to krishnaram Karthick from comment #11)
> The bug was raised with the intention that backup volfile servers should be
> updated appropriately when a device remove is done, which doesn't seem to
> happen as per comment 10. 
> 
> Also per comment # 2, replacing brick peers with TSP peers should eventually
> solve the issue reported. This doesn't seem to happen and hence FailedQA

This is a misunderstanding:

The original misbehavior was that only the nodes were listed as volfile servers that were hosting bricks. This was wrong.
As of newly, the (correct!) behavior is that all TSP nodes are volfile servers. Hence they also do not need to change on device remove (only on node remove).

Note that the volfile server is used for contacting glusterd.
And glusterd is running on all nodes irrespective of it hosting bricks for the current volume.

==> This needs to be moved to verified.

Comment 13 Humble Chirammal 2017-09-19 10:50:42 UTC

Based on c#12, I am moving this back to ON_QA.

Comment 14 krishnaram Karthick 2017-09-19 11:22:11 UTC

Discussed with Michael & Humble and this is what the fix does. Instead of looking at the peer bricks, backup-volfile-servers is updated with the peers of TSP, which ensures all the nodes of TSP is updated as part of volfile servers.

This fix is available in build - cns-deploy-5.0.0-43.el7rhgs.x86_64. 

[root@dhcp46-207 ~]# heketi-cli volume info bbbc1dfaec2d201ed3790a8dc5047a99
Name: vol_bbbc1dfaec2d201ed3790a8dc5047a99
Size: 5
Volume Id: bbbc1dfaec2d201ed3790a8dc5047a99
Cluster Id: 97c16496e5d4c13232deb1d4ba95d557
Mount: 10.70.46.193:vol_bbbc1dfaec2d201ed3790a8dc5047a99
Mount Options: backup-volfile-servers=10.70.46.197,10.70.46.201,10.70.46.203
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

Although fix is made available, there are corner cases where the backupvolfile servers aren't updated.

1) When a TSP is expanded
2) when TSP is shrunk
3) when a volume from previous build is upgraded to the build with this fix

I'll be raising separate bugs for each of this case. But this bug for now will be moved to verified.

Comment 15 Raghavendra Talur 2017-09-26 07:40:48 UTC

Provided doc text, please review.

Comment 17 Raghavendra Talur 2017-10-04 08:50:32 UTC

doc text looks good to me

Comment 19 errata-xmlrpc 2017-10-11 07:07:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2879

Note You need to log in before you can comment on or make changes to this bug.