Bug 1349875 - [ RFE ] heketi-cli should support replacement of a failed node
Summary: [ RFE ] heketi-cli should support replacement of a failed node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: CNS 3.6
Assignee: Mohamed Ashiq
QA Contact: Tejas Chaphekar
URL:
Whiteboard:
: 1446065 (view as bug list)
Depends On:
Blocks: 1358188 1445444
TreeView+ depends on / blocked
 
Reported: 2016-06-24 12:22 UTC by Prasanth
Modified: 2017-10-11 07:07 UTC (History)
10 users (show)

Fixed In Version: heketi-5.0.0-1.el7rhgs
Doc Type: Enhancement
Doc Text:
Previously, the prescribed way to replace a failed node was to run multiple commands on the node devices followed by the node delete command. With this update, the removal of a failed node has been enhanced with a single command to replace a failed node. For example: heketi-cli node remove [node-id]
Clone Of:
: 1358188 (view as bug list)
Environment:
Last Closed: 2017-10-11 07:07:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:2879 normal SHIPPED_LIVE heketi bug fix and enhancement update 2017-10-11 11:07:06 UTC
Github heketi heketi issues 161 None None None 2016-09-30 11:21:03 UTC

Description Prasanth 2016-06-24 12:22:53 UTC
Description of problem:

heketi-cli currently supports only add and delete of a node. It should also support a cleaner way to replace a failed node as well:

#####
# heketi-cli node -h
Heketi Node Management

Usage:
  heketi-cli node [command]

Available Commands:
  add         Add new node to be managed by Heketi
  delete      Deletes a node from Heketi management
  info        Retreives information about the node
  enable      Allows node to go online
  disable     Disallow usage of a node by placing it offline
#####

Version-Release number of selected component (if applicable):
heketi-templates-2.0.2-3.el7rhgs.x86_64
heketi-client-2.0.2-3.el7rhgs.x86_64

Comment 1 Luis Pabón 2016-06-27 12:59:04 UTC
Definitely very important feature.  This is planned for Release 3:

https://github.com/heketi/heketi/issues/161

Comment 2 Luis Pabón 2016-09-30 11:21:04 UTC
Work upstream has not started.  This may or may not make it to 3.4

Comment 3 Michael Adam 2016-10-19 13:54:25 UTC
moving out of 3.4 to 3.5

Comment 10 Humble Chirammal 2017-04-27 11:19:49 UTC
*** Bug 1446065 has been marked as a duplicate of this bug. ***

Comment 12 Mohamed Ashiq 2017-05-24 11:16:55 UTC
Merged Upstream:

https://github.com/heketi/heketi/pull/752

Comment 13 Tejas Chaphekar 2017-08-21 08:19:24 UTC
Multiple features are added as a part of 3.6 release and tested successfully as a part of 3.6 Testing

Plz find the logs as follows

[root@dhcp46-116 ~]# heketi-cli node --help
Heketi Node Management

Usage:
  heketi-cli node [command]

Available Commands:
  add         Add new node to be managed by Heketi
  delete      Deletes a node from Heketi management
  disable     Disallow usage of a node by placing it offline
  enable      Allows node to go online
  info        Retreives information about the node
  list        List all nodes in cluster
  remove      Removes a node and all its associated devices from Heketi

Flags:
  -h, --help   help for node

Global Flags:
      --json            
	Print response as JSON
      --secret string   
	Secret key for specified user.  Can also be
	set using the environment variable HEKETI_CLI_KEY
  -s, --server string   
	Heketi server. Can also be set using the
	environment variable HEKETI_CLI_SERVER (the default one is http://localhost:8080)
      --user string     
	Heketi user.  Can also be set using the
	environment variable HEKETI_CLI_USER



[root@dhcp47-20 ~]# heketi-cli node list

Id:75c151258eee05c64f16dcaa85549f7b	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:836b8213fa88164bc152804ceca0a13a	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:92f58358e92be0f010523f87ca4ba017	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:f7a3fe0efe012afa65250258b709ccbf	Cluster:83f2d91d36c5ea0076e6bd9ed165454a

[root@dhcp47-20 ~]# heketi-cli node info 92f58358e92be0f010523f87ca4ba017

Node Id: 92f58358e92be0f010523f87ca4ba017
State: online
Cluster Id: 83f2d91d36c5ea0076e6bd9ed165454a
Zone: 1
Management Hostname: dhcp47-175.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.175
Devices:
Id:dddb4b7848ba98763b469180fb7d2fc6   Name:/dev/sdd            State:online    Size (GiB):15      Used (GiB):0       Free (GiB):15      

[root@dhcp47-20 ~]# heketi-cli node disable 92f58358e92be0f010523f87ca4ba017

Node 92f58358e92be0f010523f87ca4ba017 is now offline

[root@dhcp47-20 ~]# heketi-cli node list

Id:75c151258eee05c64f16dcaa85549f7b	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:836b8213fa88164bc152804ceca0a13a	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:92f58358e92be0f010523f87ca4ba017	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:f7a3fe0efe012afa65250258b709ccbf	Cluster:83f2d91d36c5ea0076e6bd9ed165454a

[root@dhcp47-20 ~]# heketi-cli node info 92f58358e92be0f010523f87ca4ba017

Node Id: 92f58358e92be0f010523f87ca4ba017
State: offline
Cluster Id: 83f2d91d36c5ea0076e6bd9ed165454a
Zone: 1
Management Hostname: dhcp47-175.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.175
Devices:
Id:dddb4b7848ba98763b469180fb7d2fc6   Name:/dev/sdd            State:online    Size (GiB):15      Used (GiB):0       Free (GiB):15      

[root@dhcp47-20 ~]# heketi-cli node remove 92f58358e92be0f010523f87ca4ba017

Node 92f58358e92be0f010523f87ca4ba017 is now removed

[root@dhcp47-20 ~]# heketi-cli node list

Id:75c151258eee05c64f16dcaa85549f7b	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:836b8213fa88164bc152804ceca0a13a	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:92f58358e92be0f010523f87ca4ba017	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:f7a3fe0efe012afa65250258b709ccbf	Cluster:83f2d91d36c5ea0076e6bd9ed165454a

[root@dhcp47-20 ~]# heketi-cli node info 92f58358e92be0f010523f87ca4ba017

Node Id: 92f58358e92be0f010523f87ca4ba017
State: failed
Cluster Id: 83f2d91d36c5ea0076e6bd9ed165454a
Zone: 1
Management Hostname: dhcp47-175.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.175
Devices:
Id:dddb4b7848ba98763b469180fb7d2fc6   Name:/dev/sdd            State:failed    Size (GiB):15      Used (GiB):0       Free (GiB):15      

[root@dhcp47-20 ~]# heketi-cli node delete 92f58358e92be0f010523f87ca4ba017

Error: Unable to delete node [92f58358e92be0f010523f87ca4ba017] because it contains devices

[root@dhcp47-20 ~]# heketi-cli node info 92f58358e92be0f010523f87ca4ba017
Node Id: 92f58358e92be0f010523f87ca4ba017
State: failed
Cluster Id: 83f2d91d36c5ea0076e6bd9ed165454a
Zone: 1
Management Hostname: dhcp47-175.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.175
Devices:
Id:dddb4b7848ba98763b469180fb7d2fc6   Name:/dev/sdd            State:failed    Size (GiB):15      Used (GiB):0       Free (GiB):15      

[root@dhcp47-20 ~]# heketi-cli device delete dddb4b7848ba98763b469180fb7d2fc6

Device dddb4b7848ba98763b469180fb7d2fc6 deleted

[root@dhcp47-20 ~]# heketi-cli node info 92f58358e92be0f010523f87ca4ba017

Node Id: 92f58358e92be0f010523f87ca4ba017
State: failed
Cluster Id: 83f2d91d36c5ea0076e6bd9ed165454a
Zone: 1
Management Hostname: dhcp47-175.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.175
Devices:

[root@dhcp47-20 ~]# heketi-cli node delete 92f58358e92be0f010523f87ca4ba017

Node 92f58358e92be0f010523f87ca4ba017 deleted

[root@dhcp47-20 ~]# heketi-cli node list

Id:75c151258eee05c64f16dcaa85549f7b	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:836b8213fa88164bc152804ceca0a13a	Cluster:83f2d91d36c5ea0076e6bd9ed165454a
Id:f7a3fe0efe012afa65250258b709ccbf	Cluster:83f2d91d36c5ea0076e6bd9ed165454a

Comment 14 Tejas Chaphekar 2017-08-21 08:20:44 UTC
Following builds were used for the verification

heketi-client-5.0.0-7.el7rhgs.x86_64
cns-deploy-5.0.0-14.el7rhgs.x86_64
Gluster - rhgs-server-rhel7:3.3.0-11
Heketi -  rhgs-volmanager-rhel7:3.3.0-9

Comment 16 Srijita Mukherjee 2017-10-03 09:52:55 UTC
Talur,

This bug has been proposed for CNS-3.6 release. Kindly review the doc text and acknowledge.

Comment 17 Raghavendra Talur 2017-10-04 09:24:58 UTC
made a minor change to command in the doc text. rest looks good.

Comment 19 errata-xmlrpc 2017-10-11 07:07:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2879


Note You need to log in before you can comment on or make changes to this bug.