Bug 1461118

Summary:	gluster-blockd comes up even when rpcbind service is not running
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Pranith Kumar K <pkarampu>
Component:	gluster-block	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED ERRATA	QA Contact:	Sweta Anandpara <sanandpa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.3	CC:	amukherj, prasanna.kalever, rhs-bugs, sanandpa
Target Milestone:	---
Target Release:	RHGS 3.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	gluster-block-0.2.1-2.el7rhgs	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-09-21 04:19:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1417151

Description Pranith Kumar K 2017-06-13 14:35:00 UTC

Description of problem:
Even when rpc register fails gluster-blockd keeps running but is not operational and doesn't recieve gluster-block rpcs from other servers. Instead when it goes to this state, it should die gracefully giving the reason why it is terminating.

This is the only hint we will find in the logs as to why things are not working as expected.

[2017-06-13 09:04:54.256866] ERROR: unable to register (GLUSTER_BLOCK, GLUSTER_BLOCK_VERS: Invalid argument) [at gluster-blockd.c+188 :]

Further discussion about the issue:
https://github.com/gluster/gluster-block/issues/26

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Pranith Kumar K 2017-06-13 14:36:28 UTC

https://review.gluster.org/17535

Comment 6 Sweta Anandpara 2017-07-03 10:23:11 UTC

Tested and verified this on the build glusterfs-3.8.4-31.el7rhgs.x86_64 and gluster-block-0.2.1-3.el7rhgs.x86_64.

Killing, stopping, disabling rpcbind service and 'systemctl restart gluster-blockd' results in rpcbind service to be started in the background. After talking with Pranithk, looks like rpcbind has been added as a dependency for gluster-blockd. 

In other words, if we kill rpcbind service and restart gluster-blockd it will NOT fail with the error mentioning that rpcbind service is not running. It will actually trigger rpcbind service to start and then start gluster-blockd as well. 

Moving this bz to verified in 3.3. Logs are pasted below..

[root@dhcp47-116 ~]# systemctl status gluster-blockd
● gluster-blockd.service - Gluster block storage utility
   Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-07-03 06:05:47 EDT; 14s ago
 Main PID: 32245 (gluster-blockd)
   CGroup: /system.slice/gluster-blockd.service
           └─32245 /usr/sbin/gluster-blockd --glfs-lru-count 5 --log-level INFO

Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Started Gluster block storage utility.
Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Starting Gluster block storage utility...
[root@dhcp47-116 ~]# systemctl status rpcbind
● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
   Active: active (running) since Mon 2017-07-03 06:05:47 EDT; 23s ago
  Process: 32243 ExecStart=/sbin/rpcbind -w $RPCBIND_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 32244 (rpcbind)
   CGroup: /system.slice/rpcbind.service
           └─32244 /sbin/rpcbind -w

Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Starting RPC bind service...
Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Started RPC bind service.
[root@dhcp47-116 ~]# systemctl stop rpcbind
Warning: Stopping rpcbind.service, but it can still be activated by:
  rpcbind.socket
[root@dhcp47-116 ~]# systemctl status rpcbind
● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
   Active: inactive (dead) since Mon 2017-07-03 06:06:18 EDT; 6s ago
  Process: 32243 ExecStart=/sbin/rpcbind -w $RPCBIND_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 32244 (code=exited, status=0/SUCCESS)

Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Starting RPC bind service...
Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Started RPC bind service.
Jul 03 06:06:18 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Stopping RPC bind service...
Jul 03 06:06:18 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Stopped RPC bind service.
[root@dhcp47-116 ~]# systemctl disable rpcbind
Removed symlink /etc/systemd/system/sockets.target.wants/rpcbind.socket.
[root@dhcp47-116 ~]# systemctl status rpcbind
● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
   Active: inactive (dead) since Mon 2017-07-03 06:06:18 EDT; 14s ago
 Main PID: 32244 (code=exited, status=0/SUCCESS)

Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Starting RPC bind service...
Jul 03 06:05:47 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Started RPC bind service.
Jul 03 06:06:18 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Stopping RPC bind service...
Jul 03 06:06:18 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Stopped RPC bind service.
[root@dhcp47-116 ~]# 
[root@dhcp47-116 ~]# 
[root@dhcp47-116 ~]# 
[root@dhcp47-116 ~]# systemctl restart  gluster-blockd
[root@dhcp47-116 ~]# systemctl status gluster-blockd
● gluster-blockd.service - Gluster block storage utility
   Loaded: loaded (/usr/lib/systemd/system/gluster-blockd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-07-03 06:06:44 EDT; 3s ago
 Main PID: 32331 (gluster-blockd)
   CGroup: /system.slice/gluster-blockd.service
           └─32331 /usr/sbin/gluster-blockd --glfs-lru-count 5 --log-level INFO

Jul 03 06:06:44 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Started Gluster block storage utility.
Jul 03 06:06:44 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Starting Gluster block storage utility...
[root@dhcp47-116 ~]# systemctl status rpcbind
● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)
   Active: active (running) since Mon 2017-07-03 06:06:44 EDT; 7s ago
  Process: 32329 ExecStart=/sbin/rpcbind -w $RPCBIND_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 32330 (rpcbind)
   CGroup: /system.slice/rpcbind.service
           └─32330 /sbin/rpcbind -w

Jul 03 06:06:44 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Starting RPC bind service...
Jul 03 06:06:44 dhcp47-116.lab.eng.blr.redhat.com systemd[1]: Started RPC bind service.
[root@dhcp47-116 ~]# 




[root@dhcp47-121 gluster-block]# ls
gluster-block-cli.log          gluster-blockd.log       tcmu-runner-glfs.log
gluster-block-configshell.log  gluster-block-gfapi.log  tcmu-runner.log
[root@dhcp47-121 gluster-block]# rpm -qa | grep gluster
glusterfs-3.8.4-31.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-31.el7rhgs.x86_64
glusterfs-server-3.8.4-31.el7rhgs.x86_64
samba-vfs-glusterfs-4.6.3-3.el7rhgs.x86_64
gluster-block-0.2.1-3.el7rhgs.x86_64
glusterfs-api-3.8.4-31.el7rhgs.x86_64
python-gluster-3.8.4-31.el7rhgs.noarch
vdsm-gluster-4.17.33-1.1.el7rhgs.noarch
gluster-nagios-addons-0.2.9-1.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-26.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-libs-3.8.4-31.el7rhgs.x86_64
glusterfs-fuse-3.8.4-31.el7rhgs.x86_64
glusterfs-events-3.8.4-31.el7rhgs.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-10.el7.x86_64
glusterfs-cli-3.8.4-31.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-31.el7rhgs.x86_64
glusterfs-rdma-3.8.4-31.el7rhgs.x86_64
[root@dhcp47-121 gluster-block]# 
[root@dhcp47-121 gluster-block]# gluster pool list
UUID					Hostname                         	State
a0557927-4e5e-4ff7-8dce-94873f867707	dhcp47-113.lab.eng.blr.redhat.com	Disconnected 
c0dac197-5a4d-4db7-b709-dbf8b8eb0896	dhcp47-114.lab.eng.blr.redhat.com	Connected 
f828fdfa-e08f-4d12-85d8-2121cafcf9d0	dhcp47-115.lab.eng.blr.redhat.com	Connected 
a96e0244-b5ce-4518-895c-8eb453c71ded	dhcp47-116.lab.eng.blr.redhat.com	Disconnected 
17eb3cef-17e7-4249-954b-fc19ec608304	dhcp47-117.lab.eng.blr.redhat.com	Connected 
49610061-1788-4cbc-9205-0e59fe91d842	localhost                        	Connected 
[root@dhcp47-121 gluster-block]# 
[root@dhcp47-121 gluster-block]# gluster-block list ozone
block
testblock
[root@dhcp47-121 gluster-block]# gluster-block info ozone/testblock
NAME: testblock
VOLUME: ozone
GBID: 1c8d90c6-f4fd-4902-848e-a847bb8635c4
SIZE: 21474836480
HA: 3
PASSWORD: ea2cbe3f-5d98-4e67-8696-5893371dff6e
BLOCK CONFIG NODE(S): 10.70.47.121 10.70.47.113 10.70.47.114
[root@dhcp47-121 gluster-block]# gluster-block info ozone/block
NAME: block
VOLUME: ozone
GBID: 37dd0c20-6510-47e0-8d63-63753b55e62e
SIZE: 1073741824
HA: 3
PASSWORD: 
BLOCK CONFIG NODE(S): 10.70.47.121 10.70.47.113 10.70.47.114
[root@dhcp47-121 gluster-block]#

Comment 8 errata-xmlrpc 2017-09-21 04:19:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2773