Bug 1027113

Summary: Service glusterd failes to start after the most recent update (3.11.6-201 kernel)
Product: [Fedora] Fedora Reporter: Boris Derzhavets <bderzhavets>
Component: glusterfsAssignee: Kaleb KEITHLEY <kkeithle>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: barumuga, bderzhavets, joe, jonathansteffan, kkeithle, ndevos, silas
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-06 13:52:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Boris Derzhavets 2013-11-06 07:58:45 UTC
Description of problem:

[root@ovirt1 ~]# service glusterd status
Redirecting to /bin/systemctl status  glusterd.service
glusterd.service - GlusterFS an clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled)
   Active: failed (Result: exit-code) since Wed 2013-11-06 11:32:44 MSK; 20min ago
  Process: 2739 ExecStart=/usr/sbin/glusterd -p /run/glusterd.pid (code=exited, status=255)
   CGroup: name=systemd:/system/glusterd.service
           ├─1108 /usr/sbin/glusterd -p /run/glusterd.pid
           ├─2858 /usr/sbin/glusterfsd -s 192.168.1.135 --volfile-id cinder-vol.192.168.1.135.mnt-brick1-cinder-vol -p /var/lib/glusterd/...
           ├─2866 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nf...
           └─2892 /sbin/rpc.statd

Nov 06 11:32:44 ovirt1.localdomain systemd[1]: Stopping GlusterFS an clustered file-system server...
Nov 06 11:32:44 ovirt1.localdomain systemd[1]: Starting GlusterFS an clustered file-system server...
Nov 06 11:32:44 ovirt1.localdomain systemd[1]: glusterd.service: control process exited, code=exited status=255
Nov 06 11:32:44 ovirt1.localdomain systemd[1]: Failed to start GlusterFS an clustered file-system server.
Nov 06 11:32:44 ovirt1.localdomain systemd[1]: Unit glusterd.service entered failed state.
Nov 06 11:32:47 ovirt1.localdomain rpc.statd[2892]: Version 1.2.7 starting
Nov 06 11:32:47 ovirt1.localdomain sm-notify[2894]: Version 1.2.7 starting

Attempt to debug :

[root@ovirt1 ~]# /usr/sbin/glusterd --debug
[2013-11-06 07:54:34.935415] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.1 (/usr/sbin/glusterd --debug)
[2013-11-06 07:54:34.936254] D [glusterfsd.c:563:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
[2013-11-06 07:54:34.939370] I [glusterd.c:962:init] 0-management: Using /var/lib/glusterd as working directory
[2013-11-06 07:54:34.939544] D [glusterd.c:332:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
[2013-11-06 07:54:34.939688] D [rpcsvc.c:1900:rpcsvc_init] 0-rpc-service: RPC service inited.
[2013-11-06 07:54:34.939705] D [rpcsvc.c:1666:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
[2013-11-06 07:54:34.939753] D [rpc-transport.c:249:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.4.1/rpc-transport/socket.so
[2013-11-06 07:54:34.941652] I [socket.c:3480:socket_init] 0-socket.management: SSL support is NOT enabled
[2013-11-06 07:54:34.941672] I [socket.c:3495:socket_init] 0-socket.management: using system polling thread
[2013-11-06 07:54:34.941706] D [name.c:557:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet
[2013-11-06 07:54:34.942288] E [socket.c:695:__socket_server_bind] 0-socket.management: binding to  failed: Address already in use
[2013-11-06 07:54:34.942305] E [socket.c:698:__socket_server_bind] 0-socket.management: Port is already in use
[2013-11-06 07:54:34.942339] W [rpcsvc.c:1396:rpcsvc_transport_create] 0-rpc-service: listening on transport failed
[2013-11-06 07:54:34.942361] E [glusterd.c:1056:init] 0-management: creation of listener failed
[2013-11-06 07:54:34.942374] E [xlator.c:390:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2013-11-06 07:54:34.942393] E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed
[2013-11-06 07:54:34.942408] E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed
[2013-11-06 07:54:34.942479] W [glusterfsd.c:1002:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x3df) [0x7fef80d4c8ef] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb0) [0x7fef80d4f6f0] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x103) [0x7fef80d4f603]))) 0-: received signum (0), shutting down
[2013-11-06 07:54:34.942501] D [glusterfsd-mgmt.c:2273:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given

Version-Release number of selected component (if applicable):

3.4.1

How reproducible:


Steps to Reproduce:
1.Run yum -y update
2.Run  yum install glusterfs glusterfs-server glusterfs-fuse
3.Try to start glusterd

Actual results:

failure

Expected results:

Success
Additional info:

Comment 1 Niels de Vos 2013-11-06 09:33:00 UTC
Important bits from the logs:

[2013-11-06 07:54:34.942288] E [socket.c:695:__socket_server_bind] 0-socket.management: binding to  failed: Address already in use
[2013-11-06 07:54:34.942305] E [socket.c:698:__socket_server_bind] 0-socket.management: Port is already in use
[2013-11-06 07:54:34.942339] W [rpcsvc.c:1396:rpcsvc_transport_create] 0-rpc-service: listening on transport failed


This sounds as if there is a process listening on tcp/24007. Could you check which process that is?

# netstat -tlpn | grep 24007

Comment 2 Kaleb KEITHLEY 2013-11-06 12:46:44 UTC
You also need to install glusterfs-libs and glusterfs-cli.

(dependencies for libvirt/oVirt/vdsm and other packaging considerations preclude making these dependencies of the other RPMs.)

Comment 3 Boris Derzhavets 2013-11-06 13:48:59 UTC
(In reply to Niels de Vos from comment #1)
> Important bits from the logs:
> 
> [2013-11-06 07:54:34.942288] E [socket.c:695:__socket_server_bind]
> 0-socket.management: binding to  failed: Address already in use
> [2013-11-06 07:54:34.942305] E [socket.c:698:__socket_server_bind]
> 0-socket.management: Port is already in use
> [2013-11-06 07:54:34.942339] W [rpcsvc.c:1396:rpcsvc_transport_create]
> 0-rpc-service: listening on transport failed
> 
> 
> This sounds as if there is a process listening on tcp/24007. Could you check
> which process that is?
> 
> # netstat -tlpn | grep 24007

Thank you very much port was busy by my fault
Glusterd works fine.