Description of problem: Issuing a peer probe results in a glusterd segmentation fault. Once in this state, if the peer is removed from /var/lib/glusterd/peers, glusterd will start. Probing a peer again leads to the same problem. Problematic peer entry: cat /var/lib/glusterd/peers/ip-10-0-50-25.us-west-1.compute.internal uuid=00000000-0000-0000-0000-000000000000 state=0 hostname1=ip-10-0-50-25.us-west-1.compute.internal Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0'. Program terminated with signal SIGSEGV, Segmentation fault. #0 x86_64_fallback_frame_state (context=0x7ffe5d9a3b50, context=0x7ffe5d9a3b50, fs=0x7ffe5d9a3c40) at ./md-unwind-support.h:58 58 ./md-unwind-support.h: No such file or directory. (gdb) bt #0 x86_64_fallback_frame_state (context=0x7ffe5d9a3b50, context=0x7ffe5d9a3b50, fs=0x7ffe5d9a3c40) at ./md-unwind-support.h:58 #1 uw_frame_state_for (context=context@entry=0x7ffe5d9a3b50, fs=fs@entry=0x7ffe5d9a3c40) at ../../../src/libgcc/unwind-dw2.c:1253 #2 0x00007f6371b2f6d8 in _Unwind_Backtrace (trace=0x7f6378bc2440 <backtrace_helper>, trace_argument=0x7ffe5d9a3e00) at ../../../src/libgcc/unwind.inc:290 #3 0x00007f6378bc25b6 in __GI___backtrace (array=array@entry=0x7ffe5d9a3e40, size=size@entry=200) at ../sysdeps/x86_64/backtrace.c:109 #4 0x00007f63796f3f42 in _gf_msg_backtrace_nomem (level=level@entry=GF_LOG_ALERT, stacksize=stacksize@entry=200) at logging.c:1094 #5 0x00007f63796fd494 in gf_print_trace (signum=11, ctx=0x7f637a3ac010) at common-utils.c:737 #6 <signal handler called> #7 0x00000001725cc6c8 in ?? () #8 0x0000000000000000 in ?? () Version-Release number of selected component (if applicable): $ glusterd --version glusterfs 3.8.11 from package glusterfs-server 3.8.11-ubuntu1~trusty1 How reproducible: 1:1 Steps to Reproduce: 1. Install gluster on Ubuntu 14.04 2. sudo /usr/sbin/gluster --log-level=TRACE peer probe ip-10-0-50-25.us-west-1.compute.internal Connection failed. Please check if gluster daemon is operational. Actual results: Glusterd crashes on peer probe. Expected results: Glusterd should not crash on peer probe. Additional info: There's another issue which may be related. I noticed that glusterd.info was not self-populating. As a workaround I issue 'gluster pool list' which triggers glusterd to generate and store a UUID: cat /var/lib/glusterd/glusterd.info UUID=ad7b8337-ec4d-4917-ad6b-ca0e4d0eba42 operating-version=30800 This looks a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1293594 Crash loop: $ sudo gdb glusterd GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. [44/1919] For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from glusterd...Reading symbols from /usr/lib/debug/.build-id/fa/703514fceaf89ef9f0626ee95d362c941cc158.debug...done. done. (gdb) r --debug -p /var/run/glusterd.pid Starting program: /usr/sbin/glusterd --debug -p /var/run/glusterd.pid [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". warning: the debug information found in "/usr/lib/debug//lib/x86_64-linux-gnu/libuuid.so.1.3.0" does not match "/lib/x86_64-linux-gnu/libuuid.so.1" (CRC mismatch). warning: the debug information found in "/usr/lib/debug/lib/x86_64-linux-gnu/libuuid.so.1.3.0" does not match "/lib/x86_64-linux-gnu/libuuid.so.1" (CRC mismatch). [2017-05-03 05:12:29.642538] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 (args: /usr/sbin/glusterd --debug -p /var/run/glusterd.pid) [2017-05-03 05:12:29.642706] D [MSGID: 0] [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/glusterd.pid updated with pid 2588 [2017-05-03 05:12:29.642760] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [New Thread 0x7ffff4e19700 (LWP 2592)] [New Thread 0x7ffff4618700 (LWP 2593)] [New Thread 0x7ffff3e17700 (LWP 2594)] [New Thread 0x7ffff3616700 (LWP 2595)] [2017-05-03 05:12:29.647173] D [MSGID: 0] [glusterfsd.c:660:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2017-05-03 05:12:29.758748] I [MSGID: 106478] [glusterd.c:1381:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-05-03 05:12:29.758849] I [MSGID: 106479] [glusterd.c:1430:init] 0-management: Using /var/lib/glusterd as working directory [2017-05-03 05:12:29.758938] D [MSGID: 0] [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 128 [2017-05-03 05:12:29.759078] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-03 05:12:29.759127] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-03 05:12:29.759184] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-03 05:12:29.780522] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=60 [2017-05-03 05:12:29.780610] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-03 05:12:29.780649] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-03 05:12:29.780697] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-03 05:12:29.780819] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2017-05-03 05:12:29.780871] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2017-05-03 05:12:29.780915] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2017-05-03 05:12:29.780960] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2017-05-03 05:12:29.781004] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2017-05-03 05:12:29.781048] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2017-05-03 05:12:29.781086] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2017-05-03 05:12:29.781145] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-03 05:12:29.781173] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-03 05:12:29.781209] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-03 05:12:29.781254] D [socket.c:3887:socket_init] 0-socket.management: disabling nodelay [2017-05-03 05:12:29.781280] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-03 05:12:29.781304] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-03 05:12:29.781327] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-03 05:12:29.781350] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-03 05:12:29.781410] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2017-05-03 05:12:29.781442] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2017-05-03 05:12:29.781487] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-03 05:12:29.781519] I [MSGID: 106060] [glusterd.c:1714:init] 0-management: base-port override: 49152 [2017-05-03 05:12:29.781547] D [MSGID: 0] [glusterd.c:1722:init] 0-management: cannot get run-with-valgrind value [2017-05-03 05:12:29.836271] D [MSGID: 0] [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 [2017-05-03 05:12:29.836406] D [MSGID: 0] [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 [2017-05-03 05:12:31.196124] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-03 05:12:31.196227] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning 0 [2017-05-03 05:12:31.196281] D [MSGID: 0] [store.c:306:gf_store_retrieve_value] 0-: key operating-version found [2017-05-03 05:12:31.196353] I [MSGID: 106513] [glusterd-store.c:2098:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30800 [2017-05-03 05:12:31.196421] D [MSGID: 0] [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: Returning with 0 [2017-05-03 05:12:31.196489] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-03 05:12:31.196525] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning 0 [2017-05-03 05:12:31.196558] D [MSGID: 0] [store.c:500:gf_store_iter_new] 0-: Returning with 0 [2017-05-03 05:12:31.196594] D [MSGID: 0] [store.c:613:gf_store_iter_get_next] 0-: Returning with 0 [2017-05-03 05:12:31.196625] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-03 05:12:31.196664] D [MSGID: 0] [store.c:613:gf_store_iter_get_next] 0-: Returning with 0 [2017-05-03 05:12:31.196698] D [logging.c:1953:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk [2017-05-03 05:12:31.196694] D [MSGID: 0] [store.c:613:gf_store_iter_get_next] 0-: Returning with 0 [2017-05-03 05:12:31.196697] D [MSGID: 0] [glusterd-peer-utils.c:487:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2017-05-03 05:12:31.196778] D [MSGID: 0] [store.c:613:gf_store_iter_get_next] 0-: Returning with -1 [2017-05-03 05:12:31.196813] I [MSGID: 106498] [glusterd-handler.c:3649:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-05-03 05:12:31.196871] D [MSGID: 0] [glusterd-handler.c:3461:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2017-05-03 05:12:31.196914] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-05-03 05:12:31.196940] D [rpc-clnt.c:1058:rpc_clnt_connection_init] 0-management: setting ping-timeout to 42 [2017-05-03 05:12:31.196967] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-03 05:12:31.197020] D [socket.c:3938:socket_init] 0-management: Configued transport.tcp-user-timeout=60 [2017-05-03 05:12:31.197050] D [socket.c:4021:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2017-05-03 05:12:31.197073] D [socket.c:4024:socket_init] 0-management: SSL support for glusterd is NOT enabled [2017-05-03 05:12:31.197095] D [socket.c:4041:socket_init] 0-management: using system polling thread [2017-05-03 05:12:31.197125] D [name.c:168:client_fill_address_family] 0-management: address-family not specified, marking it as unspec for getaddrinfo to resolve from (remote-host: ip-10-0-50-25.us-west-1.compute.internal) [2017-05-03 05:12:31.209975] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-10.0.50.25 (port-24007) for hostname: ip-10-0-50-25.us-west-1.compute.internal and port: 24007 [2017-05-03 05:12:31.210052] D [socket.c:2899:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2017-05-03 05:12:31.210219] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 24007, higher: 24008 [2017-05-03 05:12:31.210267] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 32765, higher: -1 [2017-05-03 05:12:31.210340] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: -1, higher: -1 Program received signal SIGSEGV, Segmentation fault. 0x00000001f08176c8 in ?? () (gdb) bt #0 0x00000001f08176c8 in ?? () #1 0x0000000000000000 in ?? ()
Note that _Unwind_Backtrace is a libgcc function that eventually gets called when a gluster process crashed. This bug seems to be about a 2nd segfault that happened while gluster tried to gather a backtrace to show where the 1st segfault occurred. It would be good to have a way to reproduce this, or have access to the full core file (also need the debug symbols, I have no idea how to do that on Ubuntu).
ProblemType: Crash Architecture: amd64 Date: Wed May 3 21:50:49 2017 Dependencies: adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.17 attr 1:2.4.47-1ubuntu1 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 coreutils 8.21-1ubuntu5.4 cpio 2.11+dfsg-1ubuntu1.2 dbus 1.6.18-0ubuntu4.5 debconf 1.5.51ubuntu2 debconf-i18n 1.5.51ubuntu2 debianutils 4.4 dmsetup 2:1.02.77-6ubuntu2 dpkg 1.17.5ubuntu5.7 e2fslibs 1.42.9-3ubuntu1.3 e2fsprogs 1.42.9-3ubuntu1.3 file 1:5.14-2ubuntu3.3 findutils 4.4.2-7 gcc-4.8-base 4.8.4-2ubuntu1~14.04.3 gcc-4.9-base 4.9.3-0ubuntu4 ifupdown 0.7.47.2ubuntu4.4 initramfs-tools 0.103ubuntu4.6 initramfs-tools-bin 0.103ubuntu4.6 initscripts 2.88dsf-41ubuntu6.3 insserv 1.14.0-5ubuntu2 iproute2 3.12.0-2ubuntu1 isc-dhcp-client 4.2.4-7ubuntu12.8 isc-dhcp-common 4.2.4-7ubuntu12.8 klibc-utils 2.0.3-0ubuntu1.14.04.2 kmod 15-0ubuntu6 libacl1 2.2.52-1 libaio1 0.3.109-4 libapparmor1 2.10.95-0ubuntu2.5~14.04.1 libapt-inst1.5 1.0.1ubuntu2.17 libapt-pkg4.12 1.0.1ubuntu2.17 libattr1 1:2.4.47-1ubuntu1 libaudit-common 1:2.3.2-2ubuntu1 libaudit1 1:2.3.2-2ubuntu1 libblkid1 2.20.1-5.1ubuntu20.9 libbz2-1.0 1.0.6-5 libc6 2.19-0ubuntu6.11 libcap2 1:2.24-0ubuntu2 libcgmanager0 0.24-0ubuntu7.5 libcomerr2 1.42.9-3ubuntu1.3 libdb5.3 5.3.28-3ubuntu3 libdbus-1-3 1.6.18-0ubuntu4.5 libdebconfclient0 0.187ubuntu1 libdevmapper-event1.02.1 2:1.02.77-6ubuntu2 libdevmapper1.02.1 2:1.02.77-6ubuntu2 libdrm2 2.4.67-1ubuntu0.14.04.1 libexpat1 2.1.0-4ubuntu1.3 libffi6 3.1~rc1+r3.0.13-12ubuntu0.1 libgcc1 1:4.9.3-0ubuntu4 libglib2.0-0 2.40.2-0ubuntu1 libglib2.0-data 2.40.2-0ubuntu1 libgpm2 1.20.4-6.1 libibverbs1 1.1.7-1ubuntu1.1 libjson-c2 0.11-3ubuntu1.2 libjson0 0.11-3ubuntu1.2 libklibc 2.0.3-0ubuntu1.14.04.2 libkmod2 15-0ubuntu6 liblocale-gettext-perl 1.05-7build3 liblvm2app2.2 2.02.98-6ubuntu2 liblzma5 5.1.1alpha+20120614-2ubuntu2 libmagic1 1:5.14-2ubuntu3.3 libmount1 2.20.1-5.1ubuntu20.9 libncurses5 5.9+20140118-1ubuntu1 libncursesw5 5.9+20140118-1ubuntu1 libnih-dbus1 1.0.3-4ubuntu25 libnih1 1.0.3-4ubuntu25 libpam-modules 1.1.8-1ubuntu2.2 libpam-modules-bin 1.1.8-1ubuntu2.2 libpam-runtime 1.1.8-1ubuntu2.2 libpam-systemd 204-5ubuntu20.24 libpam0g 1.1.8-1ubuntu2.2 libpcre3 1:8.31-2ubuntu2.3 libplymouth2 0.8.8-0ubuntu17.1 libpng12-0 1.2.50-1ubuntu2.14.04.2 libprocps3 1:3.3.9-1ubuntu2.2 libpython2.7 2.7.6-8ubuntu0.3 libpython2.7-minimal 2.7.6-8ubuntu0.3 libpython2.7-stdlib 2.7.6-8ubuntu0.3 librdmacm1 1.0.16-1 libreadline6 6.3-4ubuntu2 libselinux1 2.2.2-1ubuntu0.1 libsemanage-common 2.2-1 libsemanage1 2.2-1 libsepol1 2.2-1ubuntu0.1 libslang2 2.2.4-15ubuntu1 libsqlite3-0 3.8.2-1ubuntu2.1 libss2 1.42.9-3ubuntu1.3 libssl1.0.0 1.0.1f-1ubuntu2.22 libstdc++6 4.8.4-2ubuntu1~14.04.3 libsystemd-daemon0 204-5ubuntu20.24 libsystemd-login0 204-5ubuntu20.24 libtext-charwidth-perl 0.04-7build3 libtext-iconv-perl 1.7-5build2 libtext-wrapi18n-perl 0.06-7 libtinfo5 5.9+20140118-1ubuntu1 libudev1 204-5ubuntu20.24 liburcu1 0.7.12-0ubuntu2 libustr-1.0-1 1.0.4-3ubuntu2 libuuid1 2.20.1-5.1ubuntu20.9 libxml2 2.9.1+dfsg1-3ubuntu4.9 libxtables10 1.4.21-1ubuntu1 lsb-base 4.1+Debian11ubuntu6.2 makedev 2.3.1-93ubuntu1 mime-support 3.54ubuntu1.1 module-init-tools 15-0ubuntu6 mount 2.20.1-5.1ubuntu20.9 mountall 2.53 multiarch-support 2.19-0ubuntu6.11 netbase 5.2 passwd 1:4.1.5.1-1ubuntu9.2 perl-base 5.18.2-2ubuntu1.1 plymouth 0.8.8-0ubuntu17.1 plymouth-theme-ubuntu-text 0.8.8-0ubuntu17.1 procps 1:3.3.9-1ubuntu2.2 psmisc 22.20-1ubuntu2 readline-common 6.3-4ubuntu2 sed 4.2.2-4ubuntu1 sensible-utils 0.0.9 sgml-base 1.26+nmu4ubuntu1 shared-mime-info 1.2-0ubuntu3 systemd-services 204-5ubuntu20.24 systemd-shim 6-2bzr1 sysv-rc 2.88dsf-41ubuntu6.3 sysvinit-utils 2.88dsf-41ubuntu6.3 tar 1.27.1-1ubuntu0.1 tzdata 2016j-0ubuntu0.14.04 udev 204-5ubuntu20.24 upstart 1.12.1-0ubuntu4.2 util-linux 2.20.1-5.1ubuntu20.9 uuid-runtime 2.20.1-5.1ubuntu20.9 xml-core 0.13+nmu2 zlib1g 1:1.2.8.dfsg-1ubuntu1 Disassembly: => 0x7f5784fbf668 <uw_frame_state_for+488>: cmpb $0x48,(%rdx) 0x7f5784fbf66b <uw_frame_state_for+491>: je 0x7f5784fbfad8 <uw_frame_state_for+1624> 0x7f5784fbf671 <uw_frame_state_for+497>: nopl 0x0(%rax) 0x7f5784fbf678 <uw_frame_state_for+504>: mov $0x5,%eax 0x7f5784fbf67d <uw_frame_state_for+509>: add $0x58,%rsp 0x7f5784fbf681 <uw_frame_state_for+513>: pop %rbx 0x7f5784fbf682 <uw_frame_state_for+514>: pop %rbp 0x7f5784fbf683 <uw_frame_state_for+515>: pop %r12 0x7f5784fbf685 <uw_frame_state_for+517>: pop %r13 0x7f5784fbf687 <uw_frame_state_for+519>: pop %r14 0x7f5784fbf689 <uw_frame_state_for+521>: pop %r15 0x7f5784fbf68b <uw_frame_state_for+523>: retq 0x7f5784fbf68c <uw_frame_state_for+524>: nopl 0x0(%rax) 0x7f5784fbf690 <uw_frame_state_for+528>: movzbl (%rbx),%eax 0x7f5784fbf693 <uw_frame_state_for+531>: add $0x1,%rbx 0x7f5784fbf697 <uw_frame_state_for+535>: mov %al,0x170(%r12) DistroRelease: Ubuntu 14.04 ExecutablePath: /usr/sbin/glusterfsd ExecutableTimestamp: 1491908992 Package: glusterfs-common 3.8.11-ubuntu1~trusty1 [origin: LP-PPA-gluster-glusterfs-3.8] PackageArchitecture: amd64 ProcCmdline: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0 ProcCwd: / ProcEnviron: TERM=linux PATH=(custom, no user) ProcMaps: 7f5778000000-7f5778204000 rw-p 00000000 00:00 0 [stack:1100] 7f5778204000-7f577c000000 ---p 00000000 00:00 0 7f577c000000-7f577c021000 rw-p 00000000 00:00 0 7f577c021000-7f5780000000 ---p 00000000 00:00 0 7f5780000000-7f5780021000 rw-p 00000000 00:00 0 7f5780021000-7f5784000000 ---p 00000000 00:00 0 7f5784fb0000-7f5784fc6000 r-xp 00000000 ca:01 396165 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f5784fc6000-7f57851c5000 ---p 00016000 ca:01 396165 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f57851c5000-7f57851c6000 rw-p 00015000 ca:01 396165 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f57851c6000-7f57851d0000 r-xp 00000000 ca:01 397593 /lib/x86_64-linux-gnu/libnss_files-2.19.so 7f57851d0000-7f57853cf000 ---p 0000a000 ca:01 397593 /lib/x86_64-linux-gnu/libnss_files-2.19.so 7f57853cf000-7f57853d0000 r--p 00009000 ca:01 397593 /lib/x86_64-linux-gnu/libnss_files-2.19.so 7f57853d0000-7f57853d1000 rw-p 0000a000 ca:01 397593 /lib/x86_64-linux-gnu/libnss_files-2.19.so 7f57853d1000-7f57853d2000 ---p 00000000 00:00 0 7f57853d2000-7f5785bd2000 rw-p 00000000 00:00 0 [stack:1542] 7f5785bd2000-7f5785bd3000 ---p 00000000 00:00 0 7f5785bd3000-7f57863d3000 rw-p 00000000 00:00 0 [stack:1541] 7f57863d3000-7f5786428000 r-xp 00000000 ca:01 396212 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f5786428000-7f5786628000 ---p 00055000 ca:01 396212 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f5786628000-7f578662b000 r--p 00055000 ca:01 396212 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f578662b000-7f5786632000 rw-p 00058000 ca:01 396212 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f5786632000-7f5786644000 r-xp 00000000 ca:01 528487 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so 7f5786644000-7f5786843000 ---p 00012000 ca:01 528487 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so 7f5786843000-7f5786844000 r--p 00011000 ca:01 528487 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so 7f5786844000-7f578684a000 rw-p 00012000 ca:01 528487 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so 7f578684a000-7f5786887000 r-xp 00000000 ca:01 396196 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f5786887000-7f5786a86000 ---p 0003d000 ca:01 396196 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f5786a86000-7f5786a87000 r--p 0003c000 ca:01 396196 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f5786a87000-7f5786a88000 rw-p 0003d000 ca:01 396196 /lib/x86_64-linux-gnu/libpcre.so.3.13.1 7f5786a88000-7f5786aa8000 r-xp 00000000 ca:01 396208 /lib/x86_64-linux-gnu/libselinux.so.1 7f5786aa8000-7f5786ca7000 ---p 00020000 ca:01 396208 /lib/x86_64-linux-gnu/libselinux.so.1 7f5786ca7000-7f5786ca8000 r--p 0001f000 ca:01 396208 /lib/x86_64-linux-gnu/libselinux.so.1 7f5786ca8000-7f5786ca9000 rw-p 00020000 ca:01 396208 /lib/x86_64-linux-gnu/libselinux.so.1 7f5786ca9000-7f5786cab000 rw-p 00000000 00:00 0 7f5786cab000-7f5786cef000 r-xp 00000000 ca:01 396158 /lib/x86_64-linux-gnu/libdbus-1.so.3.7.6 7f5786cef000-7f5786eee000 ---p 00044000 ca:01 396158 /lib/x86_64-linux-gnu/libdbus-1.so.3.7.6 7f5786eee000-7f5786eef000 r--p 00043000 ca:01 396158 /lib/x86_64-linux-gnu/libdbus-1.so.3.7.6 7f5786eef000-7f5786ef0000 rw-p 00044000 ca:01 396158 /lib/x86_64-linux-gnu/libdbus-1.so.3.7.6 7f5786ef0000-7f5786ef8000 r-xp 00000000 ca:01 396180 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 7f5786ef8000-7f57870f8000 ---p 00008000 ca:01 396180 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 7f57870f8000-7f57870f9000 r--p 00008000 ca:01 396180 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 7f57870f9000-7f57870fa000 rw-p 00009000 ca:01 396180 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 7f57870fa000-7f5787111000 r-xp 00000000 ca:01 396181 /lib/x86_64-linux-gnu/libnih.so.1.0.0 7f5787111000-7f5787310000 ---p 00017000 ca:01 396181 /lib/x86_64-linux-gnu/libnih.so.1.0.0 7f5787310000-7f5787311000 r--p 00016000 ca:01 396181 /lib/x86_64-linux-gnu/libnih.so.1.0.0 7f5787311000-7f5787312000 rw-p 00017000 ca:01 396181 /lib/x86_64-linux-gnu/libnih.so.1.0.0 7f5787312000-7f578732b000 r-xp 00000000 ca:01 396153 /lib/x86_64-linux-gnu/libcgmanager.so.0.0.0 7f578732b000-7f578752a000 ---p 00019000 ca:01 396153 /lib/x86_64-linux-gnu/libcgmanager.so.0.0.0 7f578752a000-7f578752c000 r--p 00018000 ca:01 396153 /lib/x86_64-linux-gnu/libcgmanager.so.0.0.0 7f578752c000-7f578752d000 rw-p 0001a000 ca:01 396153 /lib/x86_64-linux-gnu/libcgmanager.so.0.0.0 7f578752d000-7f5787532000 r-xp 00000000 ca:01 398552 /lib/x86_64-linux-gnu/libdevmapper-event.so.1.02.1 7f5787532000-7f5787731000 ---p 00005000 ca:01 398552 /lib/x86_64-linux-gnu/libdevmapper-event.so.1.02.1 7f5787731000-7f5787732000 r--p 00004000 ca:01 398552 /lib/x86_64-linux-gnu/libdevmapper-event.so.1.02.1 7f5787732000-7f5787733000 rw-p 00005000 ca:01 398552 /lib/x86_64-linux-gnu/libdevmapper-event.so.1.02.1 7f5787733000-7f5787768000 r-xp 00000000 ca:01 396159 /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 7f5787768000-7f5787967000 ---p 00035000 ca:01 396159 /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 7f5787967000-7f5787968000 r--p 00034000 ca:01 396159 /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 7f5787968000-7f578796b000 rw-p 00035000 ca:01 396159 /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 7f578796b000-7f578796c000 rw-p 00000000 00:00 0 7f578796c000-7f578797c000 r-xp 00000000 ca:01 396217 /lib/x86_64-linux-gnu/libudev.so.1.3.5 7f578797c000-7f5787b7b000 ---p 00010000 ca:01 396217 /lib/x86_64-linux-gnu/libudev.so.1.3.5 7f5787b7b000-7f5787b7c000 r--p 0000f000 ca:01 396217 /lib/x86_64-linux-gnu/libudev.so.1.3.5 7f5787b7c000-7f5787b7d000 rw-p 00010000 ca:01 396217 /lib/x86_64-linux-gnu/libudev.so.1.3.5 7f5787b7d000-7f5787c82000 r-xp 00000000 ca:01 397588 /lib/x86_64-linux-gnu/libm-2.19.so 7f5787c82000-7f5787e81000 ---p 00105000 ca:01 397588 /lib/x86_64-linux-gnu/libm-2.19.so 7f5787e81000-7f5787e82000 r--p 00104000 ca:01 397588 /lib/x86_64-linux-gnu/libm-2.19.so 7f5787e82000-7f5787e83000 rw-p 00105000 ca:01 397588 /lib/x86_64-linux-gnu/libm-2.19.so 7f5787e83000-7f5787ea4000 r-xp 00000000 ca:01 396173 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f5787ea4000-7f57880a3000 ---p 00021000 ca:01 396173 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f57880a3000-7f57880a4000 r--p 00020000 ca:01 396173 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f57880a4000-7f57880a5000 rw-p 00021000 ca:01 396173 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f57880a5000-7f5788141000 r-xp 00000000 ca:01 398554 /lib/x86_64-linux-gnu/liblvm2app.so.2.2 7f5788141000-7f5788341000 ---p 0009c000 ca:01 398554 /lib/x86_64-linux-gnu/liblvm2app.so.2.2 7f5788341000-7f5788342000 r--p 0009c000 ca:01 398554 /lib/x86_64-linux-gnu/liblvm2app.so.2.2 7f5788342000-7f5788345000 rw-p 0009d000 ca:01 398554 /lib/x86_64-linux-gnu/liblvm2app.so.2.2 7f5788345000-7f5788364000 rw-p 00000000 00:00 0 7f5788364000-7f5788369000 r-xp 00000000 ca:01 32571 /usr/lib/liburcu-bp.so.1.0.0 7f5788369000-7f5788569000 ---p 00005000 ca:01 32571 /usr/lib/liburcu-bp.so.1.0.0 7f5788569000-7f578856a000 r--p 00005000 ca:01 32571 /usr/lib/liburcu-bp.so.1.0.0 7f578856a000-7f578856b000 rw-p 00006000 ca:01 32571 /usr/lib/liburcu-bp.so.1.0.0 7f578856b000-7f57886c7000 r-xp 00000000 ca:01 8180 /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1 7f57886c7000-7f57888c7000 ---p 0015c000 ca:01 8180 /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1 7f57888c7000-7f57888cf000 r--p 0015c000 ca:01 8180 /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1 7f57888cf000-7f57888d1000 rw-p 00164000 ca:01 8180 /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1 7f57888d1000-7f57888d2000 rw-p 00000000 00:00 0 7f57888d2000-7f5788a2a000 r-xp 00000000 ca:01 528569 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so 7f5788a2a000-7f5788c29000 ---p 00158000 ca:01 528569 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so 7f5788c29000-7f5788c2b000 r--p 00157000 ca:01 528569 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so 7f5788c2b000-7f5788c3a000 rw-p 00159000 ca:01 528569 /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so 7f5788c3a000-7f5788c3d000 rw-p 00000000 00:00 0 7f5788c3d000-7f5788c3e000 ---p 00000000 00:00 0 7f5788c3e000-7f578943e000 rw-p 00000000 00:00 0 7f578943e000-7f578943f000 ---p 00000000 00:00 0 7f578943f000-7f5789c3f000 rw-p 00000000 00:00 0 [stack:1099] 7f5789c3f000-7f5789c40000 ---p 00000000 00:00 0 7f5789c40000-7f578a440000 rw-p 00000000 00:00 0 [stack:1098] 7f578a440000-7f578a441000 ---p 00000000 00:00 0 7f578a441000-7f578ac41000 rw-p 00000000 00:00 0 [stack:1097] 7f578ac41000-7f578c124000 rw-p 00000000 00:00 0 7f578c124000-7f578c2d7000 r-xp 00000000 ca:01 396157 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f578c2d7000-7f578c4d6000 ---p 001b3000 ca:01 396157 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f578c4d6000-7f578c4f1000 r--p 001b2000 ca:01 396157 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f578c4f1000-7f578c4fc000 rw-p 001cd000 ca:01 396157 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f578c4fc000-7f578c500000 rw-p 00000000 00:00 0 7f578c500000-7f578c507000 r-xp 00000000 ca:01 397575 /lib/x86_64-linux-gnu/librt-2.19.so 7f578c507000-7f578c706000 ---p 00007000 ca:01 397575 /lib/x86_64-linux-gnu/librt-2.19.so 7f578c706000-7f578c707000 r--p 00006000 ca:01 397575 /lib/x86_64-linux-gnu/librt-2.19.so 7f578c707000-7f578c708000 rw-p 00007000 ca:01 397575 /lib/x86_64-linux-gnu/librt-2.19.so 7f578c708000-7f578c70c000 r-xp 00000000 ca:01 394652 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f578c70c000-7f578c90b000 ---p 00004000 ca:01 394652 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f578c90b000-7f578c90c000 r--p 00003000 ca:01 394652 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f578c90c000-7f578c90d000 rw-p 00004000 ca:01 394652 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f578c90d000-7f578c925000 r-xp 00000000 ca:01 396224 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f578c925000-7f578cb24000 ---p 00018000 ca:01 396224 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f578cb24000-7f578cb25000 r--p 00017000 ca:01 396224 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f578cb25000-7f578cb26000 rw-p 00018000 ca:01 396224 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f578cb26000-7f578cce4000 r-xp 00000000 ca:01 397584 /lib/x86_64-linux-gnu/libc-2.19.so 7f578cce4000-7f578cee3000 ---p 001be000 ca:01 397584 /lib/x86_64-linux-gnu/libc-2.19.so 7f578cee3000-7f578cee7000 r--p 001bd000 ca:01 397584 /lib/x86_64-linux-gnu/libc-2.19.so 7f578cee7000-7f578cee9000 rw-p 001c1000 ca:01 397584 /lib/x86_64-linux-gnu/libc-2.19.so 7f578cee9000-7f578ceee000 rw-p 00000000 00:00 0 7f578ceee000-7f578cf07000 r-xp 00000000 ca:01 397595 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f578cf07000-7f578d106000 ---p 00019000 ca:01 397595 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f578d106000-7f578d107000 r--p 00018000 ca:01 397595 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f578d107000-7f578d108000 rw-p 00019000 ca:01 397595 /lib/x86_64-linux-gnu/libpthread-2.19.so 7f578d108000-7f578d10c000 rw-p 00000000 00:00 0 7f578d10c000-7f578d10f000 r-xp 00000000 ca:01 397594 /lib/x86_64-linux-gnu/libdl-2.19.so 7f578d10f000-7f578d30e000 ---p 00003000 ca:01 397594 /lib/x86_64-linux-gnu/libdl-2.19.so 7f578d30e000-7f578d30f000 r--p 00002000 ca:01 397594 /lib/x86_64-linux-gnu/libdl-2.19.so 7f578d30f000-7f578d310000 rw-p 00003000 ca:01 397594 /lib/x86_64-linux-gnu/libdl-2.19.so 7f578d310000-7f578d327000 r-xp 00000000 ca:01 32607 /usr/lib/x86_64-linux-gnu/libgfxdr.so.0.0.1 7f578d327000-7f578d526000 ---p 00017000 ca:01 32607 /usr/lib/x86_64-linux-gnu/libgfxdr.so.0.0.1 7f578d526000-7f578d527000 r--p 00016000 ca:01 32607 /usr/lib/x86_64-linux-gnu/libgfxdr.so.0.0.1 7f578d527000-7f578d528000 rw-p 00017000 ca:01 32607 /usr/lib/x86_64-linux-gnu/libgfxdr.so.0.0.1 7f578d528000-7f578d542000 r-xp 00000000 ca:01 32600 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0.0.1 7f578d542000-7f578d741000 ---p 0001a000 ca:01 32600 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0.0.1 7f578d741000-7f578d742000 r--p 00019000 ca:01 32600 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0.0.1 7f578d742000-7f578d743000 rw-p 0001a000 ca:01 32600 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0.0.1 7f578d743000-7f578d827000 r-xp 00000000 ca:01 32604 /usr/lib/x86_64-linux-gnu/libglusterfs.so.0.0.1 7f578d827000-7f578da27000 ---p 000e4000 ca:01 32604 /usr/lib/x86_64-linux-gnu/libglusterfs.so.0.0.1 7f578da27000-7f578da28000 r--p 000e4000 ca:01 32604 /usr/lib/x86_64-linux-gnu/libglusterfs.so.0.0.1 7f578da28000-7f578da29000 rw-p 000e5000 ca:01 32604 /usr/lib/x86_64-linux-gnu/libglusterfs.so.0.0.1 7f578da29000-7f578da2c000 rw-p 00000000 00:00 0 7f578da2c000-7f578da4f000 r-xp 00000000 ca:01 397592 /lib/x86_64-linux-gnu/ld-2.19.so 7f578da79000-7f578dc4e000 rw-p 00000000 00:00 0 7f578dc4e000-7f578dc4f000 r--p 00022000 ca:01 397592 /lib/x86_64-linux-gnu/ld-2.19.so 7f578dc4f000-7f578dc50000 rw-p 00023000 ca:01 397592 /lib/x86_64-linux-gnu/ld-2.19.so 7f578dc50000-7f578dc51000 rw-p 00000000 00:00 0 7f578dc51000-7f578dc66000 r-xp 00000000 ca:01 32614 /usr/sbin/glusterfsd 7f578dca0000-7f578dcf7000 rw-p 00000000 00:00 0 7f578dcf7000-7f578de5d000 rw-p 00000000 00:00 0 7f578de5d000-7f578de60000 rw-p 00000000 00:00 0 7f578de61000-7f578de63000 rw-p 00000000 00:00 0 7f578de63000-7f578de66000 rw-p 00000000 00:00 0 7f578de66000-7f578de67000 r--p 00015000 ca:01 32614 /usr/sbin/glusterfsd 7f578de67000-7f578de69000 rw-p 00016000 ca:01 32614 /usr/sbin/glusterfsd 7f578e58c000-7f578e5d3000 rw-p 00000000 00:00 0 [heap] 7f578e5d3000-7f578e637000 rw-p 00000000 00:00 0 [heap] 7ffffd662000-7ffffd683000 rw-p 00000000 00:00 0 [stack] 7ffffd7e1000-7ffffd7e3000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] ProcStatus: Name: glusterd State: D (disk sleep) Tgid: 1077 Ngid: 0 Pid: 1077 PPid: 1 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 64 Groups: 0 VmPeak: 343208 kB VmSize: 343180 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 17052 kB VmRSS: 17052 kB VmData: 271772 kB VmStk: 136 kB VmExe: 84 kB VmLib: 11316 kB VmPTE: 224 kB VmSwap: 0 kB Threads: 7 SigQ: 0/15776 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000004a01 SigIgn: 0000000000001000 SigCgt: 00000001800000fa CapInh: 0000000000000000 CapPrm: 0000001fffffffff CapEff: 0000001fffffffff CapBnd: 0000001fffffffff Seccomp: 0 Cpus_allowed: 7fff Cpus_allowed_list: 0-14 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 107 nonvoluntary_ctxt_switches: 4 Registers: rax 0x7f57781c32f0 140013653996272 rbx 0x0 0 rcx 0x1 1 rdx 0x18663e6c8 6549661384 rsi 0x0 0 rdi 0x7f578dc4f990 140014017378704 rbp 0x7f57781c21b0 0x7f57781c21b0 rsp 0x7f57781c1e80 0x7f57781c1e80 r8 0x0 0 r9 0x7f57851c5490 140013872108688 r10 0x7f57851c5900 140013872109824 r11 0x7f5784fb0000 140013869924352 r12 0x7f57781c2000 140013653991424 r13 0x7f57781c1f10 140013653991184 r14 0x7f57781c21c0 140013653991872 r15 0x0 0 rip 0x7f5784fbf668 0x7f5784fbf668 <uw_frame_state_for+488> eflags 0x10246 [ PF ZF IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 RetraceOutdatedPackages: makedev version 2.3.1-93ubuntu1 required, but 2.3.1-93ubuntu2~ubuntu14.04.1 is available initramfs-tools-bin version 0.103ubuntu4.6 required, but 0.103ubuntu4.7 is available libapparmor1 version 2.10.95-0ubuntu2.5~14.04.1 required, but 2.10.95-0ubuntu2.6~14.04.1 is available initramfs-tools version 0.103ubuntu4.6 required, but 0.103ubuntu4.7 is available Signal: 11 SourcePackage: glusterfs Stacktrace: #0 x86_64_fallback_frame_state (context=0x7f57781c1f10, context=0x7f57781c1f10, fs=0x7f57781c2000) at ./md-unwind-support.h:58 pc = 0x18663e6c8 <error: Cannot access memory at address 0x18663e6c8> sc = <optimized out> new_cfa = <optimized out> #1 uw_frame_state_for (context=context@entry=0x7f57781c1f10, fs=fs@entry=0x7f57781c2000) at ../../../src/libgcc/unwind-dw2.c:1253 fde = 0x0 cie = <optimized out> aug = <optimized out> insn = <optimized out> end = <optimized out> #2 0x00007f5784fc06d8 in _Unwind_Backtrace (trace=0x7f578cc34440 <backtrace_helper>, trace_argument=0x7f57781c21c0) at ../../../src/libgcc/unwind.inc:290 fs = {regs = {reg = {{loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 18 times>}, prev = 0x0, cfa_offset = 0, cfa_reg = 0, cfa_exp = 0x0, cfa_how = CFA_UNSET}, pc = 0x0, personality = 0x0, data_align = 0, code_align = 0, retaddr_column = 0, fde_encoding = 0 '\000', lsda_encoding = 0 '\000', saw_z = 0 '\000', signal_frame = 0 '\000', eh_ptr = 0x0} context = {reg = {0x7f57781c2dd0, 0x7f57781c2dc8, 0x7f57781c2dd8, 0x7f57781c2dc0, 0x7f57781c2db0, 0x7f57781c2da8, 0x7f57781c2db8, 0x7f57781c2de0, 0x7f57781c2d68, 0x7f57781c2d70, 0x7f57781c2d78, 0x7f57781c2d80, 0x7f57781c2d88, 0x7f57781c2d90, 0x7f57781c2d98, 0x7f57781c2da0, 0x7f57781c2de8, 0x0}, cfa = 0x7f57781c32f0, ra = 0x18663e6c8, lsda = 0x0, bases = {tbase = 0x0, dbase = 0x0, func = 0x7f578cb5ccaf}, flags = 13835058055282163712, version = 0, args_size = 0, by_value = '\000' <repeats 17 times>} code = <optimized out> #3 0x00007f578cc345b6 in __GI___backtrace (array=array@entry=0x7f57781c2200, size=size@entry=200) at ../sysdeps/x86_64/backtrace.c:109 arg = {array = 0x7f57781c2200, cfa = 140013653994816, cnt = 3, size = 200} once = 6 #4 0x00007f578d765f42 in _gf_msg_backtrace_nomem (level=level@entry=GF_LOG_ALERT, stacksize=stacksize@entry=200) at logging.c:1094 this = <optimized out> ctx = 0x7f578e58c010 array = {0x7f578d765f42 <_gf_msg_backtrace_nomem+146>, 0x7f578d76f494 <gf_print_trace+804>, 0x7f578cb5ccb0 <__restore_rt>, 0x0, 0x0, 0x0, 0xffffffffffffffff, 0x20, 0x20000000000000, 0x0, 0x0, 0x7f578d8011a8, 0x100000022, 0x21, 0x7f578d8011ab, 0x0 <repeats 13 times>, 0x3000000018, 0x7f57781c2840, 0x7f57781c2770, 0x0, 0x7f578cefe2e0 <__funlockfile>, 0x7f578e5d01b0, 0x0 <repeats 37 times>, 0x7f578cbe3341 <__strftime_internal+5905>, 0x0, 0x1ff5, 0x6, 0x900000000, 0x7f57781c26e0, 0x7f57781c24b0, 0x7f578cee5040 <_nl_C_LC_TIME>, 0x7f57781c2660, 0x7f578e5905c0, 0x7f57781c24d0, 0x7f57781c24d0, 0xffff80a887e3db31, 0x7f57781c24cf, 0x0, 0x0, 0x0, 0x393400000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f5780000020, 0x2010, 0x2000, 0x7f5780007130, 0x7f578cca7eab, 0x7f57800070eb, 0x7f578cee5400 <_nl_C_locobj>, 0x8, 0x7f578cca5e8c, 0x1, 0x20000, 0x1000, 0x1ffc, 0x0, 0x7f5780006e80, 0x7f578d8011a8, 0x7f57781c2768, 0x22, 0x7f57800070d0, 0x2010, 0x7f5780000020, 0x7f57800090e0, 0x58, 0x2010, 0x7f578cba767c <_int_realloc+284>, 0x7f57781c2650, 0x7f57781c2640, 0x7f5780000020, 0x7f57800070e0, 0x100000000, 0x60, 0x58, 0x7f578cba8ee9 <__GI___libc_realloc+217>, 0x7f5700000000, 0x7f5780006e80, 0x7f57781c2680, 0x7f578d8011a8, 0x7f57781c2768, 0x7f5780006e80, 0x0, 0x7f578cc24cdd <__libc_send+141>, 0x7f57781c2768, 0x7f5780006e80, 0x1, 0x7f578d8011a8, 0x7f57781c2768, 0x22, 0x14, 0x7f578cc1d9b9 <__GI___vsyslog_chk+649>, 0x7f57800070e0, 0x7f57781c26b6, 0x57, 0x590a50b9, 0x7f57781c2cff, 0x7f578cee5fe0 <_IO_strn_jumps>, 0x0, 0x7f578d800325, 0x7f57800070e0, 0x0, 0x27, 0x7f578d800325, 0x3200000031, 0x300000015, 0x7500000004, 0x7a00000003, 0xffffffff00000000, 0x0, 0x7f578e5905c0, 0x7f578e5d01b0, 0x2, 0x7f578e58c010, 0x7f5780006270, 0x2, 0x7f578e58c2b8, 0xb, 0xbfff, 0x7f578cc1dda2 <__syslog_chk+130>, 0x7f57781c2840, 0x3000000020, 0x7f57781c2840, 0x7f57781c2780, 0xb, 0x7f5780006270, 0x7f578e58c2b8, 0x7f5780006270, 0x7f578943d700, 0x6672657473756c67, 0x0, 0x7f578cb9ef13 <_IO_new_file_write+67>, 0x7f578d7fd396, 0x7f578e5d01b0, 0x21, 0x7f578de63000, 0x7f578e58c2b8, 0x7f578cba03ec <_IO_new_do_write+124>, 0x0, 0x7f578d78fa01 <__glusterfs_this_location+17>, 0x7f5780006270, 0x7f5780006270, 0x7f57781c2900, 0x7f578d78b6c5 <__gf_free+469>, 0x7f578e5d01b0, 0x7f578cb93e92 <__GI__IO_fflush+114>, 0x7f578e58c010, 0x7f57781c2880} bt_size = 0 fd = -1 #5 0x00007f578d76f494 in gf_print_trace (signum=11, ctx=0x7f578e58c010) at common-utils.c:737 msg = "signal received: 11\000gluster.com/glusterfs.git", '\000' <repeats 978 times> timestr = "2017-05-03 21:50:49", '\000' <repeats 44 times> stack = <optimized out> #6 <signal handler called> No locals. #7 0x000000018663e6c8 in ?? () No symbol table info available. #8 0x0000000000000000 in ?? () No symbol table info available. StacktraceSource: #0 x86_64_fallback_frame_state (context=0x7f57781c1f10, context=0x7f57781c1f10, fs=0x7f57781c2000) at ./md-unwind-support.h:58 [Error: ./md-unwind-support.h was not found in source tree] #1 uw_frame_state_for (context=context@entry=0x7f57781c1f10, fs=fs@entry=0x7f57781c2000) at ../../../src/libgcc/unwind-dw2.c:1253 [Error: ../../../src/libgcc/unwind-dw2.c was not found in source tree] #2 0x00007f5784fc06d8 in _Unwind_Backtrace (trace=0x7f578cc34440 <backtrace_helper>, trace_argument=0x7f57781c21c0) at ../../../src/libgcc/unwind.inc:290 [Error: ../../../src/libgcc/unwind.inc was not found in source tree] #3 0x00007f578cc345b6 in __GI___backtrace (array=array@entry=0x7f57781c2200, size=size@entry=200) at ../sysdeps/x86_64/backtrace.c:109 [Error: ../sysdeps/x86_64/backtrace.c was not found in source tree] #4 0x00007f578d765f42 in _gf_msg_backtrace_nomem (level=level@entry=GF_LOG_ALERT, stacksize=stacksize@entry=200) at logging.c:1094 1089: goto out; 1090: } 1091: if (level > ctx->log.loglevel || level == GF_LOG_NONE) 1092: goto out; 1093: 1094: bt_size = backtrace (array, ((stacksize <= 200)? stacksize : 200)); 1095: pthread_mutex_lock (&ctx->log.logfile_mutex); 1096: { 1097: fd = ctx->log.logfile? 1098: fileno (ctx->log.logfile) : 1099: fileno (stderr); #5 0x00007f578d76f494 in gf_print_trace (signum=11, ctx=0x7f578e58c010) at common-utils.c:737 732: gf_msg_plain_nomem (GF_LOG_ALERT, "time of crash: "); 733: gf_msg_plain_nomem (GF_LOG_ALERT, timestr); 734: } 735: 736: gf_dump_config_flags (); 737: gf_msg_backtrace_nomem (GF_LOG_ALERT, 200); 738: sprintf (msg, "---------"); 739: gf_msg_plain_nomem (GF_LOG_ALERT, msg); 740: 741: /* Send a signal to terminate the process */ 742: signal (signum, SIG_DFL); #6 <signal handler called> #7 0x000000018663e6c8 in ?? () #8 0x0000000000000000 in ?? () StacktraceTop: ?? () ?? () ThreadStacktrace: . Thread 7 (Thread 0x7f57863d2700 (LWP 1541)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 No locals. #1 0x00007f578899c733 in hooks_worker (args=<optimized out>) at glusterd-hooks.c:530 conf = <optimized out> hooks_priv = 0x7f578e6177d0 stub = <optimized out> #2 0x00007f578cef6184 in start_thread (arg=0x7f57863d2700) at pthread_create.c:312 __res = <optimized out> pd = 0x7f57863d2700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140013891036928, 6694029262299455749, 0, 0, 140013891037632, 140013891036928, -6752315998223867643, -6752336223035465467}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> __PRETTY_FUNCTION__ = "start_thread" #3 0x00007f578cc23bed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. . Thread 6 (Thread 0x7f5785bd1700 (LWP 1542)): #0 0x00007f578cc24283 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f578d7be3d0 in event_dispatch_epoll_worker (data=0x7f578e5d4c40) at event-epoll.c:664 event = {events = 1073741827, data = {ptr = 0x100000002, fd = 2, u32 = 2, u64 = 4294967298}} ret = <optimized out> ev_data = 0x7f578e5d4c40 event_pool = 0x7f578e5c2e10 myindex = 1 timetodie = 0 __FUNCTION__ = "event_dispatch_epoll_worker" #2 0x00007f578cef6184 in start_thread (arg=0x7f5785bd1700) at pthread_create.c:312 __res = <optimized out> pd = 0x7f5785bd1700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140013882644224, 6694029262299455749, 0, 0, 140013882644928, 140013882644224, -6752317098272366331, -6752336223035465467}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> __PRETTY_FUNCTION__ = "start_thread" #3 0x00007f578cc23bed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. . Thread 5 (Thread 0x7f578ac40700 (LWP 1097)): #0 0x00007f578cefdb9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f578d7728d6 in gf_timer_proc (data=0x7f578e5d09b0) at timer.c:176 now = 52662157751 now_ts = {tv_sec = 52, tv_nsec = 662157751} reg = 0x7f578e5d09b0 sleepts = {tv_sec = 1, tv_nsec = 0} event = 0x7f578e5d07c0 old_THIS = <optimized out> #2 0x00007f578cef6184 in start_thread (arg=0x7f578ac40700) at pthread_create.c:312 __res = <optimized out> pd = 0x7f578ac40700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140013966984960, 6694029262299455749, 0, 0, 140013966985664, 140013966984960, -6752323410263678715, -6752336223035465467}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> __PRETTY_FUNCTION__ = "start_thread" #3 0x00007f578cc23bed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. . Thread 4 (Thread 0x7f5789c3e700 (LWP 1099)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 No locals. #1 0x00007f578d79d678 in syncenv_task (proc=proc@entry=0x7f578e5d0ce0) at syncop.c:603 env = 0x7f578e5d0ce0 task = 0x0 sleep_till = {tv_sec = 1493848849, tv_nsec = 0} ret = <optimized out> #2 0x00007f578d79e4b0 in syncenv_processor (thdata=0x7f578e5d0ce0) at syncop.c:695 env = 0x7f578e5d0ce0 proc = 0x7f578e5d0ce0 task = <optimized out> #3 0x00007f578cef6184 in start_thread (arg=0x7f5789c3e700) at pthread_create.c:312 __res = <optimized out> pd = 0x7f5789c3e700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140013950199552, 6694029262299455749, 0, 0, 140013950200256, 140013950199552, -6752325608213192443, -6752336223035465467}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> __PRETTY_FUNCTION__ = "start_thread" #4 0x00007f578cc23bed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. . Thread 3 (Thread 0x7f578a43f700 (LWP 1098)): #0 0x00007f578cefe0c1 in do_sigwait (sig=0x7f578a43ee2c, set=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:60 __arg4 = 8 __arg2 = 0 _a3 = 0 _a1 = 140013958590000 resultvar = <optimized out> __arg3 = 0 __arg1 = 140013958590000 _a4 = 8 _a2 = 0 ret = <optimized out> tmpset = {__val = {0 <repeats 16 times>}} #1 __sigwait (set=0x7f578a43ee30, sig=0x7f578a43ee2c) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:97 oldtype = 0 result = 128 #2 0x00007f578dc5abfb in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2094 set = {__val = {18947, 0 <repeats 15 times>}} ret = <optimized out> sig = 0 #3 0x00007f578cef6184 in start_thread (arg=0x7f578a43f700) at pthread_create.c:312 __res = <optimized out> pd = 0x7f578a43f700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140013958592256, 6694029262299455749, 0, 0, 140013958592960, 140013958592256, -6752324510312177403, -6752336223035465467}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> __PRETTY_FUNCTION__ = "start_thread" #4 0x00007f578cc23bed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 No locals. . Thread 2 (Thread 0x7f578de57780 (LWP 1077)): #0 0x00007f578cef765b in pthread_join (threadid=140013882644224, thread_return=thread_return@entry=0x0) at pthread_join.c:92 _tid = 1542 _buffer = {__routine = 0x7f578cef7590 <cleanup>, __arg = 0x7f5785bd1d28, __canceltype = 0, __prev = 0x0} oldtype = 0 pd = 0x7f5785bd1700 self = 0x7f578de57780 result = 0 #1 0x00007f578d7be968 in event_dispatch_epoll (event_pool=0x7f578e5c2e10) at event-epoll.c:758 i = <optimized out> t_id = 140013882644224 pollercount = 1 ret = 0 ev_data = <optimized out> __FUNCTION__ = "event_dispatch_epoll" #2 0x00007f578dc57ada in main (argc=5, argv=<optimized out>) at glusterfsd.c:2492 ctx = 0x7f578e58c010 ret = <optimized out> cmdlinestr = "/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0", '\000' <repeats 4017 times> cmd = 0x7f578e58c010 __FUNCTION__ = "main" . Thread 1 (Thread 0x7f578943d700 (LWP 1100)): #0 x86_64_fallback_frame_state (context=0x7f57781c1f10, context=0x7f57781c1f10, fs=0x7f57781c2000) at ./md-unwind-support.h:58 pc = 0x18663e6c8 <error: Cannot access memory at address 0x18663e6c8> sc = <optimized out> new_cfa = <optimized out> #1 uw_frame_state_for (context=context@entry=0x7f57781c1f10, fs=fs@entry=0x7f57781c2000) at ../../../src/libgcc/unwind-dw2.c:1253 fde = 0x0 cie = <optimized out> aug = <optimized out> insn = <optimized out> end = <optimized out> #2 0x00007f5784fc06d8 in _Unwind_Backtrace (trace=0x7f578cc34440 <backtrace_helper>, trace_argument=0x7f57781c21c0) at ../../../src/libgcc/unwind.inc:290 fs = {regs = {reg = {{loc = {reg = 0, offset = 0, exp = 0x0}, how = REG_UNSAVED} <repeats 18 times>}, prev = 0x0, cfa_offset = 0, cfa_reg = 0, cfa_exp = 0x0, cfa_how = CFA_UNSET}, pc = 0x0, personality = 0x0, data_align = 0, code_align = 0, retaddr_column = 0, fde_encoding = 0 '\000', lsda_encoding = 0 '\000', saw_z = 0 '\000', signal_frame = 0 '\000', eh_ptr = 0x0} context = {reg = {0x7f57781c2dd0, 0x7f57781c2dc8, 0x7f57781c2dd8, 0x7f57781c2dc0, 0x7f57781c2db0, 0x7f57781c2da8, 0x7f57781c2db8, 0x7f57781c2de0, 0x7f57781c2d68, 0x7f57781c2d70, 0x7f57781c2d78, 0x7f57781c2d80, 0x7f57781c2d88, 0x7f57781c2d90, 0x7f57781c2d98, 0x7f57781c2da0, 0x7f57781c2de8, 0x0}, cfa = 0x7f57781c32f0, ra = 0x18663e6c8, lsda = 0x0, bases = {tbase = 0x0, dbase = 0x0, func = 0x7f578cb5ccaf}, flags = 13835058055282163712, version = 0, args_size = 0, by_value = '\000' <repeats 17 times>} code = <optimized out> #3 0x00007f578cc345b6 in __GI___backtrace (array=array@entry=0x7f57781c2200, size=size@entry=200) at ../sysdeps/x86_64/backtrace.c:109 arg = {array = 0x7f57781c2200, cfa = 140013653994816, cnt = 3, size = 200} once = 6 #4 0x00007f578d765f42 in _gf_msg_backtrace_nomem (level=level@entry=GF_LOG_ALERT, stacksize=stacksize@entry=200) at logging.c:1094 this = <optimized out> ctx = 0x7f578e58c010 array = {0x7f578d765f42 <_gf_msg_backtrace_nomem+146>, 0x7f578d76f494 <gf_print_trace+804>, 0x7f578cb5ccb0 <__restore_rt>, 0x0, 0x0, 0x0, 0xffffffffffffffff, 0x20, 0x20000000000000, 0x0, 0x0, 0x7f578d8011a8, 0x100000022, 0x21, 0x7f578d8011ab, 0x0 <repeats 13 times>, 0x3000000018, 0x7f57781c2840, 0x7f57781c2770, 0x0, 0x7f578cefe2e0 <__funlockfile>, 0x7f578e5d01b0, 0x0 <repeats 37 times>, 0x7f578cbe3341 <__strftime_internal+5905>, 0x0, 0x1ff5, 0x6, 0x900000000, 0x7f57781c26e0, 0x7f57781c24b0, 0x7f578cee5040 <_nl_C_LC_TIME>, 0x7f57781c2660, 0x7f578e5905c0, 0x7f57781c24d0, 0x7f57781c24d0, 0xffff80a887e3db31, 0x7f57781c24cf, 0x0, 0x0, 0x0, 0x393400000000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f5780000020, 0x2010, 0x2000, 0x7f5780007130, 0x7f578cca7eab, 0x7f57800070eb, 0x7f578cee5400 <_nl_C_locobj>, 0x8, 0x7f578cca5e8c, 0x1, 0x20000, 0x1000, 0x1ffc, 0x0, 0x7f5780006e80, 0x7f578d8011a8, 0x7f57781c2768, 0x22, 0x7f57800070d0, 0x2010, 0x7f5780000020, 0x7f57800090e0, 0x58, 0x2010, 0x7f578cba767c <_int_realloc+284>, 0x7f57781c2650, 0x7f57781c2640, 0x7f5780000020, 0x7f57800070e0, 0x100000000, 0x60, 0x58, 0x7f578cba8ee9 <__GI___libc_realloc+217>, 0x7f5700000000, 0x7f5780006e80, 0x7f57781c2680, 0x7f578d8011a8, 0x7f57781c2768, 0x7f5780006e80, 0x0, 0x7f578cc24cdd <__libc_send+141>, 0x7f57781c2768, 0x7f5780006e80, 0x1, 0x7f578d8011a8, 0x7f57781c2768, 0x22, 0x14, 0x7f578cc1d9b9 <__GI___vsyslog_chk+649>, 0x7f57800070e0, 0x7f57781c26b6, 0x57, 0x590a50b9, 0x7f57781c2cff, 0x7f578cee5fe0 <_IO_strn_jumps>, 0x0, 0x7f578d800325, 0x7f57800070e0, 0x0, 0x27, 0x7f578d800325, 0x3200000031, 0x300000015, 0x7500000004, 0x7a00000003, 0xffffffff00000000, 0x0, 0x7f578e5905c0, 0x7f578e5d01b0, 0x2, 0x7f578e58c010, 0x7f5780006270, 0x2, 0x7f578e58c2b8, 0xb, 0xbfff, 0x7f578cc1dda2 <__syslog_chk+130>, 0x7f57781c2840, 0x3000000020, 0x7f57781c2840, 0x7f57781c2780, 0xb, 0x7f5780006270, 0x7f578e58c2b8, 0x7f5780006270, 0x7f578943d700, 0x6672657473756c67, 0x0, 0x7f578cb9ef13 <_IO_new_file_write+67>, 0x7f578d7fd396, 0x7f578e5d01b0, 0x21, 0x7f578de63000, 0x7f578e58c2b8, 0x7f578cba03ec <_IO_new_do_write+124>, 0x0, 0x7f578d78fa01 <__glusterfs_this_location+17>, 0x7f5780006270, 0x7f5780006270, 0x7f57781c2900, 0x7f578d78b6c5 <__gf_free+469>, 0x7f578e5d01b0, 0x7f578cb93e92 <__GI__IO_fflush+114>, 0x7f578e58c010, 0x7f57781c2880} bt_size = 0 fd = -1 #5 0x00007f578d76f494 in gf_print_trace (signum=11, ctx=0x7f578e58c010) at common-utils.c:737 msg = "signal received: 11\000gluster.com/glusterfs.git", '\000' <repeats 978 times> timestr = "2017-05-03 21:50:49", '\000' <repeats 44 times> stack = <optimized out> #6 <signal handler called> No locals. #7 0x000000018663e6c8 in ?? () No symbol table info available. #8 0x0000000000000000 in ?? () No symbol table info available. Uname: Linux 3.13.0-117-generic x86_64 UserGroups:
Created attachment 1276039 [details] retrace apport-retrace -v -R -p glusterfs-dbg -S system /var/crash/_usr_sbin_glusterfsd.0.crash -o /var/crash/_usr_sbin_glusterfsd.0.crash.retrace
Created attachment 1276040 [details] apport crash
Created attachment 1276041 [details] Core File
This is the tool for resolving debugging symbols: https://wiki.ubuntu.com/DebuggingProgramCrash#Using_apport-retrace
I try recreating the issue locally with 2 ubuntu servers, and I was not able to recreate the issue locally. Repro steps 1. Install gluster on Ubuntu 14.04 2. sudo gluster peer probe 192.168.122.159 I am not seeing any issue while peer probing...command getting executed succesfully. I have few queries regarding the info you have provided.. 1. Problematic peer entry: 2. cat /var/lib/glusterd/peers/ip-10-0-50-25.us-west-1.compute.internal 3. uuid=00000000-0000-0000-0000-000000000000 4. state=0 5. hostname1=ip-10-0-50-25.us-west-1.compute.internal Line 2 shows file name as "ip-10-0-50-25.us-west-1.compute.internal" inside peers directory, however glusterd generates file and named as per uuid. In your case filename is similar to hostname which is wrong. Could you please let me know whether it is glusterd generated file or manually some changes have been done.? Answer to your query.. There's another issue which may be related. I noticed that glusterd.info was not self-populating. As a workaround I issue 'gluster pool list' which triggers glusterd to generate and store a UUID: glusterd.info file gets generated whenever we do probe and create volume for the first time. It contains UUID of local node which is used by glusterd for validation purpose.
(In reply to Gaurav Yadav from comment #7) > I try recreating the issue locally with 2 ubuntu servers, and I was not able > to recreate the issue locally. > > Repro steps > 1. Install gluster on Ubuntu 14.04 > 2. sudo gluster peer probe 192.168.122.159 > > I am not seeing any issue while peer probing...command getting executed > succesfully. > > > I have few queries regarding the info you have provided.. > > 1. Problematic peer entry: > 2. cat /var/lib/glusterd/peers/ip-10-0-50-25.us-west-1.compute.internal > 3. uuid=00000000-0000-0000-0000-000000000000 > 4. state=0 > 5. hostname1=ip-10-0-50-25.us-west-1.compute.internal > > Line 2 shows file name as "ip-10-0-50-25.us-west-1.compute.internal" inside > peers directory, however glusterd generates file and named as per uuid. In > your case filename is similar to hostname which is wrong. > > Could you please let me know whether it is glusterd generated file or > manually some changes have been done.? glusterd generates this file. In that example I used a hostname, but the same is true with an IP address. 1. create new instance 2. echo "exec /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0" > /etc/init/glusterfs-server.override 3. install glusterfs-server 4. ln -sf /lib/init/upstart-job /etc/init.d/glusterfs-server 5. /usr/sbin/gluster pool list 6. /usr/sbin/gluster --loglevel=TRACE volume info 7. /usr/sbin/gluster volume info 8. /usr/sbin/gluster --log-level=TRACE pool list 9. /usr/sbin/gluster --log-level=TRACE peer probe 10.0.50.215 stdout=Connection failed. Please check if gluster daemon is operational. stderr= rc=1 glusterd segfaults glusterd logs at [1] gluster cli logs at [2] 10. cat /var/lib/glusterd/peers/10.0.50.215 uuid=00000000-0000-0000-0000-000000000000 state=0 hostname1=10.0.50.215 > > > Answer to your query.. > There's another issue which may be related. I noticed that glusterd.info was > not self-populating. As a workaround I issue 'gluster pool list' which > triggers glusterd to generate and store a UUID: > > glusterd.info file gets generated whenever we do probe and create volume for > the first time. It contains UUID of local node which is used by glusterd for > validation purpose. Thank you for this info. [1] cat /var/log/glusterfs/cli.log [2017-05-04 15:54:37.044769] I [cli.c:728:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.11 [2017-05-04 15:54:37.102299] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-04 15:54:37.102339] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:54:37.109025] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2017-05-04 15:55:02.509217] I [cli.c:728:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.11 [2017-05-04 15:55:02.566970] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-04 15:55:02.567011] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:55:02.567602] I [cli-rpc-ops.c:817:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2017-05-04 15:55:02.567645] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2017-05-04 15:55:02.796716] I [cli.c:728:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.11 [2017-05-04 15:55:02.796739] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to glusterd using default socket [2017-05-04 15:55:02.796761] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-04 15:55:02.796767] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-04 15:55:02.796776] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-04 15:55:02.798821] D [socket.c:3887:socket_init] 0-glusterfs: disabling nodelay [2017-05-04 15:55:02.798835] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-04 15:55:02.798841] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-04 15:55:02.798845] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-04 15:55:02.798849] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-04 15:55:02.798856] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-04 15:55:02.798863] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f559a168d40, state=0 gen=0 sock=-1 [2017-05-04 15:55:02.798870] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/glusterd.socket [2017-05-04 15:55:02.798879] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-04 15:55:02.798988] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-04 15:55:02.798997] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-04 15:55:02.799003] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-04 15:55:02.799019] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-04 15:55:02.799025] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-04 15:55:02.799029] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-04 15:55:02.799033] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-04 15:55:02.799038] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-04 15:55:02.799042] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f559a19c3f0, state=0 gen=0 sock=-1 [2017-05-04 15:55:02.799047] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-05-04 15:55:02.799055] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-04 15:55:02.799075] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799094] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799109] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799114] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799119] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799130] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799136] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799142] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799149] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799156] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799163] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799169] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799174] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799179] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799184] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799189] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799194] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.799199] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854105] D [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 [2017-05-04 15:55:02.854146] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854157] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854177] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854183] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854195] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854204] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854217] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854224] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854229] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854237] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854242] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854247] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854259] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854266] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854272] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854276] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854280] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854284] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854292] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854298] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854303] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854308] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854313] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854317] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854322] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854326] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854330] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854335] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854340] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854344] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854348] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854352] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854362] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854370] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854375] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854379] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854385] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854390] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854395] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854404] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854410] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854415] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854419] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854424] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854428] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.854484] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-04 15:55:02.854506] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-04 15:55:02.854518] T [cli-quotad-client.c:89:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-04 15:55:02.854523] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:55:02.854674] T [rpc-clnt.c:1474:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-05-04 15:55:02.854690] T [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 72, payload: 8, rpc hdr: 64 [2017-05-04 15:55:02.855348] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f5598d00992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f559867569e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7f5598671c5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f55986724f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f559866e6b3] ))))) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-05-04 15:55:02.855362] T [cli-quotad-client.c:95:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-05-04 15:55:02.855368] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-04 15:55:02.855461] T [socket.c:2945:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f5598d00992] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so(+0x9b8b)[0x7f5595db0b8b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x33f)[0x7f5598672acf] (--> /usr/sbin/gluster(cli_submit_request+0x149)[0x7f55991f7189] (--> /usr/sbin/gluster(cli_cmd_submit+0x93)[0x7f55991f9013] ))))) 0-glusterfs: connect () called on transport already connected [2017-05-04 15:55:02.856424] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 3) to rpc-transport (glusterfs) [2017-05-04 15:55:02.856437] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-05-04 15:55:02.856457] T [rpc-clnt.c:688:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 3) from rpc-transport (glusterfs) [2017-05-04 15:55:02.856466] D [cli-rpc-ops.c:419:gf_cli_list_friends_cbk] 0-cli: Received resp to list: 0 [2017-05-04 15:55:02.856497] D [cli-cmd.c:390:cli_cmd_submit] 0-cli: Returning 0 [2017-05-04 15:55:02.856503] D [cli-rpc-ops.c:4151:gf_cli_list_friends] 0-cli: Returning 0 [2017-05-04 15:55:02.856507] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2017-05-04 15:55:02.882825] I [cli.c:728:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.11 [2017-05-04 15:55:02.882845] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to glusterd using default socket [2017-05-04 15:55:02.882869] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-04 15:55:02.882875] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-04 15:55:02.882883] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-04 15:55:02.884833] D [socket.c:3887:socket_init] 0-glusterfs: disabling nodelay [2017-05-04 15:55:02.884845] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-04 15:55:02.884851] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-04 15:55:02.884855] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-04 15:55:02.884858] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-04 15:55:02.884866] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-04 15:55:02.884873] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f7711480d40, state=0 gen=0 sock=-1 [2017-05-04 15:55:02.884879] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/glusterd.socket [2017-05-04 15:55:02.884889] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-04 15:55:02.885060] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-04 15:55:02.885070] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-04 15:55:02.885077] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-04 15:55:02.885093] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-04 15:55:02.885099] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-04 15:55:02.885103] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-04 15:55:02.885107] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-04 15:55:02.885112] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-04 15:55:02.885116] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f77114b43f0, state=0 gen=0 sock=-1 [2017-05-04 15:55:02.885121] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-05-04 15:55:02.885129] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-04 15:55:02.885149] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885167] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885173] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885178] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885183] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885194] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885200] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885206] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885214] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885221] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885228] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885234] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885239] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885253] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885259] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885264] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885269] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.885274] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941103] D [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 [2017-05-04 15:55:02.941143] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941153] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941173] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941179] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941191] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941200] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941213] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941220] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941225] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941233] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941238] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941243] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941255] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941262] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941268] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941272] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941276] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941281] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941287] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941293] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941299] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941303] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941308] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941313] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941317] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941322] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941326] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941331] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941335] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941340] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941344] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941348] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941353] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941360] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941365] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941369] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941375] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941380] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941384] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941394] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941399] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941404] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941414] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941419] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941423] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-04 15:55:02.941479] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-04 15:55:02.941501] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-04 15:55:02.941512] T [cli-quotad-client.c:89:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-04 15:55:02.941517] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:55:02.941724] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f770fe6b69e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7f770fe67c5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f770fe684f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f770fe646b3] ))))) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-05-04 15:55:02.941738] T [cli-quotad-client.c:95:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-05-04 15:55:02.941745] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-04 15:55:02.941805] T [rpc-clnt.c:1474:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-05-04 15:55:02.941819] T [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 160, payload: 96, rpc hdr: 64 [2017-05-04 15:55:02.941958] T [socket.c:2945:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so(+0x9b8b)[0x7f770d5a6b8b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x33f)[0x7f770fe68acf] (--> /usr/sbin/gluster(cli_submit_request+0x149)[0x7f77109ed189] (--> /usr/sbin/gluster(cli_cmd_submit+0x93)[0x7f77109ef013] ))))) 0-glusterfs: connect () called on transport already connected [2017-05-04 15:55:02.943162] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to rpc-transport (glusterfs) [2017-05-04 15:55:02.944901] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-05-04 15:55:04.458391] D [socket.c:559:__socket_rwv] 0-glusterfs: EOF on socket [2017-05-04 15:55:04.458417] W [socket.c:590:__socket_rwv] 0-glusterfs: readv on /var/run/glusterd.socket failed (No data available) [2017-05-04 15:55:04.458427] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:55:04.458550] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f770fe6b69e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7f770fe67c5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f770fe684f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f770fe646b3] ))))) 0-: /var/run/glusterd.socket: ping timer event already removed [2017-05-04 15:55:04.458632] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f770fe6649e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f770fe665ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f770fe67c84] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f770fe684f4] ))))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(1)) called at 2017-05-04 15:55:02.943160 (xid=0x1) [2017-05-04 15:55:04.458655] T [cli.c:272:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2] cat /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2017-05-04 15:54:33.967651] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0) [2017-05-04 15:54:33.968085] T [MSGID: 0] [glusterfsd.c:573:create_fuse_mount] 0-glusterfsd: mount point not found, not a client process [2017-05-04 15:54:33.968553] D [MSGID: 0] [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/glusterd.pid updated with pid 13120 [2017-05-04 15:54:33.968584] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-04 15:54:33.968687] D [MSGID: 0] [glusterfsd.c:660:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2017-05-04 15:54:33.968797] T [MSGID: 0] [graph.y:186:new_volume] 0-parser: New node for 'management' [2017-05-04 15:54:33.968810] T [MSGID: 0] [xlator.c:193:xlator_dynload] 0-xlator: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so [2017-05-04 15:54:33.974984] T [MSGID: 0] [xlator.c:244:xlator_dynload] 0-xlator: dlsym(reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so: undefined symbol: reconfigure -- neglecting [2017-05-04 15:54:33.975006] T [MSGID: 0] [graph.y:217:volume_type] 0-parser: Type:management:mgmt/glusterd [2017-05-04 15:54:33.975032] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:working-directory:/var/lib/glusterd [2017-05-04 15:54:33.975042] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport-type:socket,rdma [2017-05-04 15:54:33.975048] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-time:10 [2017-05-04 15:54:33.975054] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-interval:2 [2017-05-04 15:54:33.975060] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.read-fail-log:off [2017-05-04 15:54:33.975065] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:ping-timeout:0 [2017-05-04 15:54:33.975071] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:event-threads:1 [2017-05-04 15:54:33.975076] T [MSGID: 0] [graph.y:323:volume_end] 0-parser: end:management [2017-05-04 15:54:33.975143] I [MSGID: 106478] [glusterd.c:1381:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-05-04 15:54:33.975158] I [MSGID: 106479] [glusterd.c:1430:init] 0-management: Using /var/lib/glusterd as working directory [2017-05-04 15:54:33.975325] D [MSGID: 0] [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 128 [2017-05-04 15:54:33.975340] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 64 [2017-05-04 15:54:33.975459] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-04 15:54:33.975469] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-04 15:54:33.975473] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-04 15:54:33.975478] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-04 15:54:33.975481] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-04 15:54:33.975488] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-04 15:54:33.975502] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-04 15:54:33.977568] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.listen-backlog 128' [2017-05-04 15:54:33.977589] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-04 15:54:33.977600] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-04 15:54:33.977613] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-04 15:54:33.977620] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-04 15:54:33.977625] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-04 15:54:33.977628] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-04 15:54:33.977635] D [name.c:584:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2017-05-04 15:54:33.977648] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 10 [2017-05-04 15:54:33.977725] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so [2017-05-04 15:54:33.984039] D [rpc-transport.c:321:rpc_transport_load] 0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so: undefined symbol: reconfigure [2017-05-04 15:54:33.984081] T [MSGID: 0] [rdma.c:4560:gf_rdma_options_init] 0-rpc-transport/rdma: rdma.management: defaulting MTU to '2048' [2017-05-04 15:54:33.984120] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-05-04 15:54:33.984129] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2017-05-04 15:54:33.984133] W [rpc-transport.c:354:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-05-04 15:54:33.984167] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-05-04 15:54:33.984175] E [MSGID: 106243] [glusterd.c:1654:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-05-04 15:54:33.984180] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2017-05-04 15:54:33.984185] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2017-05-04 15:54:33.984189] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2017-05-04 15:54:33.984193] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2017-05-04 15:54:33.984201] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2017-05-04 15:54:33.984205] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2017-05-04 15:54:33.984209] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2017-05-04 15:54:33.984218] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 8 [2017-05-04 15:54:33.984233] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-04 15:54:33.984238] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-04 15:54:33.984242] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-04 15:54:33.984246] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-04 15:54:33.984249] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-04 15:54:33.984253] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-04 15:54:33.984266] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-04 15:54:33.984289] D [socket.c:3887:socket_init] 0-socket.management: disabling nodelay [2017-05-04 15:54:33.984294] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-04 15:54:33.984299] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-04 15:54:33.984303] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-04 15:54:33.984307] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-04 15:54:33.984336] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2017-05-04 15:54:33.984343] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2017-05-04 15:54:33.984362] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-04 15:54:33.984369] D [MSGID: 0] [glusterd.c:1722:init] 0-management: cannot get run-with-valgrind value [2017-05-04 15:54:34.115026] D [MSGID: 0] [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 [2017-05-04 15:54:34.115121] D [MSGID: 0] [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 [2017-05-04 15:54:36.932822] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-04 15:54:36.932848] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-04 15:54:36.932856] D [MSGID: 0] [glusterd-store.c:1985:glusterd_retrieve_op_version] 0-management: Unable to get store handle! [2017-05-04 15:54:36.932863] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-04 15:54:36.932869] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-04 15:54:36.932873] D [MSGID: 0] [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get storehandle! [2017-05-04 15:54:36.932878] D [MSGID: 0] [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 [2017-05-04 15:54:36.932883] I [MSGID: 106514] [glusterd-store.c:2120:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30800 [2017-05-04 15:54:36.932907] D [MSGID: 0] [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: Returning with 0 [2017-05-04 15:54:36.932939] D [MSGID: 0] [glusterd-store.c:4257:glusterd_store_retrieve_peers] 0-management: Returning with 0 [2017-05-04 15:54:36.932958] I [MSGID: 106194] [glusterd-store.c:3628:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-05-04 15:54:36.932965] T [MSGID: 0] [glusterd-store.c:3687:glusterd_store_retrieve_missed_snaps_list] 0-management: Returning with 0 [2017-05-04 15:54:36.932970] D [MSGID: 0] [glusterd-store.c:3748:glusterd_store_retrieve_snaps] 0-management: Returning with 0 [2017-05-04 15:54:36.932975] T [MSGID: 0] [glusterd-store.c:4425:glusterd_resolve_all_bricks] 0-management: Returning with 0 [2017-05-04 15:54:36.932998] T [MSGID: 0] [glusterd-store.c:4379:glusterd_snap_cleanup] 0-management: Returning with 0 [2017-05-04 15:54:36.933003] T [MSGID: 0] [glusterd-store.c:4311:glusterd_recreate_all_snap_brick_mounts] 0-management: Returning with 0 [2017-05-04 15:54:36.933008] D [MSGID: 0] [glusterd-store.c:4478:glusterd_restore] 0-management: Returning 0 [2017-05-04 15:54:36.933044] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] [2017-05-04 15:54:36.933051] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-04 15:54:36.933056] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-04 15:54:36.933148] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fa21389b800, remaining locks=0 [2017-05-04 15:54:36.935326] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-04 15:54:36.935368] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-04 15:54:36.939235] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-04 15:54:36.939277] D [MSGID: 0] [options.c:1214:xlator_option_init_int32] 0-management: option event-threads using set value 1 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-05-04 15:54:36.939682] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-04 15:54:37.102439] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2017-05-04 15:54:37.102456] T [rpcsvc.c:490:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, ProgVers: 2, Proc: 3) from rpc-transport (socket.management) [2017-05-04 15:54:37.102472] T [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2017-05-04 15:54:37.102478] T [rpcsvc.c:311:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterD svc cli - LIST_FRIENDS for [2017-05-04 15:54:37.102556] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-05-04 15:54:37.102575] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-04 15:54:37.102584] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-04 15:54:37.102589] D [MSGID: 0] [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get storehandle! [2017-05-04 15:54:37.102594] D [MSGID: 0] [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 [2017-05-04 15:54:37.102631] I [MSGID: 106477] [glusterd.c:187:glusterd_uuid_generate_save] 0-management: generated UUID: 8d317e96-3df6-4995-8b9b-694ccf780ce8 [2017-05-04 15:54:37.105157] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-04 15:54:37.105208] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-04 15:54:37.105224] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-04 15:54:37.108905] T [rpcsvc.c:1313:rpcsvc_submit_generic] 0-rpc-service: Tx message: 156 [2017-05-04 15:54:37.108920] T [rpcsvc.c:859:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 180, payload: 156, rpc hdr: 24 [2017-05-04 15:54:37.108932] T [rpcsvc.c:1361:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 3) to rpc-transport (socket.management) [2017-05-04 15:54:37.108945] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fa1fc002280, remaining locks=0 [2017-05-04 15:54:37.108951] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fa1fc002280, remaining locks=0 [2017-05-04 15:54:37.111912] D [socket.c:559:__socket_rwv] 0-socket.management: EOF on socket [2017-05-04 15:54:37.111925] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:54:37.111940] T [socket.c:4267:fini] 0-socket.management: transport 0x7fa1fc000920 destroyed [2017-05-04 15:54:37.111948] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=2 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-04 15:55:02.567101] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2017-05-04 15:55:02.567118] T [rpcsvc.c:490:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, ProgVers: 2, Proc: 5) from rpc-transport (socket.management) [2017-05-04 15:55:02.567128] T [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2017-05-04 15:55:02.567134] T [rpcsvc.c:311:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterD svc cli - GET_VOLUME for [2017-05-04 15:55:02.567400] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-05-04 15:55:02.567419] T [rpcsvc.c:1313:rpcsvc_submit_generic] 0-rpc-service: Tx message: 36 [2017-05-04 15:55:02.567426] T [rpcsvc.c:859:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 60, payload: 36, rpc hdr: 24 [2017-05-04 15:55:02.567435] T [rpcsvc.c:1361:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 5) to rpc-transport (socket.management) [2017-05-04 15:55:02.567447] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fa1fc002280, remaining locks=0 [2017-05-04 15:55:02.567453] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fa1fc002280, remaining locks=0 [2017-05-04 15:55:02.567893] D [socket.c:559:__socket_rwv] 0-socket.management: EOF on socket [2017-05-04 15:55:02.567904] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:55:02.567912] T [socket.c:4267:fini] 0-socket.management: transport 0x7fa1fc000920 destroyed [2017-05-04 15:55:02.567919] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=2 from gen=4 to slot->gen=5, fd=7, slot->fd=7 [2017-05-04 15:55:02.855502] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2017-05-04 15:55:02.855518] T [rpcsvc.c:490:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, ProgVers: 2, Proc: 3) from rpc-transport (socket.management) [2017-05-04 15:55:02.855529] T [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2017-05-04 15:55:02.855535] T [rpcsvc.c:311:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterD svc cli - LIST_FRIENDS for [2017-05-04 15:55:02.856265] I [MSGID: 106487] [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-05-04 15:55:02.856293] T [rpcsvc.c:1313:rpcsvc_submit_generic] 0-rpc-service: Tx message: 156 [2017-05-04 15:55:02.856300] T [rpcsvc.c:859:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 180, payload: 156, rpc hdr: 24 [2017-05-04 15:55:02.856310] T [rpcsvc.c:1361:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 3) to rpc-transport (socket.management) [2017-05-04 15:55:02.856320] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fa1fc002280, remaining locks=0 [2017-05-04 15:55:02.856326] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fa1fc002280, remaining locks=0 [2017-05-04 15:55:02.856946] D [socket.c:559:__socket_rwv] 0-socket.management: EOF on socket [2017-05-04 15:55:02.856958] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-04 15:55:02.856967] T [socket.c:4267:fini] 0-socket.management: transport 0x7fa1fc000920 destroyed [2017-05-04 15:55:02.856974] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=2 from gen=7 to slot->gen=8, fd=7, slot->fd=7 [2017-05-04 15:55:02.942003] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2017-05-04 15:55:02.942020] T [rpcsvc.c:490:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, ProgVers: 2, Proc: 1) from rpc-transport (socket.management) [2017-05-04 15:55:02.942030] T [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2017-05-04 15:55:02.942035] T [rpcsvc.c:311:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterD svc cli - CLI_PROBE for [2017-05-04 15:55:02.942774] I [MSGID: 106487] [glusterd-handler.c:1241:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.0.50.215 24007 [2017-05-04 15:55:02.942794] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 [2017-05-04 15:55:02.942897] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 [2017-05-04 15:55:02.942936] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 [2017-05-04 15:55:02.942970] D [MSGID: 0] [common-utils.c:3357:gf_is_local_addr] 0-management: 10.0.50.215 is not local [2017-05-04 15:55:02.943003] D [MSGID: 0] [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.50.215 [2017-05-04 15:55:02.943019] D [MSGID: 0] [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.50.215 [2017-05-04 15:55:02.943025] D [MSGID: 0] [glusterd-peer-utils.c:247:glusterd_peerinfo_find] 0-management: Unable to find hostname: 10.0.50.215 [2017-05-04 15:55:02.943031] I [MSGID: 106129] [glusterd-handler.c:3670:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 10.0.50.215 (24007) [2017-05-04 15:55:02.943040] D [MSGID: 0] [glusterd-peer-utils.c:487:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2017-05-04 15:55:02.943045] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-04 15:55:02.943054] D [MSGID: 0] [glusterd-store.c:3943:glusterd_store_create_peer_dir] 0-glusterd: Returning with 0 [2017-05-04 15:55:02.945468] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-04 15:55:02.945516] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-04 15:55:02.945530] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-04 15:55:02.945541] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-04 15:55:02.945545] D [MSGID: 0] [glusterd-store.c:4072:glusterd_store_peer_write] 0-glusterd: Returning with 0 [2017-05-04 15:55:02.949254] D [MSGID: 0] [glusterd-store.c:4098:glusterd_store_perform_peer_store] 0-glusterd: Returning 0 [2017-05-04 15:55:02.949266] D [MSGID: 0] [glusterd-store.c:4119:glusterd_store_peerinfo] 0-glusterd: Returning with 0 [2017-05-04 15:55:02.949282] D [MSGID: 0] [glusterd-handler.c:3461:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2017-05-04 15:55:02.949302] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-05-04 15:55:02.949309] D [rpc-clnt.c:1058:rpc_clnt_connection_init] 0-management: setting ping-timeout to 0 [2017-05-04 15:55:02.949316] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-04 15:55:02.949341] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.tcp-user-timeout -1' [2017-05-04 15:55:02.949348] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-04 15:55:02.949354] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-04 15:55:02.949361] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option remote-port 24007' [2017-05-04 15:55:02.949373] D [socket.c:3938:socket_init] 0-management: Configued transport.tcp-user-timeout=-1 [2017-05-04 15:55:02.949381] D [socket.c:4021:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2017-05-04 15:55:02.949385] D [socket.c:4024:socket_init] 0-management: SSL support for glusterd is NOT enabled [2017-05-04 15:55:02.949389] D [socket.c:4041:socket_init] 0-management: using system polling thread [2017-05-04 15:55:02.949396] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-management: attempting reconnect [2017-05-04 15:55:02.949401] T [socket.c:2953:socket_connect] 0-management: connecting 0x7fa204005040, state=0 gen=0 sock=-1 [2017-05-04 15:55:02.949407] D [name.c:168:client_fill_address_family] 0-management: address-family not specified, marking it as unspec for getaddrinfo to resolve from (remote-host: 10.0.50.215) [2017-05-04 15:55:02.949413] T [MSGID: 0] [common-utils.c:281:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: 10.0.50.215 [2017-05-04 15:55:02.949670] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-10.0.50.215 (port-24007) for hostname: 10.0.50.215 and port: 24007 [2017-05-04 15:55:02.949679] D [socket.c:2899:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2017-05-04 15:55:02.949691] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 12 [2017-05-04 15:55:02.949696] T [socket.c:906:__socket_keepalive] 0-management: Keep-alive enabled for socket 12, interval 2, idle: 10, timeout: -1 [2017-05-04 15:55:02.949747] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 24007, higher: 24008 [2017-05-04 15:55:02.949753] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 32765, higher: -1 [2017-05-04 15:55:02.949758] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: -1, higher: -1 pending frames: frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2017-05-04 15:55:02 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.8.11
(In reply to Ben Werthmann from comment #8) > (In reply to Gaurav Yadav from comment #7) > > I try recreating the issue locally with 2 ubuntu servers, and I was not able > > to recreate the issue locally. > > > > Repro steps > > 1. Install gluster on Ubuntu 14.04 > > 2. sudo gluster peer probe 192.168.122.159 > > > > I am not seeing any issue while peer probing...command getting executed > > succesfully. > > > > > > I have few queries regarding the info you have provided.. > > > > 1. Problematic peer entry: > > 2. cat /var/lib/glusterd/peers/ip-10-0-50-25.us-west-1.compute.internal > > 3. uuid=00000000-0000-0000-0000-000000000000 > > 4. state=0 > > 5. hostname1=ip-10-0-50-25.us-west-1.compute.internal > > > > Line 2 shows file name as "ip-10-0-50-25.us-west-1.compute.internal" inside > > peers directory, however glusterd generates file and named as per uuid. In > > your case filename is similar to hostname which is wrong. > > > > Could you please let me know whether it is glusterd generated file or > > manually some changes have been done.? > > glusterd generates this file. In that example I used a hostname, but the > same is true with an IP address. > > 1. create new instance > 2. echo "exec /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE > --log-buf-size=0" > /etc/init/glusterfs-server.override > 3. install glusterfs-server > 4. ln -sf /lib/init/upstart-job /etc/init.d/glusterfs-server > 5. /usr/sbin/gluster pool list > 6. /usr/sbin/gluster --loglevel=TRACE volume info > 7. /usr/sbin/gluster volume info > 8. /usr/sbin/gluster --log-level=TRACE pool list > 9. /usr/sbin/gluster --log-level=TRACE peer probe 10.0.50.215 > stdout=Connection failed. Please check if gluster daemon is operational. > stderr= > rc=1 > > glusterd segfaults > glusterd logs at [1] > gluster cli logs at [2] > > 10. cat /var/lib/glusterd/peers/10.0.50.215 > uuid=00000000-0000-0000-0000-000000000000 > state=0 > hostname1=10.0.50.215 > > > > > > Answer to your query.. > > There's another issue which may be related. I noticed that glusterd.info was > > not self-populating. As a workaround I issue 'gluster pool list' which > > triggers glusterd to generate and store a UUID: > > > > glusterd.info file gets generated whenever we do probe and create volume for > > the first time. It contains UUID of local node which is used by glusterd for > > validation purpose. > > Thank you for this info. > > > > [1] cat /var/log/glusterfs/cli.log > [2017-05-04 15:54:37.044769] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:54:37.102299] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:54:37.102339] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:54:37.109025] I [input.c:31:cli_batch] 0-: Exiting with: 0 > [2017-05-04 15:55:02.509217] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:55:02.566970] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:55:02.567011] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.567602] I [cli-rpc-ops.c:817:gf_cli_get_volume_cbk] > 0-cli: Received resp to get vol: 0 > [2017-05-04 15:55:02.567645] I [input.c:31:cli_batch] 0-: Exiting with: 0 > [2017-05-04 15:55:02.796716] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:55:02.796739] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to > glusterd using default socket > [2017-05-04 15:55:02.796761] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.796767] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.796776] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.798821] D [socket.c:3887:socket_init] 0-glusterfs: > disabling nodelay > [2017-05-04 15:55:02.798835] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.798841] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.798845] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.798849] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.798856] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.798863] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f559a168d40, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.798870] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/glusterd.socket > [2017-05-04 15:55:02.798879] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.798988] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.798997] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.799003] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.799019] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.799025] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.799029] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.799033] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.799038] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.799042] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f559a19c3f0, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.799047] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/gluster/quotad.socket > [2017-05-04 15:55:02.799055] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.799075] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799094] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799109] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799114] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799119] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799130] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799136] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799142] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799149] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799156] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799163] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799169] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799174] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799179] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799184] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799189] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799194] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799199] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854105] D > [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 > [2017-05-04 15:55:02.854146] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854157] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854177] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854183] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854195] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854204] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854217] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854224] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854229] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854237] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854242] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854247] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854259] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854266] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854272] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854276] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854280] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854284] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854292] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854298] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854303] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854308] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854313] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854317] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854322] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854326] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854330] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854335] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854340] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854344] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854348] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854352] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854362] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854370] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854375] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854379] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854385] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854390] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854395] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854404] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854410] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854415] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854419] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854424] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854428] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854484] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:55:02.854506] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got > RPC_CLNT_CONNECT > [2017-05-04 15:55:02.854518] T [cli-quotad-client.c:89:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_CONNECT > [2017-05-04 15:55:02.854523] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.854674] T [rpc-clnt.c:1474:rpc_clnt_record] > 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: > [2017-05-04 15:55:02.854690] T > [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen > 72, payload: 8, rpc hdr: 64 > [2017-05-04 15:55:02.855348] D > [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f5598d00992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f559867569e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x5b)[0x7f5598671c5b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f55986724f4] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_transport_notify+0x23)[0x7f559866e6b3] ))))) 0-: > /var/run/gluster/quotad.socket: ping timer event already removed > [2017-05-04 15:55:02.855362] T [cli-quotad-client.c:95:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_DISCONNECT > [2017-05-04 15:55:02.855368] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 > [2017-05-04 15:55:02.855461] T [socket.c:2945:socket_connect] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f5598d00992] (--> > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket. > so(+0x9b8b)[0x7f5595db0b8b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_submit+0x33f)[0x7f5598672acf] (--> > /usr/sbin/gluster(cli_submit_request+0x149)[0x7f55991f7189] (--> > /usr/sbin/gluster(cli_cmd_submit+0x93)[0x7f55991f9013] ))))) 0-glusterfs: > connect () called on transport already connected > [2017-05-04 15:55:02.856424] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: > submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 3) to > rpc-transport (glusterfs) > [2017-05-04 15:55:02.856437] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] > 0-glusterfs: ping timeout is 0, returning > [2017-05-04 15:55:02.856457] T [rpc-clnt.c:688:rpc_clnt_reply_init] > 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI, > ProgVers: 2, Proc: 3) from rpc-transport (glusterfs) > [2017-05-04 15:55:02.856466] D [cli-rpc-ops.c:419:gf_cli_list_friends_cbk] > 0-cli: Received resp to list: 0 > [2017-05-04 15:55:02.856497] D [cli-cmd.c:390:cli_cmd_submit] 0-cli: > Returning 0 > [2017-05-04 15:55:02.856503] D [cli-rpc-ops.c:4151:gf_cli_list_friends] > 0-cli: Returning 0 > [2017-05-04 15:55:02.856507] I [input.c:31:cli_batch] 0-: Exiting with: 0 > [2017-05-04 15:55:02.882825] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:55:02.882845] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to > glusterd using default socket > [2017-05-04 15:55:02.882869] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.882875] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.882883] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.884833] D [socket.c:3887:socket_init] 0-glusterfs: > disabling nodelay > [2017-05-04 15:55:02.884845] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.884851] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.884855] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.884858] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.884866] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.884873] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f7711480d40, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.884879] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/glusterd.socket > [2017-05-04 15:55:02.884889] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.885060] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.885070] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.885077] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.885093] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.885099] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.885103] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.885107] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.885112] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.885116] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f77114b43f0, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.885121] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/gluster/quotad.socket > [2017-05-04 15:55:02.885129] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.885149] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885167] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885173] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885178] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885183] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885194] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885200] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885206] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885214] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885221] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885228] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885234] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885239] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885253] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885259] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885264] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885269] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885274] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941103] D > [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 > [2017-05-04 15:55:02.941143] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941153] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941173] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941179] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941191] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941200] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941213] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941220] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941225] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941233] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941238] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941243] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941255] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941262] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941268] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941272] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941276] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941281] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941287] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941293] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941299] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941303] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941308] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941313] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941317] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941322] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941326] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941331] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941335] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941340] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941344] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941348] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941353] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941360] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941365] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941369] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941375] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941380] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941384] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941394] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941399] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941404] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941414] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941419] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941423] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941479] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:55:02.941501] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got > RPC_CLNT_CONNECT > [2017-05-04 15:55:02.941512] T [cli-quotad-client.c:89:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_CONNECT > [2017-05-04 15:55:02.941517] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.941724] D > [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f770fe6b69e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x5b)[0x7f770fe67c5b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f770fe684f4] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_transport_notify+0x23)[0x7f770fe646b3] ))))) 0-: > /var/run/gluster/quotad.socket: ping timer event already removed > [2017-05-04 15:55:02.941738] T [cli-quotad-client.c:95:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_DISCONNECT > [2017-05-04 15:55:02.941745] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 > [2017-05-04 15:55:02.941805] T [rpc-clnt.c:1474:rpc_clnt_record] > 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: > [2017-05-04 15:55:02.941819] T > [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen > 160, payload: 96, rpc hdr: 64 > [2017-05-04 15:55:02.941958] T [socket.c:2945:socket_connect] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket. > so(+0x9b8b)[0x7f770d5a6b8b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_submit+0x33f)[0x7f770fe68acf] (--> > /usr/sbin/gluster(cli_submit_request+0x149)[0x7f77109ed189] (--> > /usr/sbin/gluster(cli_cmd_submit+0x93)[0x7f77109ef013] ))))) 0-glusterfs: > connect () called on transport already connected > [2017-05-04 15:55:02.943162] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: > submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to > rpc-transport (glusterfs) > [2017-05-04 15:55:02.944901] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] > 0-glusterfs: ping timeout is 0, returning > [2017-05-04 15:55:04.458391] D [socket.c:559:__socket_rwv] 0-glusterfs: EOF > on socket > [2017-05-04 15:55:04.458417] W [socket.c:590:__socket_rwv] 0-glusterfs: > readv on /var/run/glusterd.socket failed (No data available) > [2017-05-04 15:55:04.458427] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:04.458550] D > [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f770fe6b69e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x5b)[0x7f770fe67c5b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f770fe684f4] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_transport_notify+0x23)[0x7f770fe646b3] ))))) 0-: > /var/run/glusterd.socket: ping timer event already removed > [2017-05-04 15:55:04.458632] E [rpc-clnt.c:365:saved_frames_unwind] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(saved_frames_unwind+0x1de)[0x7f770fe6649e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(saved_frames_destroy+0xe)[0x7f770fe665ae] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x84)[0x7f770fe67c84] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f770fe684f4] ))))) 0-glusterfs: forced unwinding > frame type(Gluster CLI) op(--(1)) called at 2017-05-04 15:55:02.943160 > (xid=0x1) > [2017-05-04 15:55:04.458655] T [cli.c:272:cli_rpc_notify] 0-glusterfs: got > RPC_CLNT_DISCONNECT > > > > > [2] cat /var/log/glusterfs/etc-glusterfs-glusterd.vol.log > [2017-05-04 15:54:33.967651] I [MSGID: 100030] [glusterfsd.c:2454:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 > (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE > --log-buf-size=0) > [2017-05-04 15:54:33.968085] T [MSGID: 0] > [glusterfsd.c:573:create_fuse_mount] 0-glusterfsd: mount point not found, > not a client process > [2017-05-04 15:54:33.968553] D [MSGID: 0] > [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile > /var/run/glusterd.pid updated with pid 13120 > [2017-05-04 15:54:33.968584] D [logging.c:1791:__gf_log_inject_timer_event] > 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 > [2017-05-04 15:54:33.968687] D [MSGID: 0] [glusterfsd.c:660:get_volfp] > 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol > [2017-05-04 15:54:33.968797] T [MSGID: 0] [graph.y:186:new_volume] 0-parser: > New node for 'management' > [2017-05-04 15:54:33.968810] T [MSGID: 0] [xlator.c:193:xlator_dynload] > 0-xlator: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so > [2017-05-04 15:54:33.974984] T [MSGID: 0] [xlator.c:244:xlator_dynload] > 0-xlator: dlsym(reconfigure) on > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so: > undefined symbol: reconfigure -- neglecting > [2017-05-04 15:54:33.975006] T [MSGID: 0] [graph.y:217:volume_type] > 0-parser: Type:management:mgmt/glusterd > [2017-05-04 15:54:33.975032] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:working-directory:/var/lib/glusterd > [2017-05-04 15:54:33.975042] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport-type:socket,rdma > [2017-05-04 15:54:33.975048] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport.socket.keepalive-time:10 > [2017-05-04 15:54:33.975054] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport.socket.keepalive-interval:2 > [2017-05-04 15:54:33.975060] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport.socket.read-fail-log:off > [2017-05-04 15:54:33.975065] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:ping-timeout:0 > [2017-05-04 15:54:33.975071] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:event-threads:1 > [2017-05-04 15:54:33.975076] T [MSGID: 0] [graph.y:323:volume_end] 0-parser: > end:management > [2017-05-04 15:54:33.975143] I [MSGID: 106478] [glusterd.c:1381:init] > 0-management: Maximum allowed open file descriptors set to 65536 > [2017-05-04 15:54:33.975158] I [MSGID: 106479] [glusterd.c:1430:init] > 0-management: Using /var/lib/glusterd as working directory > [2017-05-04 15:54:33.975325] D [MSGID: 0] > [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog > value: 128 > [2017-05-04 15:54:33.975340] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx > pool: 64 > [2017-05-04 15:54:33.975459] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS > [2017-05-04 15:54:33.975469] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 > [2017-05-04 15:54:33.975473] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_UNIX > [2017-05-04 15:54:33.975478] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_NULL > [2017-05-04 15:54:33.975481] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: > RPC service inited. > [2017-05-04 15:54:33.975488] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, > Port: 0 > [2017-05-04 15:54:33.975502] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:54:33.977568] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.listen-backlog 128' > [2017-05-04 15:54:33.977589] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-interval 2' > [2017-05-04 15:54:33.977600] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-time 10' > [2017-05-04 15:54:33.977613] D [socket.c:3938:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [2017-05-04 15:54:33.977620] D [socket.c:4021:socket_init] > 0-socket.management: SSL support on the I/O path is NOT enabled > [2017-05-04 15:54:33.977625] D [socket.c:4024:socket_init] > 0-socket.management: SSL support for glusterd is NOT enabled > [2017-05-04 15:54:33.977628] D [socket.c:4041:socket_init] > 0-socket.management: using system polling thread > [2017-05-04 15:54:33.977635] D [name.c:584:server_fill_address_family] > 0-socket.management: option address-family not specified, defaulting to inet > [2017-05-04 15:54:33.977648] T [socket.c:831:__socket_nodelay] 0-management: > NODELAY enabled for socket 10 > [2017-05-04 15:54:33.977725] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so > [2017-05-04 15:54:33.984039] D [rpc-transport.c:321:rpc_transport_load] > 0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so: undefined > symbol: reconfigure > [2017-05-04 15:54:33.984081] T [MSGID: 0] [rdma.c:4560:gf_rdma_options_init] > 0-rpc-transport/rdma: rdma.management: defaulting MTU to '2048' > [2017-05-04 15:54:33.984120] W [MSGID: 103071] > [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event > channel creation failed [No such device] > [2017-05-04 15:54:33.984129] W [MSGID: 103055] [rdma.c:4896:init] > 0-rdma.management: Failed to initialize IB Device > [2017-05-04 15:54:33.984133] W [rpc-transport.c:354:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2017-05-04 15:54:33.984167] W [rpcsvc.c:1667:rpcsvc_create_listener] > 0-rpc-service: cannot create listener, initing the transport failed > [2017-05-04 15:54:33.984175] E [MSGID: 106243] [glusterd.c:1654:init] > 0-management: creation of 1 listeners failed, continuing with succeeded > transport > [2017-05-04 15:54:33.984180] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: > 2, Port: 0 > [2017-05-04 15:54:33.984185] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: > 1238463, Ver: 2, Port: 0 > [2017-05-04 15:54:33.984189] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: > 2, Port: 0 > [2017-05-04 15:54:33.984193] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, > Ver: 3, Port: 0 > [2017-05-04 15:54:33.984201] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: > 1, Port: 0 > [2017-05-04 15:54:33.984205] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, > Ver: 2, Port: 0 > [2017-05-04 15:54:33.984209] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, > Ver: 1, Port: 0 > [2017-05-04 15:54:33.984218] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx > pool: 8 > [2017-05-04 15:54:33.984233] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS > [2017-05-04 15:54:33.984238] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 > [2017-05-04 15:54:33.984242] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_UNIX > [2017-05-04 15:54:33.984246] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_NULL > [2017-05-04 15:54:33.984249] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: > RPC service inited. > [2017-05-04 15:54:33.984253] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, > Port: 0 > [2017-05-04 15:54:33.984266] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:54:33.984289] D [socket.c:3887:socket_init] > 0-socket.management: disabling nodelay > [2017-05-04 15:54:33.984294] D [socket.c:3938:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [2017-05-04 15:54:33.984299] D [socket.c:4021:socket_init] > 0-socket.management: SSL support on the I/O path is NOT enabled > [2017-05-04 15:54:33.984303] D [socket.c:4024:socket_init] > 0-socket.management: SSL support for glusterd is NOT enabled > [2017-05-04 15:54:33.984307] D [socket.c:4041:socket_init] > 0-socket.management: using system polling thread > [2017-05-04 15:54:33.984336] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: > 2, Port: 0 > [2017-05-04 15:54:33.984343] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: > 14398633, Ver: 2, Port: 0 > [2017-05-04 15:54:33.984362] D [MSGID: 0] > [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 > [2017-05-04 15:54:33.984369] D [MSGID: 0] [glusterd.c:1722:init] > 0-management: cannot get run-with-valgrind value > [2017-05-04 15:54:34.115026] D [MSGID: 0] > [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 > [2017-05-04 15:54:34.115121] D [MSGID: 0] > [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 > [2017-05-04 15:54:36.932822] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [2017-05-04 15:54:36.932848] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:36.932856] D [MSGID: 0] > [glusterd-store.c:1985:glusterd_retrieve_op_version] 0-management: Unable to > get store handle! > [2017-05-04 15:54:36.932863] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [2017-05-04 15:54:36.932869] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:36.932873] D [MSGID: 0] > [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get > storehandle! > [2017-05-04 15:54:36.932878] D [MSGID: 0] > [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 > [2017-05-04 15:54:36.932883] I [MSGID: 106514] > [glusterd-store.c:2120:glusterd_restore_op_version] 0-management: Detected > new install. Setting op-version to maximum : 30800 > [2017-05-04 15:54:36.932907] D [MSGID: 0] > [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: > Returning with 0 > [2017-05-04 15:54:36.932939] D [MSGID: 0] > [glusterd-store.c:4257:glusterd_store_retrieve_peers] 0-management: > Returning with 0 > [2017-05-04 15:54:36.932958] I [MSGID: 106194] > [glusterd-store.c:3628:glusterd_store_retrieve_missed_snaps_list] > 0-management: No missed snaps list. > [2017-05-04 15:54:36.932965] T [MSGID: 0] > [glusterd-store.c:3687:glusterd_store_retrieve_missed_snaps_list] > 0-management: Returning with 0 > [2017-05-04 15:54:36.932970] D [MSGID: 0] > [glusterd-store.c:3748:glusterd_store_retrieve_snaps] 0-management: > Returning with 0 > [2017-05-04 15:54:36.932975] T [MSGID: 0] > [glusterd-store.c:4425:glusterd_resolve_all_bricks] 0-management: Returning > with 0 > [2017-05-04 15:54:36.932998] T [MSGID: 0] > [glusterd-store.c:4379:glusterd_snap_cleanup] 0-management: Returning with 0 > [2017-05-04 15:54:36.933003] T [MSGID: 0] > [glusterd-store.c:4311:glusterd_recreate_all_snap_brick_mounts] > 0-management: Returning with 0 > [2017-05-04 15:54:36.933008] D [MSGID: 0] > [glusterd-store.c:4478:glusterd_restore] 0-management: Returning 0 > [2017-05-04 15:54:36.933044] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/options. [No such file or directory] > [2017-05-04 15:54:36.933051] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:36.933056] D [MSGID: 0] > [store.c:459:gf_store_handle_destroy] 0-: Returning 0 > [2017-05-04 15:54:36.933148] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa21389b800, remaining locks=0 > [2017-05-04 15:54:36.935326] D [MSGID: 0] [store.c:420:gf_store_handle_new] > 0-: Returning 0 > [2017-05-04 15:54:36.935368] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:54:36.939235] D [MSGID: 0] > [store.c:459:gf_store_handle_destroy] 0-: Returning 0 > [2017-05-04 15:54:36.939277] D [MSGID: 0] > [options.c:1214:xlator_option_init_int32] 0-management: option event-threads > using set value 1 > Final graph: > +---------------------------------------------------------------------------- > --+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.socket.listen-backlog 128 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: > +---------------------------------------------------------------------------- > --+ > [2017-05-04 15:54:36.939682] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:54:37.102439] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:54:37.102456] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 3) from rpc-transport (socket.management) > [2017-05-04 15:54:37.102472] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:54:37.102478] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - LIST_FRIENDS for > [2017-05-04 15:54:37.102556] I [MSGID: 106487] > [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: > Received cli list req > [2017-05-04 15:54:37.102575] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [2017-05-04 15:54:37.102584] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:37.102589] D [MSGID: 0] > [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get > storehandle! > [2017-05-04 15:54:37.102594] D [MSGID: 0] > [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 > [2017-05-04 15:54:37.102631] I [MSGID: 106477] > [glusterd.c:187:glusterd_uuid_generate_save] 0-management: generated UUID: > 8d317e96-3df6-4995-8b9b-694ccf780ce8 > [2017-05-04 15:54:37.105157] D [MSGID: 0] [store.c:420:gf_store_handle_new] > 0-: Returning 0 > [2017-05-04 15:54:37.105208] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:54:37.105224] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:54:37.108905] T [rpcsvc.c:1313:rpcsvc_submit_generic] > 0-rpc-service: Tx message: 156 > [2017-05-04 15:54:37.108920] T [rpcsvc.c:859:rpcsvc_record_build_header] > 0-rpc-service: Reply fraglen 180, payload: 156, rpc hdr: 24 > [2017-05-04 15:54:37.108932] T [rpcsvc.c:1361:rpcsvc_submit_generic] > 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD > svc cli, ProgVers: 2, Proc: 3) to rpc-transport (socket.management) > [2017-05-04 15:54:37.108945] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:54:37.108951] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:54:37.111912] D [socket.c:559:__socket_rwv] > 0-socket.management: EOF on socket > [2017-05-04 15:54:37.111925] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:54:37.111940] T [socket.c:4267:fini] 0-socket.management: > transport 0x7fa1fc000920 destroyed > [2017-05-04 15:54:37.111948] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=2 from gen=1 to slot->gen=2, fd=7, slot->fd=7 > [2017-05-04 15:55:02.567101] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:55:02.567118] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 5) from rpc-transport (socket.management) > [2017-05-04 15:55:02.567128] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:55:02.567134] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - GET_VOLUME for > [2017-05-04 15:55:02.567400] I [MSGID: 106488] > [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: > Received get vol req > [2017-05-04 15:55:02.567419] T [rpcsvc.c:1313:rpcsvc_submit_generic] > 0-rpc-service: Tx message: 36 > [2017-05-04 15:55:02.567426] T [rpcsvc.c:859:rpcsvc_record_build_header] > 0-rpc-service: Reply fraglen 60, payload: 36, rpc hdr: 24 > [2017-05-04 15:55:02.567435] T [rpcsvc.c:1361:rpcsvc_submit_generic] > 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD > svc cli, ProgVers: 2, Proc: 5) to rpc-transport (socket.management) > [2017-05-04 15:55:02.567447] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.567453] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.567893] D [socket.c:559:__socket_rwv] > 0-socket.management: EOF on socket > [2017-05-04 15:55:02.567904] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.567912] T [socket.c:4267:fini] 0-socket.management: > transport 0x7fa1fc000920 destroyed > [2017-05-04 15:55:02.567919] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=2 from gen=4 to slot->gen=5, fd=7, slot->fd=7 > [2017-05-04 15:55:02.855502] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:55:02.855518] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 3) from rpc-transport (socket.management) > [2017-05-04 15:55:02.855529] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:55:02.855535] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - LIST_FRIENDS for > [2017-05-04 15:55:02.856265] I [MSGID: 106487] > [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: > Received cli list req > [2017-05-04 15:55:02.856293] T [rpcsvc.c:1313:rpcsvc_submit_generic] > 0-rpc-service: Tx message: 156 > [2017-05-04 15:55:02.856300] T [rpcsvc.c:859:rpcsvc_record_build_header] > 0-rpc-service: Reply fraglen 180, payload: 156, rpc hdr: 24 > [2017-05-04 15:55:02.856310] T [rpcsvc.c:1361:rpcsvc_submit_generic] > 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD > svc cli, ProgVers: 2, Proc: 3) to rpc-transport (socket.management) > [2017-05-04 15:55:02.856320] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.856326] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.856946] D [socket.c:559:__socket_rwv] > 0-socket.management: EOF on socket > [2017-05-04 15:55:02.856958] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.856967] T [socket.c:4267:fini] 0-socket.management: > transport 0x7fa1fc000920 destroyed > [2017-05-04 15:55:02.856974] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=2 from gen=7 to slot->gen=8, fd=7, slot->fd=7 > [2017-05-04 15:55:02.942003] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:55:02.942020] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 1) from rpc-transport (socket.management) > [2017-05-04 15:55:02.942030] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:55:02.942035] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - CLI_PROBE for > [2017-05-04 15:55:02.942774] I [MSGID: 106487] > [glusterd-handler.c:1241:__glusterd_handle_cli_probe] 0-glusterd: Received > CLI probe req 10.0.50.215 24007 > [2017-05-04 15:55:02.942794] D [MSGID: 0] > [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 > [2017-05-04 15:55:02.942897] D [MSGID: 0] > [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 > [2017-05-04 15:55:02.942936] D [MSGID: 0] > [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 > [2017-05-04 15:55:02.942970] D [MSGID: 0] > [common-utils.c:3357:gf_is_local_addr] 0-management: 10.0.50.215 is not local > [2017-05-04 15:55:02.943003] D [MSGID: 0] > [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: > Unable to find friend: 10.0.50.215 > [2017-05-04 15:55:02.943019] D [MSGID: 0] > [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: > Unable to find friend: 10.0.50.215 > [2017-05-04 15:55:02.943025] D [MSGID: 0] > [glusterd-peer-utils.c:247:glusterd_peerinfo_find] 0-management: Unable to > find hostname: 10.0.50.215 > [2017-05-04 15:55:02.943031] I [MSGID: 106129] > [glusterd-handler.c:3670:glusterd_probe_begin] 0-glusterd: Unable to find > peerinfo for host: 10.0.50.215 (24007) > [2017-05-04 15:55:02.943040] D [MSGID: 0] > [glusterd-peer-utils.c:487:glusterd_peer_hostname_new] 0-glusterd: Returning > 0 > [2017-05-04 15:55:02.943045] D [MSGID: 0] > [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 > [2017-05-04 15:55:02.943054] D [MSGID: 0] > [glusterd-store.c:3943:glusterd_store_create_peer_dir] 0-glusterd: Returning > with 0 > [2017-05-04 15:55:02.945468] D [MSGID: 0] [store.c:420:gf_store_handle_new] > 0-: Returning 0 > [2017-05-04 15:55:02.945516] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:55:02.945530] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:55:02.945541] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:55:02.945545] D [MSGID: 0] > [glusterd-store.c:4072:glusterd_store_peer_write] 0-glusterd: Returning with > 0 > [2017-05-04 15:55:02.949254] D [MSGID: 0] > [glusterd-store.c:4098:glusterd_store_perform_peer_store] 0-glusterd: > Returning 0 > [2017-05-04 15:55:02.949266] D [MSGID: 0] > [glusterd-store.c:4119:glusterd_store_peerinfo] 0-glusterd: Returning with 0 > [2017-05-04 15:55:02.949282] D [MSGID: 0] > [glusterd-handler.c:3461:glusterd_transport_inet_options_build] 0-glusterd: > Returning 0 > [2017-05-04 15:55:02.949302] I [rpc-clnt.c:1046:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2017-05-04 15:55:02.949309] D [rpc-clnt.c:1058:rpc_clnt_connection_init] > 0-management: setting ping-timeout to 0 > [2017-05-04 15:55:02.949316] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.949341] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.tcp-user-timeout -1' > [2017-05-04 15:55:02.949348] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-time 10' > [2017-05-04 15:55:02.949354] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-interval 2' > [2017-05-04 15:55:02.949361] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option remote-port 24007' > [2017-05-04 15:55:02.949373] D [socket.c:3938:socket_init] 0-management: > Configued transport.tcp-user-timeout=-1 > [2017-05-04 15:55:02.949381] D [socket.c:4021:socket_init] 0-management: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.949385] D [socket.c:4024:socket_init] 0-management: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.949389] D [socket.c:4041:socket_init] 0-management: > using system polling thread > [2017-05-04 15:55:02.949396] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-management: attempting reconnect > [2017-05-04 15:55:02.949401] T [socket.c:2953:socket_connect] 0-management: > connecting 0x7fa204005040, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.949407] D [name.c:168:client_fill_address_family] > 0-management: address-family not specified, marking it as unspec for > getaddrinfo to resolve from (remote-host: 10.0.50.215) > [2017-05-04 15:55:02.949413] T [MSGID: 0] > [common-utils.c:281:gf_resolve_ip6] 0-resolver: DNS cache not present, > freshly probing hostname: 10.0.50.215 > [2017-05-04 15:55:02.949670] D [MSGID: 0] > [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-10.0.50.215 > (port-24007) for hostname: 10.0.50.215 and port: 24007 > [2017-05-04 15:55:02.949679] D [socket.c:2899:socket_fix_ssl_opts] > 0-management: disabling SSL for portmapper connection > [2017-05-04 15:55:02.949691] T [socket.c:831:__socket_nodelay] 0-management: > NODELAY enabled for socket 12 > [2017-05-04 15:55:02.949696] T [socket.c:906:__socket_keepalive] > 0-management: Keep-alive enabled for socket 12, interval 2, idle: 10, > timeout: -1 > [2017-05-04 15:55:02.949747] D [MSGID: 0] > [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 24007, higher: > 24008 > [2017-05-04 15:55:02.949753] D [MSGID: 0] > [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 32765, higher: -1 > [2017-05-04 15:55:02.949758] D [MSGID: 0] > [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: -1, higher: -1 > pending frames: > frame : type(0) op(0) > patchset: git://git.gluster.com/glusterfs.git > signal received: 11 > time of crash: > 2017-05-04 15:55:02 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string content of cat /var/lib/glusterd/peers/10.0.50.215 shows uuid as 00000000-0000-0000-0000-000000000000, which is wrong. I believe uuid is not getting generated properly in your setup. However in glusterd.logs I could see uuid is being generated properly, "generated UUID:8d317e96-3df6-4995-8b9b-694ccf780ce8" Could you please share your setup details so that I can also test in the sanme environment in order to reproduce the issue.g: glusterfs 3.8.11
(In reply to Ben Werthmann from comment #8) > (In reply to Gaurav Yadav from comment #7) > > I try recreating the issue locally with 2 ubuntu servers, and I was not able > > to recreate the issue locally. > > > > Repro steps > > 1. Install gluster on Ubuntu 14.04 > > 2. sudo gluster peer probe 192.168.122.159 > > > > I am not seeing any issue while peer probing...command getting executed > > succesfully. > > > > > > I have few queries regarding the info you have provided.. > > > > 1. Problematic peer entry: > > 2. cat /var/lib/glusterd/peers/ip-10-0-50-25.us-west-1.compute.internal > > 3. uuid=00000000-0000-0000-0000-000000000000 > > 4. state=0 > > 5. hostname1=ip-10-0-50-25.us-west-1.compute.internal > > > > Line 2 shows file name as "ip-10-0-50-25.us-west-1.compute.internal" inside > > peers directory, however glusterd generates file and named as per uuid. In > > your case filename is similar to hostname which is wrong. > > > > Could you please let me know whether it is glusterd generated file or > > manually some changes have been done.? > > glusterd generates this file. In that example I used a hostname, but the > same is true with an IP address. > > 1. create new instance > 2. echo "exec /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE > --log-buf-size=0" > /etc/init/glusterfs-server.override > 3. install glusterfs-server > 4. ln -sf /lib/init/upstart-job /etc/init.d/glusterfs-server > 5. /usr/sbin/gluster pool list > 6. /usr/sbin/gluster --loglevel=TRACE volume info > 7. /usr/sbin/gluster volume info > 8. /usr/sbin/gluster --log-level=TRACE pool list > 9. /usr/sbin/gluster --log-level=TRACE peer probe 10.0.50.215 > stdout=Connection failed. Please check if gluster daemon is operational. > stderr= > rc=1 > > glusterd segfaults > glusterd logs at [1] > gluster cli logs at [2] > > 10. cat /var/lib/glusterd/peers/10.0.50.215 > uuid=00000000-0000-0000-0000-000000000000 > state=0 > hostname1=10.0.50.215 > > > > > > Answer to your query.. > > There's another issue which may be related. I noticed that glusterd.info was > > not self-populating. As a workaround I issue 'gluster pool list' which > > triggers glusterd to generate and store a UUID: > > > > glusterd.info file gets generated whenever we do probe and create volume for > > the first time. It contains UUID of local node which is used by glusterd for > > validation purpose. > > Thank you for this info. > > > > [1] cat /var/log/glusterfs/cli.log > [2017-05-04 15:54:37.044769] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:54:37.102299] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:54:37.102339] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:54:37.109025] I [input.c:31:cli_batch] 0-: Exiting with: 0 > [2017-05-04 15:55:02.509217] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:55:02.566970] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:55:02.567011] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.567602] I [cli-rpc-ops.c:817:gf_cli_get_volume_cbk] > 0-cli: Received resp to get vol: 0 > [2017-05-04 15:55:02.567645] I [input.c:31:cli_batch] 0-: Exiting with: 0 > [2017-05-04 15:55:02.796716] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:55:02.796739] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to > glusterd using default socket > [2017-05-04 15:55:02.796761] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.796767] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.796776] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.798821] D [socket.c:3887:socket_init] 0-glusterfs: > disabling nodelay > [2017-05-04 15:55:02.798835] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.798841] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.798845] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.798849] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.798856] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.798863] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f559a168d40, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.798870] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/glusterd.socket > [2017-05-04 15:55:02.798879] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.798988] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.798997] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.799003] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.799019] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.799025] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.799029] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.799033] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.799038] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.799042] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f559a19c3f0, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.799047] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/gluster/quotad.socket > [2017-05-04 15:55:02.799055] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.799075] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799094] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799109] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799114] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799119] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799130] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799136] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799142] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799149] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799156] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799163] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799169] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799174] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799179] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799184] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799189] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799194] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.799199] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854105] D > [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 > [2017-05-04 15:55:02.854146] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854157] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854177] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854183] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854195] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854204] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854217] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854224] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854229] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854237] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854242] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854247] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854259] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854266] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854272] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854276] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854280] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854284] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854292] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854298] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854303] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854308] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854313] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854317] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854322] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854326] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854330] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854335] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854340] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854344] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854348] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854352] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854362] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854370] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854375] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854379] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854385] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854390] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854395] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854404] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854410] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854415] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854419] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854424] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854428] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.854484] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:55:02.854506] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got > RPC_CLNT_CONNECT > [2017-05-04 15:55:02.854518] T [cli-quotad-client.c:89:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_CONNECT > [2017-05-04 15:55:02.854523] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.854674] T [rpc-clnt.c:1474:rpc_clnt_record] > 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: > [2017-05-04 15:55:02.854690] T > [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen > 72, payload: 8, rpc hdr: 64 > [2017-05-04 15:55:02.855348] D > [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f5598d00992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f559867569e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x5b)[0x7f5598671c5b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f55986724f4] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_transport_notify+0x23)[0x7f559866e6b3] ))))) 0-: > /var/run/gluster/quotad.socket: ping timer event already removed > [2017-05-04 15:55:02.855362] T [cli-quotad-client.c:95:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_DISCONNECT > [2017-05-04 15:55:02.855368] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 > [2017-05-04 15:55:02.855461] T [socket.c:2945:socket_connect] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f5598d00992] (--> > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket. > so(+0x9b8b)[0x7f5595db0b8b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_submit+0x33f)[0x7f5598672acf] (--> > /usr/sbin/gluster(cli_submit_request+0x149)[0x7f55991f7189] (--> > /usr/sbin/gluster(cli_cmd_submit+0x93)[0x7f55991f9013] ))))) 0-glusterfs: > connect () called on transport already connected > [2017-05-04 15:55:02.856424] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: > submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 3) to > rpc-transport (glusterfs) > [2017-05-04 15:55:02.856437] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] > 0-glusterfs: ping timeout is 0, returning > [2017-05-04 15:55:02.856457] T [rpc-clnt.c:688:rpc_clnt_reply_init] > 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI, > ProgVers: 2, Proc: 3) from rpc-transport (glusterfs) > [2017-05-04 15:55:02.856466] D [cli-rpc-ops.c:419:gf_cli_list_friends_cbk] > 0-cli: Received resp to list: 0 > [2017-05-04 15:55:02.856497] D [cli-cmd.c:390:cli_cmd_submit] 0-cli: > Returning 0 > [2017-05-04 15:55:02.856503] D [cli-rpc-ops.c:4151:gf_cli_list_friends] > 0-cli: Returning 0 > [2017-05-04 15:55:02.856507] I [input.c:31:cli_batch] 0-: Exiting with: 0 > [2017-05-04 15:55:02.882825] I [cli.c:728:main] 0-cli: Started running > /usr/sbin/gluster with version 3.8.11 > [2017-05-04 15:55:02.882845] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to > glusterd using default socket > [2017-05-04 15:55:02.882869] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.882875] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.882883] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.884833] D [socket.c:3887:socket_init] 0-glusterfs: > disabling nodelay > [2017-05-04 15:55:02.884845] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.884851] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.884855] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.884858] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.884866] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.884873] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f7711480d40, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.884879] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/glusterd.socket > [2017-05-04 15:55:02.884889] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.885060] D [rpc-clnt.c:1049:rpc_clnt_connection_init] > 0-glusterfs: defaulting frame-timeout to 30mins > [2017-05-04 15:55:02.885070] D [rpc-clnt.c:1063:rpc_clnt_connection_init] > 0-glusterfs: disable ping-timeout > [2017-05-04 15:55:02.885077] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.885093] D [socket.c:3938:socket_init] 0-glusterfs: > Configued transport.tcp-user-timeout=0 > [2017-05-04 15:55:02.885099] D [socket.c:4021:socket_init] 0-glusterfs: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.885103] D [socket.c:4024:socket_init] 0-glusterfs: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.885107] D [socket.c:4041:socket_init] 0-glusterfs: > using system polling thread > [2017-05-04 15:55:02.885112] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-glusterfs: attempting reconnect > [2017-05-04 15:55:02.885116] T [socket.c:2953:socket_connect] 0-glusterfs: > connecting 0x7f77114b43f0, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.885121] T > [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using > connect-path /var/run/gluster/quotad.socket > [2017-05-04 15:55:02.885129] T [name.c:125:af_unix_client_bind] 0-glusterfs: > bind-path not specified for unix socket, letting connect to assign default > value > [2017-05-04 15:55:02.885149] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885167] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885173] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885178] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885183] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885194] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885200] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885206] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885214] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885221] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885228] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885234] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885239] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885253] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885259] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885264] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885269] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.885274] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941103] D > [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 > [2017-05-04 15:55:02.941143] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941153] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941173] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941179] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941191] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941200] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941213] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941220] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941225] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941233] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941238] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941243] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941255] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941262] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941268] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941272] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941276] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941281] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941287] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941293] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941299] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941303] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941308] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941313] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941317] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941322] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941326] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941331] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941335] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941340] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941344] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941348] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941353] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941360] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941365] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941369] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941375] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941380] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941384] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941394] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941399] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941404] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941414] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941419] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941423] D [registry.c:403:cli_cmd_register] 0-cli: > Returning 0 > [2017-05-04 15:55:02.941479] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:55:02.941501] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got > RPC_CLNT_CONNECT > [2017-05-04 15:55:02.941512] T [cli-quotad-client.c:89:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_CONNECT > [2017-05-04 15:55:02.941517] I [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.941724] D > [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f770fe6b69e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x5b)[0x7f770fe67c5b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f770fe684f4] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_transport_notify+0x23)[0x7f770fe646b3] ))))) 0-: > /var/run/gluster/quotad.socket: ping timer event already removed > [2017-05-04 15:55:02.941738] T [cli-quotad-client.c:95:cli_quotad_notify] > 0-glusterfs: got RPC_CLNT_DISCONNECT > [2017-05-04 15:55:02.941745] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 > [2017-05-04 15:55:02.941805] T [rpc-clnt.c:1474:rpc_clnt_record] > 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: > [2017-05-04 15:55:02.941819] T > [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen > 160, payload: 96, rpc hdr: 64 > [2017-05-04 15:55:02.941958] T [socket.c:2945:socket_connect] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket. > so(+0x9b8b)[0x7f770d5a6b8b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_submit+0x33f)[0x7f770fe68acf] (--> > /usr/sbin/gluster(cli_submit_request+0x149)[0x7f77109ed189] (--> > /usr/sbin/gluster(cli_cmd_submit+0x93)[0x7f77109ef013] ))))) 0-glusterfs: > connect () called on transport already connected > [2017-05-04 15:55:02.943162] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: > submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to > rpc-transport (glusterfs) > [2017-05-04 15:55:02.944901] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] > 0-glusterfs: ping timeout is 0, returning > [2017-05-04 15:55:04.458391] D [socket.c:559:__socket_rwv] 0-glusterfs: EOF > on socket > [2017-05-04 15:55:04.458417] W [socket.c:590:__socket_rwv] 0-glusterfs: > readv on /var/run/glusterd.socket failed (No data available) > [2017-05-04 15:55:04.458427] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:04.458550] D > [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f770fe6b69e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x5b)[0x7f770fe67c5b] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f770fe684f4] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_transport_notify+0x23)[0x7f770fe646b3] ))))) 0-: > /var/run/glusterd.socket: ping timer event already removed > [2017-05-04 15:55:04.458632] E [rpc-clnt.c:365:saved_frames_unwind] (--> > /usr/lib/x86_64-linux-gnu/libglusterfs.so. > 0(_gf_log_callingfn+0x192)[0x7f77104f6992] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(saved_frames_unwind+0x1de)[0x7f770fe6649e] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(saved_frames_destroy+0xe)[0x7f770fe665ae] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_connection_cleanup+0x84)[0x7f770fe67c84] (--> > /usr/lib/x86_64-linux-gnu/libgfrpc.so. > 0(rpc_clnt_notify+0x94)[0x7f770fe684f4] ))))) 0-glusterfs: forced unwinding > frame type(Gluster CLI) op(--(1)) called at 2017-05-04 15:55:02.943160 > (xid=0x1) > [2017-05-04 15:55:04.458655] T [cli.c:272:cli_rpc_notify] 0-glusterfs: got > RPC_CLNT_DISCONNECT > > > > > [2] cat /var/log/glusterfs/etc-glusterfs-glusterd.vol.log > [2017-05-04 15:54:33.967651] I [MSGID: 100030] [glusterfsd.c:2454:main] > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 > (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE > --log-buf-size=0) > [2017-05-04 15:54:33.968085] T [MSGID: 0] > [glusterfsd.c:573:create_fuse_mount] 0-glusterfsd: mount point not found, > not a client process > [2017-05-04 15:54:33.968553] D [MSGID: 0] > [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile > /var/run/glusterd.pid updated with pid 13120 > [2017-05-04 15:54:33.968584] D [logging.c:1791:__gf_log_inject_timer_event] > 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 > [2017-05-04 15:54:33.968687] D [MSGID: 0] [glusterfsd.c:660:get_volfp] > 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol > [2017-05-04 15:54:33.968797] T [MSGID: 0] [graph.y:186:new_volume] 0-parser: > New node for 'management' > [2017-05-04 15:54:33.968810] T [MSGID: 0] [xlator.c:193:xlator_dynload] > 0-xlator: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so > [2017-05-04 15:54:33.974984] T [MSGID: 0] [xlator.c:244:xlator_dynload] > 0-xlator: dlsym(reconfigure) on > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so: > undefined symbol: reconfigure -- neglecting > [2017-05-04 15:54:33.975006] T [MSGID: 0] [graph.y:217:volume_type] > 0-parser: Type:management:mgmt/glusterd > [2017-05-04 15:54:33.975032] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:working-directory:/var/lib/glusterd > [2017-05-04 15:54:33.975042] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport-type:socket,rdma > [2017-05-04 15:54:33.975048] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport.socket.keepalive-time:10 > [2017-05-04 15:54:33.975054] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport.socket.keepalive-interval:2 > [2017-05-04 15:54:33.975060] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:transport.socket.read-fail-log:off > [2017-05-04 15:54:33.975065] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:ping-timeout:0 > [2017-05-04 15:54:33.975071] T [MSGID: 0] [graph.y:252:volume_option] > 0-parser: Option:management:event-threads:1 > [2017-05-04 15:54:33.975076] T [MSGID: 0] [graph.y:323:volume_end] 0-parser: > end:management > [2017-05-04 15:54:33.975143] I [MSGID: 106478] [glusterd.c:1381:init] > 0-management: Maximum allowed open file descriptors set to 65536 > [2017-05-04 15:54:33.975158] I [MSGID: 106479] [glusterd.c:1430:init] > 0-management: Using /var/lib/glusterd as working directory > [2017-05-04 15:54:33.975325] D [MSGID: 0] > [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog > value: 128 > [2017-05-04 15:54:33.975340] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx > pool: 64 > [2017-05-04 15:54:33.975459] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS > [2017-05-04 15:54:33.975469] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 > [2017-05-04 15:54:33.975473] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_UNIX > [2017-05-04 15:54:33.975478] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_NULL > [2017-05-04 15:54:33.975481] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: > RPC service inited. > [2017-05-04 15:54:33.975488] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, > Port: 0 > [2017-05-04 15:54:33.975502] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:54:33.977568] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.listen-backlog 128' > [2017-05-04 15:54:33.977589] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-interval 2' > [2017-05-04 15:54:33.977600] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-time 10' > [2017-05-04 15:54:33.977613] D [socket.c:3938:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [2017-05-04 15:54:33.977620] D [socket.c:4021:socket_init] > 0-socket.management: SSL support on the I/O path is NOT enabled > [2017-05-04 15:54:33.977625] D [socket.c:4024:socket_init] > 0-socket.management: SSL support for glusterd is NOT enabled > [2017-05-04 15:54:33.977628] D [socket.c:4041:socket_init] > 0-socket.management: using system polling thread > [2017-05-04 15:54:33.977635] D [name.c:584:server_fill_address_family] > 0-socket.management: option address-family not specified, defaulting to inet > [2017-05-04 15:54:33.977648] T [socket.c:831:__socket_nodelay] 0-management: > NODELAY enabled for socket 10 > [2017-05-04 15:54:33.977725] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so > [2017-05-04 15:54:33.984039] D [rpc-transport.c:321:rpc_transport_load] > 0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so: undefined > symbol: reconfigure > [2017-05-04 15:54:33.984081] T [MSGID: 0] [rdma.c:4560:gf_rdma_options_init] > 0-rpc-transport/rdma: rdma.management: defaulting MTU to '2048' > [2017-05-04 15:54:33.984120] W [MSGID: 103071] > [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event > channel creation failed [No such device] > [2017-05-04 15:54:33.984129] W [MSGID: 103055] [rdma.c:4896:init] > 0-rdma.management: Failed to initialize IB Device > [2017-05-04 15:54:33.984133] W [rpc-transport.c:354:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2017-05-04 15:54:33.984167] W [rpcsvc.c:1667:rpcsvc_create_listener] > 0-rpc-service: cannot create listener, initing the transport failed > [2017-05-04 15:54:33.984175] E [MSGID: 106243] [glusterd.c:1654:init] > 0-management: creation of 1 listeners failed, continuing with succeeded > transport > [2017-05-04 15:54:33.984180] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: > 2, Port: 0 > [2017-05-04 15:54:33.984185] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: > 1238463, Ver: 2, Port: 0 > [2017-05-04 15:54:33.984189] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: > 2, Port: 0 > [2017-05-04 15:54:33.984193] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, > Ver: 3, Port: 0 > [2017-05-04 15:54:33.984201] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: > 1, Port: 0 > [2017-05-04 15:54:33.984205] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, > Ver: 2, Port: 0 > [2017-05-04 15:54:33.984209] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, > Ver: 1, Port: 0 > [2017-05-04 15:54:33.984218] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx > pool: 8 > [2017-05-04 15:54:33.984233] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS > [2017-05-04 15:54:33.984238] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 > [2017-05-04 15:54:33.984242] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_UNIX > [2017-05-04 15:54:33.984246] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] > 0-rpc-service: Authentication enabled: AUTH_NULL > [2017-05-04 15:54:33.984249] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: > RPC service inited. > [2017-05-04 15:54:33.984253] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, > Port: 0 > [2017-05-04 15:54:33.984266] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:54:33.984289] D [socket.c:3887:socket_init] > 0-socket.management: disabling nodelay > [2017-05-04 15:54:33.984294] D [socket.c:3938:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [2017-05-04 15:54:33.984299] D [socket.c:4021:socket_init] > 0-socket.management: SSL support on the I/O path is NOT enabled > [2017-05-04 15:54:33.984303] D [socket.c:4024:socket_init] > 0-socket.management: SSL support for glusterd is NOT enabled > [2017-05-04 15:54:33.984307] D [socket.c:4041:socket_init] > 0-socket.management: using system polling thread > [2017-05-04 15:54:33.984336] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: > 2, Port: 0 > [2017-05-04 15:54:33.984343] D [rpcsvc.c:1895:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: > 14398633, Ver: 2, Port: 0 > [2017-05-04 15:54:33.984362] D [MSGID: 0] > [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 > [2017-05-04 15:54:33.984369] D [MSGID: 0] [glusterd.c:1722:init] > 0-management: cannot get run-with-valgrind value > [2017-05-04 15:54:34.115026] D [MSGID: 0] > [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 > [2017-05-04 15:54:34.115121] D [MSGID: 0] > [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 > [2017-05-04 15:54:36.932822] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [2017-05-04 15:54:36.932848] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:36.932856] D [MSGID: 0] > [glusterd-store.c:1985:glusterd_retrieve_op_version] 0-management: Unable to > get store handle! > [2017-05-04 15:54:36.932863] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [2017-05-04 15:54:36.932869] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:36.932873] D [MSGID: 0] > [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get > storehandle! > [2017-05-04 15:54:36.932878] D [MSGID: 0] > [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 > [2017-05-04 15:54:36.932883] I [MSGID: 106514] > [glusterd-store.c:2120:glusterd_restore_op_version] 0-management: Detected > new install. Setting op-version to maximum : 30800 > [2017-05-04 15:54:36.932907] D [MSGID: 0] > [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: > Returning with 0 > [2017-05-04 15:54:36.932939] D [MSGID: 0] > [glusterd-store.c:4257:glusterd_store_retrieve_peers] 0-management: > Returning with 0 > [2017-05-04 15:54:36.932958] I [MSGID: 106194] > [glusterd-store.c:3628:glusterd_store_retrieve_missed_snaps_list] > 0-management: No missed snaps list. > [2017-05-04 15:54:36.932965] T [MSGID: 0] > [glusterd-store.c:3687:glusterd_store_retrieve_missed_snaps_list] > 0-management: Returning with 0 > [2017-05-04 15:54:36.932970] D [MSGID: 0] > [glusterd-store.c:3748:glusterd_store_retrieve_snaps] 0-management: > Returning with 0 > [2017-05-04 15:54:36.932975] T [MSGID: 0] > [glusterd-store.c:4425:glusterd_resolve_all_bricks] 0-management: Returning > with 0 > [2017-05-04 15:54:36.932998] T [MSGID: 0] > [glusterd-store.c:4379:glusterd_snap_cleanup] 0-management: Returning with 0 > [2017-05-04 15:54:36.933003] T [MSGID: 0] > [glusterd-store.c:4311:glusterd_recreate_all_snap_brick_mounts] > 0-management: Returning with 0 > [2017-05-04 15:54:36.933008] D [MSGID: 0] > [glusterd-store.c:4478:glusterd_restore] 0-management: Returning 0 > [2017-05-04 15:54:36.933044] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/options. [No such file or directory] > [2017-05-04 15:54:36.933051] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:36.933056] D [MSGID: 0] > [store.c:459:gf_store_handle_destroy] 0-: Returning 0 > [2017-05-04 15:54:36.933148] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa21389b800, remaining locks=0 > [2017-05-04 15:54:36.935326] D [MSGID: 0] [store.c:420:gf_store_handle_new] > 0-: Returning 0 > [2017-05-04 15:54:36.935368] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:54:36.939235] D [MSGID: 0] > [store.c:459:gf_store_handle_destroy] 0-: Returning 0 > [2017-05-04 15:54:36.939277] D [MSGID: 0] > [options.c:1214:xlator_option_init_int32] 0-management: option event-threads > using set value 1 > Final graph: > +---------------------------------------------------------------------------- > --+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.socket.listen-backlog 128 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: > +---------------------------------------------------------------------------- > --+ > [2017-05-04 15:54:36.939682] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2017-05-04 15:54:37.102439] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:54:37.102456] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 3) from rpc-transport (socket.management) > [2017-05-04 15:54:37.102472] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:54:37.102478] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - LIST_FRIENDS for > [2017-05-04 15:54:37.102556] I [MSGID: 106487] > [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: > Received cli list req > [2017-05-04 15:54:37.102575] E [MSGID: 101032] > [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [2017-05-04 15:54:37.102584] D [MSGID: 0] > [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 > [2017-05-04 15:54:37.102589] D [MSGID: 0] > [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get > storehandle! > [2017-05-04 15:54:37.102594] D [MSGID: 0] > [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 > [2017-05-04 15:54:37.102631] I [MSGID: 106477] > [glusterd.c:187:glusterd_uuid_generate_save] 0-management: generated UUID: > 8d317e96-3df6-4995-8b9b-694ccf780ce8 > [2017-05-04 15:54:37.105157] D [MSGID: 0] [store.c:420:gf_store_handle_new] > 0-: Returning 0 > [2017-05-04 15:54:37.105208] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:54:37.105224] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:54:37.108905] T [rpcsvc.c:1313:rpcsvc_submit_generic] > 0-rpc-service: Tx message: 156 > [2017-05-04 15:54:37.108920] T [rpcsvc.c:859:rpcsvc_record_build_header] > 0-rpc-service: Reply fraglen 180, payload: 156, rpc hdr: 24 > [2017-05-04 15:54:37.108932] T [rpcsvc.c:1361:rpcsvc_submit_generic] > 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD > svc cli, ProgVers: 2, Proc: 3) to rpc-transport (socket.management) > [2017-05-04 15:54:37.108945] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:54:37.108951] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:54:37.111912] D [socket.c:559:__socket_rwv] > 0-socket.management: EOF on socket > [2017-05-04 15:54:37.111925] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:54:37.111940] T [socket.c:4267:fini] 0-socket.management: > transport 0x7fa1fc000920 destroyed > [2017-05-04 15:54:37.111948] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=2 from gen=1 to slot->gen=2, fd=7, slot->fd=7 > [2017-05-04 15:55:02.567101] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:55:02.567118] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 5) from rpc-transport (socket.management) > [2017-05-04 15:55:02.567128] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:55:02.567134] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - GET_VOLUME for > [2017-05-04 15:55:02.567400] I [MSGID: 106488] > [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: > Received get vol req > [2017-05-04 15:55:02.567419] T [rpcsvc.c:1313:rpcsvc_submit_generic] > 0-rpc-service: Tx message: 36 > [2017-05-04 15:55:02.567426] T [rpcsvc.c:859:rpcsvc_record_build_header] > 0-rpc-service: Reply fraglen 60, payload: 36, rpc hdr: 24 > [2017-05-04 15:55:02.567435] T [rpcsvc.c:1361:rpcsvc_submit_generic] > 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD > svc cli, ProgVers: 2, Proc: 5) to rpc-transport (socket.management) > [2017-05-04 15:55:02.567447] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.567453] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.567893] D [socket.c:559:__socket_rwv] > 0-socket.management: EOF on socket > [2017-05-04 15:55:02.567904] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.567912] T [socket.c:4267:fini] 0-socket.management: > transport 0x7fa1fc000920 destroyed > [2017-05-04 15:55:02.567919] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=2 from gen=4 to slot->gen=5, fd=7, slot->fd=7 > [2017-05-04 15:55:02.855502] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:55:02.855518] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 3) from rpc-transport (socket.management) > [2017-05-04 15:55:02.855529] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:55:02.855535] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - LIST_FRIENDS for > [2017-05-04 15:55:02.856265] I [MSGID: 106487] > [glusterd-handler.c:1474:__glusterd_handle_cli_list_friends] 0-glusterd: > Received cli list req > [2017-05-04 15:55:02.856293] T [rpcsvc.c:1313:rpcsvc_submit_generic] > 0-rpc-service: Tx message: 156 > [2017-05-04 15:55:02.856300] T [rpcsvc.c:859:rpcsvc_record_build_header] > 0-rpc-service: Reply fraglen 180, payload: 156, rpc hdr: 24 > [2017-05-04 15:55:02.856310] T [rpcsvc.c:1361:rpcsvc_submit_generic] > 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD > svc cli, ProgVers: 2, Proc: 3) to rpc-transport (socket.management) > [2017-05-04 15:55:02.856320] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.856326] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] > 0-: Unlock success 0x7fa1fc002280, remaining locks=0 > [2017-05-04 15:55:02.856946] D [socket.c:559:__socket_rwv] > 0-socket.management: EOF on socket > [2017-05-04 15:55:02.856958] D [socket.c:2403:socket_event_handler] > 0-transport: disconnecting now > [2017-05-04 15:55:02.856967] T [socket.c:4267:fini] 0-socket.management: > transport 0x7fa1fc000920 destroyed > [2017-05-04 15:55:02.856974] D [MSGID: 0] > [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped > on idx=2 from gen=7 to slot->gen=8, fd=7, slot->fd=7 > [2017-05-04 15:55:02.942003] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] > 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 > [2017-05-04 15:55:02.942020] T [rpcsvc.c:490:rpcsvc_request_create] > 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, > ProgVers: 2, Proc: 1) from rpc-transport (socket.management) > [2017-05-04 15:55:02.942030] T > [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth > Info: pid: 0, uid: 0, gid: 0, owner: 00000000 > [2017-05-04 15:55:02.942035] T [rpcsvc.c:311:rpcsvc_program_actor] > 0-rpc-service: Actor found: GlusterD svc cli - CLI_PROBE for > [2017-05-04 15:55:02.942774] I [MSGID: 106487] > [glusterd-handler.c:1241:__glusterd_handle_cli_probe] 0-glusterd: Received > CLI probe req 10.0.50.215 24007 > [2017-05-04 15:55:02.942794] D [MSGID: 0] > [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 > [2017-05-04 15:55:02.942897] D [MSGID: 0] > [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 > [2017-05-04 15:55:02.942936] D [MSGID: 0] > [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.215 > [2017-05-04 15:55:02.942970] D [MSGID: 0] > [common-utils.c:3357:gf_is_local_addr] 0-management: 10.0.50.215 is not local > [2017-05-04 15:55:02.943003] D [MSGID: 0] > [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: > Unable to find friend: 10.0.50.215 > [2017-05-04 15:55:02.943019] D [MSGID: 0] > [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: > Unable to find friend: 10.0.50.215 > [2017-05-04 15:55:02.943025] D [MSGID: 0] > [glusterd-peer-utils.c:247:glusterd_peerinfo_find] 0-management: Unable to > find hostname: 10.0.50.215 > [2017-05-04 15:55:02.943031] I [MSGID: 106129] > [glusterd-handler.c:3670:glusterd_probe_begin] 0-glusterd: Unable to find > peerinfo for host: 10.0.50.215 (24007) > [2017-05-04 15:55:02.943040] D [MSGID: 0] > [glusterd-peer-utils.c:487:glusterd_peer_hostname_new] 0-glusterd: Returning > 0 > [2017-05-04 15:55:02.943045] D [MSGID: 0] > [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 > [2017-05-04 15:55:02.943054] D [MSGID: 0] > [glusterd-store.c:3943:glusterd_store_create_peer_dir] 0-glusterd: Returning > with 0 > [2017-05-04 15:55:02.945468] D [MSGID: 0] [store.c:420:gf_store_handle_new] > 0-: Returning 0 > [2017-05-04 15:55:02.945516] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:55:02.945530] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:55:02.945541] D [MSGID: 0] [store.c:374:gf_store_save_value] > 0-management: returning: 0 > [2017-05-04 15:55:02.945545] D [MSGID: 0] > [glusterd-store.c:4072:glusterd_store_peer_write] 0-glusterd: Returning with > 0 > [2017-05-04 15:55:02.949254] D [MSGID: 0] > [glusterd-store.c:4098:glusterd_store_perform_peer_store] 0-glusterd: > Returning 0 > [2017-05-04 15:55:02.949266] D [MSGID: 0] > [glusterd-store.c:4119:glusterd_store_peerinfo] 0-glusterd: Returning with 0 > [2017-05-04 15:55:02.949282] D [MSGID: 0] > [glusterd-handler.c:3461:glusterd_transport_inet_options_build] 0-glusterd: > Returning 0 > [2017-05-04 15:55:02.949302] I [rpc-clnt.c:1046:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2017-05-04 15:55:02.949309] D [rpc-clnt.c:1058:rpc_clnt_connection_init] > 0-management: setting ping-timeout to 0 > [2017-05-04 15:55:02.949316] D [rpc-transport.c:283:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so > [2017-05-04 15:55:02.949341] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.tcp-user-timeout -1' > [2017-05-04 15:55:02.949348] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-time 10' > [2017-05-04 15:55:02.949354] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option transport.socket.keepalive-interval 2' > [2017-05-04 15:55:02.949361] T [MSGID: 0] > [options.c:86:xlator_option_validate_int] 0-management: no range check > required for 'option remote-port 24007' > [2017-05-04 15:55:02.949373] D [socket.c:3938:socket_init] 0-management: > Configued transport.tcp-user-timeout=-1 > [2017-05-04 15:55:02.949381] D [socket.c:4021:socket_init] 0-management: SSL > support on the I/O path is NOT enabled > [2017-05-04 15:55:02.949385] D [socket.c:4024:socket_init] 0-management: SSL > support for glusterd is NOT enabled > [2017-05-04 15:55:02.949389] D [socket.c:4041:socket_init] 0-management: > using system polling thread > [2017-05-04 15:55:02.949396] T [rpc-clnt.c:422:rpc_clnt_reconnect] > 0-management: attempting reconnect > [2017-05-04 15:55:02.949401] T [socket.c:2953:socket_connect] 0-management: > connecting 0x7fa204005040, state=0 gen=0 sock=-1 > [2017-05-04 15:55:02.949407] D [name.c:168:client_fill_address_family] > 0-management: address-family not specified, marking it as unspec for > getaddrinfo to resolve from (remote-host: 10.0.50.215) > [2017-05-04 15:55:02.949413] T [MSGID: 0] > [common-utils.c:281:gf_resolve_ip6] 0-resolver: DNS cache not present, > freshly probing hostname: 10.0.50.215 > [2017-05-04 15:55:02.949670] D [MSGID: 0] > [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-10.0.50.215 > (port-24007) for hostname: 10.0.50.215 and port: 24007 > [2017-05-04 15:55:02.949679] D [socket.c:2899:socket_fix_ssl_opts] > 0-management: disabling SSL for portmapper connection > [2017-05-04 15:55:02.949691] T [socket.c:831:__socket_nodelay] 0-management: > NODELAY enabled for socket 12 > [2017-05-04 15:55:02.949696] T [socket.c:906:__socket_keepalive] > 0-management: Keep-alive enabled for socket 12, interval 2, idle: 10, > timeout: -1 > [2017-05-04 15:55:02.949747] D [MSGID: 0] > [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 24007, higher: > 24008 > [2017-05-04 15:55:02.949753] D [MSGID: 0] > [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 32765, higher: -1 > [2017-05-04 15:55:02.949758] D [MSGID: 0] > [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: -1, higher: -1 > pending frames: > frame : type(0) op(0) > patchset: git://git.gluster.com/glusterfs.git > signal received: 11 > time of crash: > 2017-05-04 15:55:02 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string content of cat /var/lib/glusterd/peers/10.0.50.215 shows uuid as 00000000-0000-0000-0000-000000000000, mean it is not generating properly. I believe uuid is not getting generated properly in your setup. Can you check if the following steps work: 1. Remove the faulty peer file from /var/lib/glusterd/peers/ 2. On the host that is to be probed, i.e. 10.0.50.25, clean everything in /var/lib/glusterd/* 3. Restart glusterd services 4. Peer probe 10.0.50.25 again 5. check the content of /var/lib/glusterd/glusterd.info 6. check whether file inside /var/lib/glusterd/peers is generatd with the UUID. If still issue is reproducing, please share the all logs from both the nodes. Please share your setup details also, so that I can also test in the same environment in order to reproduce the issue.
(In reply to Gaurav Yadav from comment #10) > > > content of cat /var/lib/glusterd/peers/10.0.50.215 shows uuid as > 00000000-0000-0000-0000-000000000000, mean it is not generating properly. I > believe uuid is not getting generated properly in your setup. > > Can you check if the following steps work: > 1. Remove the faulty peer file from /var/lib/glusterd/peers/ > 2. On the host that is to be probed, i.e. 10.0.50.25, clean everything in > /var/lib/glusterd/* > 3. Restart glusterd services > 4. Peer probe 10.0.50.25 again > 5. check the content of /var/lib/glusterd/glusterd.info > 6. check whether file inside /var/lib/glusterd/peers is generatd with the > UUID. $ ssh 10.0.50.128 $ cat /var/lib/glusterd/peers/10.0.50.125 cat: /var/lib/glusterd/peers/10.0.50.125: Permission denied ubuntu@bwerthmann5-dc0936f3:~$ sudo cat /var/lib/glusterd/peers/10.0.50.125 uuid=00000000-0000-0000-0000-000000000000 state=0 hostname1=10.0.50.125 $ start glusterfs-server glusterfs-server start/running, process 14453 $ logout $ ssh 10.0.50.125 $ sudo ls -l /var/lib/glusterd/peers/ total 0 $ stop glusterfs-server glusterfs-server stop/waiting $ rm -rf /var/lib/glusterd/* $ start glusterfs-server glusterfs-server start/running, process 15170 $ logout $ ssh 10.0.50.128 $ sudo gluster --log-level=TRACE peer probe 10.0.50.125 Connection failed. Please check if gluster daemon is operational. $ cat /var/lib/glusterd/peers/10.0.50.125 uuid=00000000-0000-0000-0000-000000000000 state=0 hostname1=10.0.50.125 $ logout /var/lib/glusterd/glusterd.info does not exist on either node. (I removed the 'peer list' command from the setup which used to generate this). > > > If still issue is reproducing, please share the all logs from both the nodes. I'll post them shortly. > > Please share your setup details also, so that I can also test in the same > environment in order to reproduce the issue. I'll attach a "apport" bundle which should have all of the info you need.
# 10.0.50.128 /var/logs/glusterfs/cli.log [2017-05-05 09:44:30.680051] I [cli.c:728:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.11 [2017-05-05 09:44:30.735278] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 09:44:30.735315] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 09:44:30.735683] I [cli-rpc-ops.c:817:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2017-05-05 09:44:30.735727] I [input.c:31:cli_batch] 0-: Exiting with: 0 [2017-05-05 09:44:30.925946] I [cli.c:728:main] 0-cli: Started running /usr/sbin/gluster with version 3.8.11 [2017-05-05 09:44:30.925965] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to glusterd using default socket [2017-05-05 09:44:30.925986] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-05 09:44:30.925992] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-05 09:44:30.926000] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:44:30.927843] D [socket.c:3887:socket_init] 0-glusterfs: disabling nodelay [2017-05-05 09:44:30.927855] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-05 09:44:30.927861] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-05 09:44:30.927864] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-05 09:44:30.927868] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-05 09:44:30.927875] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-05 09:44:30.927881] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f967c447d40, state=0 gen=0 sock=-1 [2017-05-05 09:44:30.927890] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/glusterd.socket [2017-05-05 09:44:30.927899] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-05 09:44:30.927997] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-05 09:44:30.928005] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-05 09:44:30.928022] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:44:30.928039] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-05 09:44:30.928044] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-05 09:44:30.928048] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-05 09:44:30.928052] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-05 09:44:30.928057] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-05 09:44:30.928061] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f967c47b3f0, state=0 gen=0 sock=-1 [2017-05-05 09:44:30.928066] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-05-05 09:44:30.928074] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-05 09:44:30.928107] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928123] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928128] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928135] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928140] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928150] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928156] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928168] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928175] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928182] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928188] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928193] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928198] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928202] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928207] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928211] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928216] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.928221] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982205] D [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 [2017-05-05 09:44:30.982243] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982253] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982272] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982278] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982290] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982299] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982312] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982320] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982324] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982332] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982337] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982343] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982354] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982361] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982366] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982371] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982375] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982391] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982410] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982416] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982421] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982424] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982429] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982433] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982437] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982441] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982445] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982449] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982454] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982458] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982461] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982465] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982470] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982476] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982481] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982485] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982494] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982499] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982503] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982512] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982517] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982521] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982525] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982529] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982533] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 09:44:30.982576] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 09:44:30.982593] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-05 09:44:30.982602] T [cli-quotad-client.c:89:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-05 09:44:30.982606] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 09:44:30.982759] T [rpc-clnt.c:1474:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-05-05 09:44:30.982775] T [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 160, payload: 96, rpc hdr: 64 [2017-05-05 09:44:30.983591] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f967a8ac992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f967a22169e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7f967a21dc5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f967a21e4f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f967a21a6b3] ))))) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-05-05 09:44:30.983605] T [cli-quotad-client.c:95:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-05-05 09:44:30.983611] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-05 09:44:30.983698] T [socket.c:2945:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f967a8ac992] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so(+0x9b8b)[0x7f967795cb8b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x33f)[0x7f967a21eacf] (--> /usr/sbin/gluster(cli_submit_request+0x149)[0x7f967ada3189] (--> /usr/sbin/gluster(cli_cmd_submit+0x93)[0x7f967ada5013] ))))) 0-glusterfs: connect () called on transport already connected [2017-05-05 09:44:30.984616] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to rpc-transport (glusterfs) [2017-05-05 09:44:30.986033] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-05-05 09:44:32.612667] D [socket.c:559:__socket_rwv] 0-glusterfs: EOF on socket [2017-05-05 09:44:32.612691] W [socket.c:590:__socket_rwv] 0-glusterfs: readv on /var/run/glusterd.socket failed (No data available) [2017-05-05 09:44:32.612701] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 09:44:32.612815] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f967a8ac992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f967a22169e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7f967a21dc5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f967a21e4f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f967a21a6b3] ))))) 0-: /var/run/glusterd.socket: ping timer event already removed [2017-05-05 09:44:32.612894] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f967a8ac992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f967a21c49e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f967a21c5ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f967a21dc84] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f967a21e4f4] ))))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(1)) called at 2017-05-05 09:44:30.984614 (xid=0x1) [2017-05-05 09:44:32.612913] T [cli.c:272:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-05-05 10:01:54.594119] I [cli.c:728:main] 0-cli: Started running gluster with version 3.8.11 [2017-05-05 10:01:54.594149] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to glusterd using default socket [2017-05-05 10:01:54.594178] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-05 10:01:54.594185] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-05 10:01:54.594196] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 10:01:54.596407] D [socket.c:3887:socket_init] 0-glusterfs: disabling nodelay [2017-05-05 10:01:54.596425] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-05 10:01:54.596431] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-05 10:01:54.596435] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-05 10:01:54.596438] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-05 10:01:54.596447] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-05 10:01:54.596456] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f199c4cfd80, state=0 gen=0 sock=-1 [2017-05-05 10:01:54.596462] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/glusterd.socket [2017-05-05 10:01:54.596473] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-05 10:01:54.596634] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-05 10:01:54.596647] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-05 10:01:54.596653] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 10:01:54.596672] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-05 10:01:54.596678] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-05 10:01:54.596682] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-05 10:01:54.596686] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-05 10:01:54.596691] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-05 10:01:54.596695] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7f199c503400, state=0 gen=0 sock=-1 [2017-05-05 10:01:54.596700] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-05-05 10:01:54.596709] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-05 10:01:54.596734] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596753] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596758] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596763] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596768] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596789] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596795] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596800] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596807] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596814] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596820] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596825] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596830] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596834] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596839] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596843] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596848] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.596853] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655737] D [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 [2017-05-05 10:01:54.655771] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655780] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655798] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655804] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655814] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655822] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655834] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655841] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655845] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655852] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655857] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655861] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655872] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655878] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655883] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655887] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655891] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655895] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655901] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655906] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655911] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655914] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655919] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655923] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655927] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655931] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655935] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655939] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655943] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655947] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655951] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655954] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655959] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655966] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655975] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655979] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655984] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655989] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.655993] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.656001] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.656007] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.656011] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.656015] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.656019] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.656023] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:01:54.656071] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 10:01:54.656090] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-05 10:01:54.656101] T [cli-quotad-client.c:89:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-05 10:01:54.656106] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 10:01:54.656298] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f199b5ff992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7f199af7469e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7f199af70c5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f199af714f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f199af6d6b3] ))))) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-05-05 10:01:54.656312] T [cli-quotad-client.c:95:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-05-05 10:01:54.656318] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-05 10:01:54.656416] I [input.c:31:cli_batch] 0-: Exiting with: -1 [2017-05-05 10:02:37.592490] I [cli.c:728:main] 0-cli: Started running gluster with version 3.8.11 [2017-05-05 10:02:37.592508] D [cli.c:630:cli_rpc_init] 0-cli: Connecting to glusterd using default socket [2017-05-05 10:02:37.592530] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-05 10:02:37.592535] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-05 10:02:37.592544] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 10:02:37.594379] D [socket.c:3887:socket_init] 0-glusterfs: disabling nodelay [2017-05-05 10:02:37.594392] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-05 10:02:37.594398] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-05 10:02:37.594402] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-05 10:02:37.594406] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-05 10:02:37.594414] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-05 10:02:37.594422] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7fc12c4d1d80, state=0 gen=0 sock=-1 [2017-05-05 10:02:37.594428] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/glusterd.socket [2017-05-05 10:02:37.594437] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-05 10:02:37.594551] D [rpc-clnt.c:1049:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2017-05-05 10:02:37.594567] D [rpc-clnt.c:1063:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2017-05-05 10:02:37.594573] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 10:02:37.594589] D [socket.c:3938:socket_init] 0-glusterfs: Configued transport.tcp-user-timeout=0 [2017-05-05 10:02:37.594607] D [socket.c:4021:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2017-05-05 10:02:37.594611] D [socket.c:4024:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2017-05-05 10:02:37.594615] D [socket.c:4041:socket_init] 0-glusterfs: using system polling thread [2017-05-05 10:02:37.594619] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2017-05-05 10:02:37.594623] T [socket.c:2953:socket_connect] 0-glusterfs: connecting 0x7fc12c505400, state=0 gen=0 sock=-1 [2017-05-05 10:02:37.594627] T [name.c:310:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/gluster/quotad.socket [2017-05-05 10:02:37.594634] T [name.c:125:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2017-05-05 10:02:37.594652] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594669] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594674] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594679] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594683] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594693] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594699] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594704] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594711] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594718] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594724] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594729] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594734] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594738] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594743] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594747] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594752] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.594757] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645106] D [cli-cmd-volume.c:1849:cli_check_gsync_present] 0-cli: Returning 0 [2017-05-05 10:02:37.645140] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645150] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645169] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645175] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645187] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645195] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645208] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645216] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645220] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645228] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645233] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645239] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645250] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645257] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645263] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645271] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645275] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645280] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645286] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645292] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645298] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645302] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645307] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645312] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645316] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645321] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645325] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645330] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645334] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645339] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645343] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645347] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645352] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645359] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645364] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645369] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645375] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645380] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645385] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645394] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645400] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645405] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645409] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645414] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645419] D [registry.c:403:cli_cmd_register] 0-cli: Returning 0 [2017-05-05 10:02:37.645464] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 10:02:37.645483] T [cli.c:266:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-05 10:02:37.645493] T [cli-quotad-client.c:89:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2017-05-05 10:02:37.645498] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 10:02:37.645685] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7fc12b93b992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7fc12b2b069e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7fc12b2acc5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7fc12b2ad4f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fc12b2a96b3] ))))) 0-: /var/run/gluster/quotad.socket: ping timer event already removed [2017-05-05 10:02:37.645698] T [cli-quotad-client.c:95:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2017-05-05 10:02:37.645705] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=1 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-05 10:02:37.645756] T [rpc-clnt.c:1474:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2017-05-05 10:02:37.645769] T [rpc-clnt.c:1331:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 160, payload: 96, rpc hdr: 64 [2017-05-05 10:02:37.645857] T [socket.c:2945:socket_connect] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7fc12b93b992] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so(+0x9b8b)[0x7fc1289ebb8b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x33f)[0x7fc12b2adacf] (--> gluster(cli_submit_request+0x149)[0x7fc12be32189] (--> gluster(cli_cmd_submit+0x93)[0x7fc12be34013] ))))) 0-glusterfs: connect () called on transport already connected [2017-05-05 10:02:37.646380] T [rpc-clnt.c:1677:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 1) to rpc-transport (glusterfs) [2017-05-05 10:02:37.647560] D [rpc-clnt-ping.c:276:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2017-05-05 10:02:38.957443] D [socket.c:559:__socket_rwv] 0-glusterfs: EOF on socket [2017-05-05 10:02:38.957467] W [socket.c:590:__socket_rwv] 0-glusterfs: readv on /var/run/glusterd.socket failed (No data available) [2017-05-05 10:02:38.957476] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 10:02:38.957593] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7fc12b93b992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x8e)[0x7fc12b2b069e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x5b)[0x7fc12b2acc5b] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7fc12b2ad4f4] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fc12b2a96b3] ))))) 0-: /var/run/glusterd.socket: ping timer event already removed [2017-05-05 10:02:38.957671] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7fc12b93b992] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fc12b2ab49e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fc12b2ab5ae] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7fc12b2acc84] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7fc12b2ad4f4] ))))) 0-glusterfs: forced unwinding frame type(Gluster CLI) op(--(1)) called at 2017-05-05 10:02:37.646378 (xid=0x1) [2017-05-05 10:02:38.957690] T [cli.c:272:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT
# 10.0.50.128/var/logs/glusterfs/etc-glusterfs-glusterd.vol.log [2017-05-05 09:43:56.163483] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0) [2017-05-05 09:43:56.167597] T [MSGID: 0] [glusterfsd.c:573:create_fuse_mount] 0-glusterfsd: mount point not found, not a client process [2017-05-05 09:43:56.168086] D [MSGID: 0] [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/glusterd.pid updated with pid 13209 [2017-05-05 09:43:56.168117] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:43:56.168219] D [MSGID: 0] [glusterfsd.c:660:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2017-05-05 09:43:56.168321] T [MSGID: 0] [graph.y:186:new_volume] 0-parser: New node for 'management' [2017-05-05 09:43:56.168334] T [MSGID: 0] [xlator.c:193:xlator_dynload] 0-xlator: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so [2017-05-05 09:43:56.173490] T [MSGID: 0] [xlator.c:244:xlator_dynload] 0-xlator: dlsym(reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so: undefined symbol: reconfigure -- neglecting [2017-05-05 09:43:56.173514] T [MSGID: 0] [graph.y:217:volume_type] 0-parser: Type:management:mgmt/glusterd [2017-05-05 09:43:56.173528] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:working-directory:/var/lib/glusterd [2017-05-05 09:43:56.173536] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport-type:socket,rdma [2017-05-05 09:43:56.173542] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-time:10 [2017-05-05 09:43:56.173547] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-interval:2 [2017-05-05 09:43:56.173564] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.read-fail-log:off [2017-05-05 09:43:56.173570] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:ping-timeout:0 [2017-05-05 09:43:56.173575] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:event-threads:1 [2017-05-05 09:43:56.173580] T [MSGID: 0] [graph.y:323:volume_end] 0-parser: end:management [2017-05-05 09:43:56.173678] I [MSGID: 106478] [glusterd.c:1381:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-05-05 09:43:56.173695] I [MSGID: 106479] [glusterd.c:1430:init] 0-management: Using /var/lib/glusterd as working directory [2017-05-05 09:43:56.173863] D [MSGID: 0] [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 128 [2017-05-05 09:43:56.173878] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 64 [2017-05-05 09:43:56.173952] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:43:56.173958] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:43:56.173961] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:43:56.173966] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:43:56.173970] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:43:56.173976] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:43:56.173989] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:43:56.176044] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.listen-backlog 128' [2017-05-05 09:43:56.176065] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-05 09:43:56.176077] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-05 09:43:56.177378] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:43:56.177394] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:43:56.177399] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:43:56.177402] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:43:56.177409] D [name.c:584:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2017-05-05 09:43:56.177422] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 10 [2017-05-05 09:43:56.177490] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so [2017-05-05 09:43:56.177747] D [rpc-transport.c:321:rpc_transport_load] 0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so: undefined symbol: reconfigure [2017-05-05 09:43:56.177781] T [MSGID: 0] [rdma.c:4560:gf_rdma_options_init] 0-rpc-transport/rdma: rdma.management: defaulting MTU to '2048' [2017-05-05 09:43:56.177810] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-05-05 09:43:56.177817] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2017-05-05 09:43:56.177821] W [rpc-transport.c:354:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-05-05 09:43:56.177862] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-05-05 09:43:56.177870] E [MSGID: 106243] [glusterd.c:1654:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-05-05 09:43:56.177875] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2017-05-05 09:43:56.177880] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:43:56.177884] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2017-05-05 09:43:56.177888] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2017-05-05 09:43:56.177892] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2017-05-05 09:43:56.177896] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:43:56.177900] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2017-05-05 09:43:56.177908] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 8 [2017-05-05 09:43:56.183828] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:43:56.183842] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:43:56.183846] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:43:56.183850] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:43:56.183853] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:43:56.183857] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:43:56.183871] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:43:56.183893] D [socket.c:3887:socket_init] 0-socket.management: disabling nodelay [2017-05-05 09:43:56.183898] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:43:56.183903] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:43:56.183907] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:43:56.183910] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:43:56.183942] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:43:56.183949] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:43:56.183967] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-05 09:43:56.183975] D [MSGID: 0] [glusterd.c:1722:init] 0-management: cannot get run-with-valgrind value [2017-05-05 09:43:57.333911] D [MSGID: 0] [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 [2017-05-05 09:43:57.334011] D [MSGID: 0] [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 [2017-05-05 09:43:59.736461] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:43:59.736485] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:43:59.736492] D [MSGID: 0] [glusterd-store.c:1985:glusterd_retrieve_op_version] 0-management: Unable to get store handle! [2017-05-05 09:43:59.736499] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:43:59.736504] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:43:59.736508] D [MSGID: 0] [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get storehandle! [2017-05-05 09:43:59.736512] D [MSGID: 0] [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 [2017-05-05 09:43:59.736517] I [MSGID: 106514] [glusterd-store.c:2120:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30800 [2017-05-05 09:43:59.736540] D [MSGID: 0] [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: Returning with 0 [2017-05-05 09:43:59.736566] D [MSGID: 0] [glusterd-store.c:4257:glusterd_store_retrieve_peers] 0-management: Returning with 0 [2017-05-05 09:43:59.736584] I [MSGID: 106194] [glusterd-store.c:3628:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-05-05 09:43:59.736589] T [MSGID: 0] [glusterd-store.c:3687:glusterd_store_retrieve_missed_snaps_list] 0-management: Returning with 0 [2017-05-05 09:43:59.736594] D [MSGID: 0] [glusterd-store.c:3748:glusterd_store_retrieve_snaps] 0-management: Returning with 0 [2017-05-05 09:43:59.736598] T [MSGID: 0] [glusterd-store.c:4425:glusterd_resolve_all_bricks] 0-management: Returning with 0 [2017-05-05 09:43:59.736606] T [MSGID: 0] [glusterd-store.c:4379:glusterd_snap_cleanup] 0-management: Returning with 0 [2017-05-05 09:43:59.736610] T [MSGID: 0] [glusterd-store.c:4311:glusterd_recreate_all_snap_brick_mounts] 0-management: Returning with 0 [2017-05-05 09:43:59.736614] D [MSGID: 0] [glusterd-store.c:4478:glusterd_restore] 0-management: Returning 0 [2017-05-05 09:43:59.736643] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] [2017-05-05 09:43:59.736649] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:43:59.736654] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-05 09:43:59.736748] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7f1bcf445800, remaining locks=0 [2017-05-05 09:43:59.738228] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-05 09:43:59.738267] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 09:43:59.741807] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-05 09:43:59.741880] D [MSGID: 0] [options.c:1214:xlator_option_init_int32] 0-management: option event-threads using set value 1 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-05-05 09:43:59.742311] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 09:44:30.735419] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2017-05-05 09:44:30.735437] T [rpcsvc.c:490:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, ProgVers: 2, Proc: 5) from rpc-transport (socket.management) [2017-05-05 09:44:30.735453] T [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2017-05-05 09:44:30.735459] T [rpcsvc.c:311:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterD svc cli - GET_VOLUME for [2017-05-05 09:44:30.735553] I [MSGID: 106488] [glusterd-handler.c:1537:__glusterd_handle_cli_get_volume] 0-management: Received get vol req [2017-05-05 09:44:30.735576] T [rpcsvc.c:1313:rpcsvc_submit_generic] 0-rpc-service: Tx message: 36 [2017-05-05 09:44:30.735583] T [rpcsvc.c:859:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 60, payload: 36, rpc hdr: 24 [2017-05-05 09:44:30.735593] T [rpcsvc.c:1361:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 5) to rpc-transport (socket.management) [2017-05-05 09:44:30.735605] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7f1bb8002280, remaining locks=0 [2017-05-05 09:44:30.735611] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7f1bb8002280, remaining locks=0 [2017-05-05 09:44:30.735955] D [socket.c:559:__socket_rwv] 0-socket.management: EOF on socket [2017-05-05 09:44:30.735965] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 09:44:30.735974] T [socket.c:4267:fini] 0-socket.management: transport 0x7f1bb8000920 destroyed [2017-05-05 09:44:30.735980] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=2 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-05 09:44:30.983738] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2017-05-05 09:44:30.983753] T [rpcsvc.c:490:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, ProgVers: 2, Proc: 1) from rpc-transport (socket.management) [2017-05-05 09:44:30.983763] T [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2017-05-05 09:44:30.983769] T [rpcsvc.c:311:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterD svc cli - CLI_PROBE for [2017-05-05 09:44:30.984292] I [MSGID: 106487] [glusterd-handler.c:1241:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.0.50.125 24007 [2017-05-05 09:44:30.984313] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.125 [2017-05-05 09:44:30.984399] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.125 [2017-05-05 09:44:30.984435] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.125 [2017-05-05 09:44:30.984465] D [MSGID: 0] [common-utils.c:3357:gf_is_local_addr] 0-management: 10.0.50.125 is not local [2017-05-05 09:44:30.984496] D [MSGID: 0] [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.50.125 [2017-05-05 09:44:30.984511] D [MSGID: 0] [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.50.125 [2017-05-05 09:44:30.984516] D [MSGID: 0] [glusterd-peer-utils.c:247:glusterd_peerinfo_find] 0-management: Unable to find hostname: 10.0.50.125 [2017-05-05 09:44:30.984522] I [MSGID: 106129] [glusterd-handler.c:3670:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 10.0.50.125 (24007) [2017-05-05 09:44:30.984529] D [MSGID: 0] [glusterd-peer-utils.c:487:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2017-05-05 09:44:30.984535] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-05 09:44:30.984543] D [MSGID: 0] [glusterd-store.c:3943:glusterd_store_create_peer_dir] 0-glusterd: Returning with 0 [2017-05-05 09:44:30.986606] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-05 09:44:30.986665] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 09:44:30.986678] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 09:44:30.986687] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 09:44:30.986691] D [MSGID: 0] [glusterd-store.c:4072:glusterd_store_peer_write] 0-glusterd: Returning with 0 [2017-05-05 09:44:30.990018] D [MSGID: 0] [glusterd-store.c:4098:glusterd_store_perform_peer_store] 0-glusterd: Returning 0 [2017-05-05 09:44:30.990041] D [MSGID: 0] [glusterd-store.c:4119:glusterd_store_peerinfo] 0-glusterd: Returning with 0 [2017-05-05 09:44:30.990070] D [MSGID: 0] [glusterd-handler.c:3461:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2017-05-05 09:44:30.990104] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-05-05 09:44:30.990118] D [rpc-clnt.c:1058:rpc_clnt_connection_init] 0-management: setting ping-timeout to 0 [2017-05-05 09:44:30.990132] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:44:30.990168] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.tcp-user-timeout -1' [2017-05-05 09:44:30.990190] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-05 09:44:30.990196] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-05 09:44:30.990202] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option remote-port 24007' [2017-05-05 09:44:30.990209] D [socket.c:3938:socket_init] 0-management: Configued transport.tcp-user-timeout=-1 [2017-05-05 09:44:30.990214] D [socket.c:4021:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2017-05-05 09:44:30.990219] D [socket.c:4024:socket_init] 0-management: SSL support for glusterd is NOT enabled [2017-05-05 09:44:30.990222] D [socket.c:4041:socket_init] 0-management: using system polling thread [2017-05-05 09:44:30.990228] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-management: attempting reconnect [2017-05-05 09:44:30.990232] T [socket.c:2953:socket_connect] 0-management: connecting 0x7f1bb0004e70, state=0 gen=0 sock=-1 [2017-05-05 09:44:30.990240] D [name.c:168:client_fill_address_family] 0-management: address-family not specified, marking it as unspec for getaddrinfo to resolve from (remote-host: 10.0.50.125) [2017-05-05 09:44:30.990247] T [MSGID: 0] [common-utils.c:281:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: 10.0.50.125 [2017-05-05 09:44:30.990461] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-10.0.50.125 (port-24007) for hostname: 10.0.50.125 and port: 24007 [2017-05-05 09:44:30.990469] D [socket.c:2899:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2017-05-05 09:44:30.990479] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 8 [2017-05-05 09:44:30.990484] T [socket.c:906:__socket_keepalive] 0-management: Keep-alive enabled for socket 8, interval 2, idle: 10, timeout: -1 [2017-05-05 09:44:30.990527] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 24007, higher: 24008 [2017-05-05 09:44:30.990533] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 32765, higher: -1 [2017-05-05 09:44:30.990537] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: -1, higher: -1 pending frames: frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2017-05-05 09:44:30 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.8.11 [2017-05-05 09:53:33.053153] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0) [2017-05-05 09:53:33.053181] T [MSGID: 0] [glusterfsd.c:573:create_fuse_mount] 0-glusterfsd: mount point not found, not a client process [2017-05-05 09:53:33.058697] D [MSGID: 0] [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/glusterd.pid updated with pid 14453 [2017-05-05 09:53:33.058731] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:53:33.058825] D [MSGID: 0] [glusterfsd.c:660:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2017-05-05 09:53:33.058931] T [MSGID: 0] [graph.y:186:new_volume] 0-parser: New node for 'management' [2017-05-05 09:53:33.058946] T [MSGID: 0] [xlator.c:193:xlator_dynload] 0-xlator: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so [2017-05-05 09:53:33.060355] T [MSGID: 0] [xlator.c:244:xlator_dynload] 0-xlator: dlsym(reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so: undefined symbol: reconfigure -- neglecting [2017-05-05 09:53:33.060376] T [MSGID: 0] [graph.y:217:volume_type] 0-parser: Type:management:mgmt/glusterd [2017-05-05 09:53:33.060389] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:working-directory:/var/lib/glusterd [2017-05-05 09:53:33.060398] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport-type:socket,rdma [2017-05-05 09:53:33.060405] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-time:10 [2017-05-05 09:53:33.060411] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-interval:2 [2017-05-05 09:53:33.060417] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.read-fail-log:off [2017-05-05 09:53:33.060423] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:ping-timeout:0 [2017-05-05 09:53:33.060428] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:event-threads:1 [2017-05-05 09:53:33.060434] T [MSGID: 0] [graph.y:323:volume_end] 0-parser: end:management [2017-05-05 09:53:33.060497] I [MSGID: 106478] [glusterd.c:1381:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-05-05 09:53:33.060514] I [MSGID: 106479] [glusterd.c:1430:init] 0-management: Using /var/lib/glusterd as working directory [2017-05-05 09:53:33.060548] D [MSGID: 0] [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 128 [2017-05-05 09:53:33.060560] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 64 [2017-05-05 09:53:33.060631] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:53:33.060637] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:53:33.060641] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:53:33.060645] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:53:33.060652] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:53:33.060659] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:53:33.060672] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:53:33.062758] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.listen-backlog 128' [2017-05-05 09:53:33.062778] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-05 09:53:33.062785] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-05 09:53:33.062797] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:53:33.062805] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:53:33.062809] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:53:33.062813] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:53:33.062819] D [name.c:584:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2017-05-05 09:53:33.062832] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 10 [2017-05-05 09:53:33.062904] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so [2017-05-05 09:53:33.063143] D [rpc-transport.c:321:rpc_transport_load] 0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so: undefined symbol: reconfigure [2017-05-05 09:53:33.063178] T [MSGID: 0] [rdma.c:4560:gf_rdma_options_init] 0-rpc-transport/rdma: rdma.management: defaulting MTU to '2048' [2017-05-05 09:53:33.063207] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-05-05 09:53:33.063214] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2017-05-05 09:53:33.063218] W [rpc-transport.c:354:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-05-05 09:53:33.063247] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-05-05 09:53:33.063254] E [MSGID: 106243] [glusterd.c:1654:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-05-05 09:53:33.063259] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2017-05-05 09:53:33.063263] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:53:33.063267] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2017-05-05 09:53:33.063275] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2017-05-05 09:53:33.063279] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2017-05-05 09:53:33.063283] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:53:33.063287] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2017-05-05 09:53:33.063296] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 8 [2017-05-05 09:53:33.063310] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:53:33.063314] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:53:33.063318] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:53:33.063322] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:53:33.063325] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:53:33.063329] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:53:33.063337] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:53:33.063358] D [socket.c:3887:socket_init] 0-socket.management: disabling nodelay [2017-05-05 09:53:33.063362] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:53:33.063367] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:53:33.063372] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:53:33.063375] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:53:33.063405] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:53:33.063412] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:53:33.063435] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-05 09:53:33.063456] D [MSGID: 0] [glusterd.c:1722:init] 0-management: cannot get run-with-valgrind value [2017-05-05 09:53:33.115752] D [MSGID: 0] [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 [2017-05-05 09:53:33.115790] D [MSGID: 0] [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 [2017-05-05 09:53:34.406695] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:53:34.406720] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:53:34.406727] D [MSGID: 0] [glusterd-store.c:1985:glusterd_retrieve_op_version] 0-management: Unable to get store handle! [2017-05-05 09:53:34.406734] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:53:34.406739] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:53:34.406743] D [MSGID: 0] [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get storehandle! [2017-05-05 09:53:34.406748] D [MSGID: 0] [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 [2017-05-05 09:53:34.406753] I [MSGID: 106514] [glusterd-store.c:2120:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30800 [2017-05-05 09:53:34.406780] D [MSGID: 0] [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: Returning with 0 [2017-05-05 09:53:34.406808] D [MSGID: 0] [glusterd-store.c:4257:glusterd_store_retrieve_peers] 0-management: Returning with 0 [2017-05-05 09:53:34.406825] I [MSGID: 106194] [glusterd-store.c:3628:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-05-05 09:53:34.406830] T [MSGID: 0] [glusterd-store.c:3687:glusterd_store_retrieve_missed_snaps_list] 0-management: Returning with 0 [2017-05-05 09:53:34.406835] D [MSGID: 0] [glusterd-store.c:3748:glusterd_store_retrieve_snaps] 0-management: Returning with 0 [2017-05-05 09:53:34.406839] T [MSGID: 0] [glusterd-store.c:4425:glusterd_resolve_all_bricks] 0-management: Returning with 0 [2017-05-05 09:53:34.406847] T [MSGID: 0] [glusterd-store.c:4379:glusterd_snap_cleanup] 0-management: Returning with 0 [2017-05-05 09:53:34.406851] T [MSGID: 0] [glusterd-store.c:4311:glusterd_recreate_all_snap_brick_mounts] 0-management: Returning with 0 [2017-05-05 09:53:34.406855] D [MSGID: 0] [glusterd-store.c:4478:glusterd_restore] 0-management: Returning 0 [2017-05-05 09:53:34.406956] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7f4faa71e800, remaining locks=0 [2017-05-05 09:53:34.408230] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-05 09:53:34.408240] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning 0 [2017-05-05 09:53:34.408251] D [MSGID: 0] [store.c:500:gf_store_iter_new] 0-: Returning with 0 [2017-05-05 09:53:34.408264] D [MSGID: 0] [store.c:613:gf_store_iter_get_next] 0-: Returning with 0 [2017-05-05 09:53:34.408272] D [MSGID: 0] [store.c:613:gf_store_iter_get_next] 0-: Returning with -1 [2017-05-05 09:53:34.408283] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-05 09:53:34.408312] D [MSGID: 0] [options.c:1214:xlator_option_init_int32] 0-management: option event-threads using set value 1 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-05-05 09:53:34.408852] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 09:55:33.084389] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:55:33.084426] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:57:33.104320] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:57:33.104362] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:59:33.124070] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:59:33.124146] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 10:01:33.142792] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 10:01:33.142892] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 10:01:54.656669] D [socket.c:559:__socket_rwv] 0-socket.management: EOF on socket [2017-05-05 10:01:54.656689] D [socket.c:2403:socket_event_handler] 0-transport: disconnecting now [2017-05-05 10:01:54.656701] T [socket.c:4267:fini] 0-socket.management: transport 0x7f4f94000920 destroyed [2017-05-05 10:01:54.656712] D [MSGID: 0] [event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=2 from gen=1 to slot->gen=2, fd=7, slot->fd=7 [2017-05-05 10:02:37.645908] T [rpcsvc-auth.c:404:rpcsvc_auth_request_init] 0-rpc-service: Auth handler: AUTH_GLUSTERFS-v2 [2017-05-05 10:02:37.645925] T [rpcsvc.c:490:rpcsvc_request_create] 0-rpc-service: received rpc-message (XID: 0x1, Ver: 2, Program: 1238463, ProgVers: 2, Proc: 1) from rpc-transport (socket.management) [2017-05-05 10:02:37.645945] T [auth-glusterfs.c:237:auth_glusterfs_v2_authenticate] 0-rpc-service: Auth Info: pid: 0, uid: 0, gid: 0, owner: 00000000 [2017-05-05 10:02:37.645951] T [rpcsvc.c:311:rpcsvc_program_actor] 0-rpc-service: Actor found: GlusterD svc cli - CLI_PROBE for [2017-05-05 10:02:37.646031] I [MSGID: 106487] [glusterd-handler.c:1241:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.0.50.125 24007 [2017-05-05 10:02:37.646050] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.125 [2017-05-05 10:02:37.646140] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.125 [2017-05-05 10:02:37.646179] D [MSGID: 0] [common-utils.c:3341:gf_is_local_addr] 0-management: 10.0.50.125 [2017-05-05 10:02:37.646214] D [MSGID: 0] [common-utils.c:3357:gf_is_local_addr] 0-management: 10.0.50.125 is not local [2017-05-05 10:02:37.646249] D [MSGID: 0] [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.50.125 [2017-05-05 10:02:37.646266] D [MSGID: 0] [glusterd-peer-utils.c:133:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: 10.0.50.125 [2017-05-05 10:02:37.646272] D [MSGID: 0] [glusterd-peer-utils.c:247:glusterd_peerinfo_find] 0-management: Unable to find hostname: 10.0.50.125 [2017-05-05 10:02:37.646279] I [MSGID: 106129] [glusterd-handler.c:3670:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 10.0.50.125 (24007) [2017-05-05 10:02:37.646287] D [MSGID: 0] [glusterd-peer-utils.c:487:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2017-05-05 10:02:37.646293] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-05 10:02:37.646304] D [MSGID: 0] [glusterd-store.c:3943:glusterd_store_create_peer_dir] 0-glusterd: Returning with 0 [2017-05-05 10:02:37.648199] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-05 10:02:37.648250] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 10:02:37.648264] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 10:02:37.648275] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 10:02:37.648280] D [MSGID: 0] [glusterd-store.c:4072:glusterd_store_peer_write] 0-glusterd: Returning with 0 [2017-05-05 10:02:37.652221] D [MSGID: 0] [glusterd-store.c:4098:glusterd_store_perform_peer_store] 0-glusterd: Returning 0 [2017-05-05 10:02:37.652233] D [MSGID: 0] [glusterd-store.c:4119:glusterd_store_peerinfo] 0-glusterd: Returning with 0 [2017-05-05 10:02:37.652249] D [MSGID: 0] [glusterd-handler.c:3461:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2017-05-05 10:02:37.652265] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-05-05 10:02:37.652272] D [rpc-clnt.c:1058:rpc_clnt_connection_init] 0-management: setting ping-timeout to 0 [2017-05-05 10:02:37.652279] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 10:02:37.652303] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.tcp-user-timeout -1' [2017-05-05 10:02:37.652310] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-05 10:02:37.652316] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-05 10:02:37.652324] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option remote-port 24007' [2017-05-05 10:02:37.652331] D [socket.c:3938:socket_init] 0-management: Configued transport.tcp-user-timeout=-1 [2017-05-05 10:02:37.652337] D [socket.c:4021:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2017-05-05 10:02:37.652342] D [socket.c:4024:socket_init] 0-management: SSL support for glusterd is NOT enabled [2017-05-05 10:02:37.652346] D [socket.c:4041:socket_init] 0-management: using system polling thread [2017-05-05 10:02:37.652353] T [rpc-clnt.c:422:rpc_clnt_reconnect] 0-management: attempting reconnect [2017-05-05 10:02:37.652357] T [socket.c:2953:socket_connect] 0-management: connecting 0x7f4f84004e70, state=0 gen=0 sock=-1 [2017-05-05 10:02:37.652362] D [name.c:168:client_fill_address_family] 0-management: address-family not specified, marking it as unspec for getaddrinfo to resolve from (remote-host: 10.0.50.125) [2017-05-05 10:02:37.652368] T [MSGID: 0] [common-utils.c:281:gf_resolve_ip6] 0-resolver: DNS cache not present, freshly probing hostname: 10.0.50.125 [2017-05-05 10:02:37.652619] D [MSGID: 0] [common-utils.c:321:gf_resolve_ip6] 0-resolver: returning ip-10.0.50.125 (port-24007) for hostname: 10.0.50.125 and port: 24007 [2017-05-05 10:02:37.652628] D [socket.c:2899:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2017-05-05 10:02:37.652639] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 8 [2017-05-05 10:02:37.652644] T [socket.c:906:__socket_keepalive] 0-management: Keep-alive enabled for socket 8, interval 2, idle: 10, timeout: -1 [2017-05-05 10:02:37.652744] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 24007, higher: 24008 [2017-05-05 10:02:37.652762] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: 32765, higher: -1 [2017-05-05 10:02:37.652767] D [MSGID: 0] [common-utils.c:3086:gf_ports_reserved] 0-glusterfs: lower: -1, higher: -1 pending frames: frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2017-05-05 10:02:37 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.8.11
# 10.0.50.125 /var/logs/glusterfs/etc-glusterfs-glusterd.vol.log [2017-05-05 09:43:56.035300] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0) [2017-05-05 09:43:56.035327] T [MSGID: 0] [glusterfsd.c:573:create_fuse_mount] 0-glusterfsd: mount point not found, not a client process [2017-05-05 09:43:56.039856] D [MSGID: 0] [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/glusterd.pid updated with pid 13600 [2017-05-05 09:43:56.039889] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:43:56.039979] D [MSGID: 0] [glusterfsd.c:660:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2017-05-05 09:43:56.040077] T [MSGID: 0] [graph.y:186:new_volume] 0-parser: New node for 'management' [2017-05-05 09:43:56.040092] T [MSGID: 0] [xlator.c:193:xlator_dynload] 0-xlator: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so [2017-05-05 09:43:56.045814] T [MSGID: 0] [xlator.c:244:xlator_dynload] 0-xlator: dlsym(reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so: undefined symbol: reconfigure -- neglecting [2017-05-05 09:43:56.045850] T [MSGID: 0] [graph.y:217:volume_type] 0-parser: Type:management:mgmt/glusterd [2017-05-05 09:43:56.045867] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:working-directory:/var/lib/glusterd [2017-05-05 09:43:56.045877] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport-type:socket,rdma [2017-05-05 09:43:56.045883] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-time:10 [2017-05-05 09:43:56.045888] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-interval:2 [2017-05-05 09:43:56.045893] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.read-fail-log:off [2017-05-05 09:43:56.045898] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:ping-timeout:0 [2017-05-05 09:43:56.045903] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:event-threads:1 [2017-05-05 09:43:56.045908] T [MSGID: 0] [graph.y:323:volume_end] 0-parser: end:management [2017-05-05 09:43:56.045975] I [MSGID: 106478] [glusterd.c:1381:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-05-05 09:43:56.045990] I [MSGID: 106479] [glusterd.c:1430:init] 0-management: Using /var/lib/glusterd as working directory [2017-05-05 09:43:56.046152] D [MSGID: 0] [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 128 [2017-05-05 09:43:56.046167] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 64 [2017-05-05 09:43:56.046229] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:43:56.046234] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:43:56.046238] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:43:56.046242] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:43:56.046245] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:43:56.046252] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:43:56.046268] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:43:56.048206] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.listen-backlog 128' [2017-05-05 09:43:56.048229] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-05 09:43:56.048240] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-05 09:43:56.048252] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:43:56.048260] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:43:56.048264] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:43:56.048267] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:43:56.048274] D [name.c:584:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2017-05-05 09:43:56.048287] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 10 [2017-05-05 09:43:56.048352] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so [2017-05-05 09:43:56.048625] D [rpc-transport.c:321:rpc_transport_load] 0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so: undefined symbol: reconfigure [2017-05-05 09:43:56.048660] T [MSGID: 0] [rdma.c:4560:gf_rdma_options_init] 0-rpc-transport/rdma: rdma.management: defaulting MTU to '2048' [2017-05-05 09:43:56.054678] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-05-05 09:43:56.054690] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2017-05-05 09:43:56.054695] W [rpc-transport.c:354:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-05-05 09:43:56.054728] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-05-05 09:43:56.054737] E [MSGID: 106243] [glusterd.c:1654:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-05-05 09:43:56.054742] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2017-05-05 09:43:56.054746] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:43:56.054750] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2017-05-05 09:43:56.054754] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2017-05-05 09:43:56.054758] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2017-05-05 09:43:56.054761] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:43:56.054765] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2017-05-05 09:43:56.054774] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 8 [2017-05-05 09:43:56.054788] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:43:56.054792] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:43:56.054796] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:43:56.054799] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:43:56.054802] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:43:56.054806] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:43:56.054818] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:43:56.054839] D [socket.c:3887:socket_init] 0-socket.management: disabling nodelay [2017-05-05 09:43:56.054844] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:43:56.054849] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:43:56.054853] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:43:56.054856] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:43:56.054889] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:43:56.054896] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:43:56.054915] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-05 09:43:56.054922] D [MSGID: 0] [glusterd.c:1722:init] 0-management: cannot get run-with-valgrind value [2017-05-05 09:43:56.515522] D [MSGID: 0] [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 [2017-05-05 09:43:56.515610] D [MSGID: 0] [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 [2017-05-05 09:43:59.414037] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:43:59.414060] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:43:59.414067] D [MSGID: 0] [glusterd-store.c:1985:glusterd_retrieve_op_version] 0-management: Unable to get store handle! [2017-05-05 09:43:59.414074] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:43:59.414079] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:43:59.414083] D [MSGID: 0] [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get storehandle! [2017-05-05 09:43:59.414087] D [MSGID: 0] [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 [2017-05-05 09:43:59.414092] I [MSGID: 106514] [glusterd-store.c:2120:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30800 [2017-05-05 09:43:59.414115] D [MSGID: 0] [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: Returning with 0 [2017-05-05 09:43:59.414144] D [MSGID: 0] [glusterd-store.c:4257:glusterd_store_retrieve_peers] 0-management: Returning with 0 [2017-05-05 09:43:59.414163] I [MSGID: 106194] [glusterd-store.c:3628:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-05-05 09:43:59.414169] T [MSGID: 0] [glusterd-store.c:3687:glusterd_store_retrieve_missed_snaps_list] 0-management: Returning with 0 [2017-05-05 09:43:59.414173] D [MSGID: 0] [glusterd-store.c:3748:glusterd_store_retrieve_snaps] 0-management: Returning with 0 [2017-05-05 09:43:59.414178] T [MSGID: 0] [glusterd-store.c:4425:glusterd_resolve_all_bricks] 0-management: Returning with 0 [2017-05-05 09:43:59.414184] T [MSGID: 0] [glusterd-store.c:4379:glusterd_snap_cleanup] 0-management: Returning with 0 [2017-05-05 09:43:59.414189] T [MSGID: 0] [glusterd-store.c:4311:glusterd_recreate_all_snap_brick_mounts] 0-management: Returning with 0 [2017-05-05 09:43:59.414193] D [MSGID: 0] [glusterd-store.c:4478:glusterd_restore] 0-management: Returning 0 [2017-05-05 09:43:59.414225] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] [2017-05-05 09:43:59.414232] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:43:59.414237] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-05 09:43:59.414288] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7f56de91f800, remaining locks=0 [2017-05-05 09:43:59.416193] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-05 09:43:59.416237] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 09:43:59.419580] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-05 09:43:59.419619] D [MSGID: 0] [options.c:1214:xlator_option_init_int32] 0-management: option event-threads using set value 1 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-05-05 09:43:59.419713] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 09:45:56.060586] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:45:56.060657] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:47:56.079709] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:47:56.079783] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:49:56.099246] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:49:56.099304] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:51:56.118550] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:51:56.118601] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:53:56.137978] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:53:56.138043] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:55:56.157222] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 09:55:56.157283] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:57:37.713968] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f56dc321184] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f56dd085c45] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x7f56dd085acb] ) 0-: received signum (15), shutting down [2017-05-05 09:57:37.714016] D [glusterfsd-mgmt.c:2385:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given [2017-05-05 09:57:37.714032] T [MSGID: 0] [glusterfsd.c:2016:glusterfs_pidfile_cleanup] 0-glusterfsd: pidfile /var/run/glusterd.pid cleanup [2017-05-05 09:57:37.714936] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:57:37.715081] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:57:37.715107] D [MSGID: 0] [glusterd.c:1160:glusterd_stop_listener] 0-management: glusterd_stop_listener function called [2017-05-05 09:57:37.715224] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2017-05-05 09:57:37.715341] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:57:37.715516] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2017-05-05 09:57:37.715639] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2017-05-05 09:57:37.715754] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2017-05-05 09:57:37.715868] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:57:37.715986] D [rpcsvc.c:1578:rpcsvc_program_unregister] 0-rpc-service: Program unregistered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2017-05-05 09:58:59.505677] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.11 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level=TRACE --log-buf-size=0) [2017-05-05 09:58:59.505706] T [MSGID: 0] [glusterfsd.c:573:create_fuse_mount] 0-glusterfsd: mount point not found, not a client process [2017-05-05 09:58:59.511561] D [MSGID: 0] [glusterfsd.c:2072:glusterfs_pidfile_update] 0-glusterfsd: pidfile /var/run/glusterd.pid updated with pid 15170 [2017-05-05 09:58:59.511597] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 09:58:59.511718] D [MSGID: 0] [glusterfsd.c:660:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2017-05-05 09:58:59.511850] T [MSGID: 0] [graph.y:186:new_volume] 0-parser: New node for 'management' [2017-05-05 09:58:59.511867] T [MSGID: 0] [xlator.c:193:xlator_dynload] 0-xlator: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so [2017-05-05 09:58:59.513326] T [MSGID: 0] [xlator.c:244:xlator_dynload] 0-xlator: dlsym(reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/mgmt/glusterd.so: undefined symbol: reconfigure -- neglecting [2017-05-05 09:58:59.513351] T [MSGID: 0] [graph.y:217:volume_type] 0-parser: Type:management:mgmt/glusterd [2017-05-05 09:58:59.513368] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:working-directory:/var/lib/glusterd [2017-05-05 09:58:59.513379] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport-type:socket,rdma [2017-05-05 09:58:59.513386] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-time:10 [2017-05-05 09:58:59.513392] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.keepalive-interval:2 [2017-05-05 09:58:59.513398] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:transport.socket.read-fail-log:off [2017-05-05 09:58:59.513403] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:ping-timeout:0 [2017-05-05 09:58:59.513409] T [MSGID: 0] [graph.y:252:volume_option] 0-parser: Option:management:event-threads:1 [2017-05-05 09:58:59.513414] T [MSGID: 0] [graph.y:323:volume_end] 0-parser: end:management [2017-05-05 09:58:59.513491] I [MSGID: 106478] [glusterd.c:1381:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-05-05 09:58:59.513507] I [MSGID: 106479] [glusterd.c:1430:init] 0-management: Using /var/lib/glusterd as working directory [2017-05-05 09:58:59.513737] D [MSGID: 0] [glusterd.c:408:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 128 [2017-05-05 09:58:59.513762] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 64 [2017-05-05 09:58:59.513838] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:58:59.513843] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:58:59.513847] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:58:59.513851] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:58:59.513854] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:58:59.513862] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:58:59.513878] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:58:59.515904] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.listen-backlog 128' [2017-05-05 09:58:59.515929] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-interval 2' [2017-05-05 09:58:59.515936] T [MSGID: 0] [options.c:86:xlator_option_validate_int] 0-management: no range check required for 'option transport.socket.keepalive-time 10' [2017-05-05 09:58:59.515948] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:58:59.515956] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:58:59.515960] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:58:59.515963] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:58:59.515970] D [name.c:584:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2017-05-05 09:58:59.515985] T [socket.c:831:__socket_nodelay] 0-management: NODELAY enabled for socket 10 [2017-05-05 09:58:59.516083] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so [2017-05-05 09:58:59.516357] D [rpc-transport.c:321:rpc_transport_load] 0-rpc-transport: dlsym (gf_rpc_transport_reconfigure) on /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/rdma.so: undefined symbol: reconfigure [2017-05-05 09:58:59.516398] T [MSGID: 0] [rdma.c:4560:gf_rdma_options_init] 0-rpc-transport/rdma: rdma.management: defaulting MTU to '2048' [2017-05-05 09:58:59.516439] W [MSGID: 103071] [rdma.c:4589:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-05-05 09:58:59.516448] W [MSGID: 103055] [rdma.c:4896:init] 0-rdma.management: Failed to initialize IB Device [2017-05-05 09:58:59.516452] W [rpc-transport.c:354:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-05-05 09:58:59.516485] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-05-05 09:58:59.516493] E [MSGID: 106243] [glusterd.c:1654:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-05-05 09:58:59.516499] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2017-05-05 09:58:59.516503] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:58:59.516507] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2017-05-05 09:58:59.516510] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2017-05-05 09:58:59.516519] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2017-05-05 09:58:59.516523] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:58:59.516527] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2017-05-05 09:58:59.516536] T [rpcsvc.c:2326:rpcsvc_init] 0-rpc-service: rx pool: 8 [2017-05-05 09:58:59.516551] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS [2017-05-05 09:58:59.516556] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_GLUSTERFS-v2 [2017-05-05 09:58:59.516559] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_UNIX [2017-05-05 09:58:59.516563] T [rpcsvc-auth.c:119:rpcsvc_auth_init_auth] 0-rpc-service: Authentication enabled: AUTH_NULL [2017-05-05 09:58:59.516566] D [rpcsvc.c:2345:rpcsvc_init] 0-rpc-service: RPC service inited. [2017-05-05 09:58:59.516569] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2017-05-05 09:58:59.516578] D [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/rpc-transport/socket.so [2017-05-05 09:58:59.516600] D [socket.c:3887:socket_init] 0-socket.management: disabling nodelay [2017-05-05 09:58:59.516606] D [socket.c:3938:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [2017-05-05 09:58:59.516610] D [socket.c:4021:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2017-05-05 09:58:59.516614] D [socket.c:4024:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2017-05-05 09:58:59.516618] D [socket.c:4041:socket_init] 0-socket.management: using system polling thread [2017-05-05 09:58:59.516651] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2017-05-05 09:58:59.516659] D [rpcsvc.c:1895:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2017-05-05 09:58:59.516679] D [MSGID: 0] [glusterd-utils.c:6384:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [2017-05-05 09:58:59.516687] D [MSGID: 0] [glusterd.c:1722:init] 0-management: cannot get run-with-valgrind value [2017-05-05 09:58:59.573447] D [MSGID: 0] [glusterd.c:467:glusterd_check_gsync_present] 0-glusterd: Returning 0 [2017-05-05 09:58:59.573499] D [MSGID: 0] [glusterd.c:603:glusterd_crt_georep_folders] 0-glusterd: Returning 0 [2017-05-05 09:59:00.833263] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:59:00.833286] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:59:00.833292] D [MSGID: 0] [glusterd-store.c:1985:glusterd_retrieve_op_version] 0-management: Unable to get store handle! [2017-05-05 09:59:00.833299] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [2017-05-05 09:59:00.833304] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:59:00.833308] D [MSGID: 0] [glusterd-store.c:2154:glusterd_retrieve_uuid] 0-management: Unable to get storehandle! [2017-05-05 09:59:00.833312] D [MSGID: 0] [glusterd-store.c:2173:glusterd_retrieve_uuid] 0-management: Returning -1 [2017-05-05 09:59:00.833317] I [MSGID: 106514] [glusterd-store.c:2120:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30800 [2017-05-05 09:59:00.833339] D [MSGID: 0] [glusterd-store.c:3190:glusterd_store_retrieve_volumes] 0-management: Returning with 0 [2017-05-05 09:59:00.833372] D [MSGID: 0] [glusterd-store.c:4257:glusterd_store_retrieve_peers] 0-management: Returning with 0 [2017-05-05 09:59:00.833391] I [MSGID: 106194] [glusterd-store.c:3628:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-05-05 09:59:00.833397] T [MSGID: 0] [glusterd-store.c:3687:glusterd_store_retrieve_missed_snaps_list] 0-management: Returning with 0 [2017-05-05 09:59:00.833401] D [MSGID: 0] [glusterd-store.c:3748:glusterd_store_retrieve_snaps] 0-management: Returning with 0 [2017-05-05 09:59:00.833405] T [MSGID: 0] [glusterd-store.c:4425:glusterd_resolve_all_bricks] 0-management: Returning with 0 [2017-05-05 09:59:00.833412] T [MSGID: 0] [glusterd-store.c:4379:glusterd_snap_cleanup] 0-management: Returning with 0 [2017-05-05 09:59:00.833416] T [MSGID: 0] [glusterd-store.c:4311:glusterd_recreate_all_snap_brick_mounts] 0-management: Returning with 0 [2017-05-05 09:59:00.833420] D [MSGID: 0] [glusterd-store.c:4478:glusterd_restore] 0-management: Returning 0 [2017-05-05 09:59:00.833450] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] [2017-05-05 09:59:00.833457] D [MSGID: 0] [store.c:438:gf_store_handle_retrieve] 0-: Returning -1 [2017-05-05 09:59:00.833462] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-05 09:59:00.833528] T [MSGID: 0] [syncop.c:1006:__synclock_unlock] 0-: Unlock success 0x7fcaaca59800, remaining locks=0 [2017-05-05 09:59:00.835579] D [MSGID: 0] [store.c:420:gf_store_handle_new] 0-: Returning 0 [2017-05-05 09:59:00.835665] D [MSGID: 0] [store.c:374:gf_store_save_value] 0-management: returning: 0 [2017-05-05 09:59:00.839234] D [MSGID: 0] [store.c:459:gf_store_handle_destroy] 0-: Returning 0 [2017-05-05 09:59:00.839269] D [MSGID: 0] [options.c:1214:xlator_option_init_int32] 0-management: option event-threads using set value 1 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-05-05 09:59:00.839739] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-05-05 10:00:59.536641] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 10:00:59.536672] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0 [2017-05-05 10:02:59.555415] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present [2017-05-05 10:02:59.555488] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 0
Created attachment 1276539 [details] apport crash bundle with some symbols and source
While debugging core file attached by you, I find that it does not loads any symbols. Hence I am unable to debug with the core you have attached. Could you please help me in recreating the issue locally or provide the setup where issue is been seen. It will be easy in either of the case to debug the issue. If both are not possible, in such case I need your help in debugging the core file. As I have already mentioned that core attached by you does not loads any symbol and I am not able to see the back trace as well..
Gaurav, I can grant you access to EC2 instances that are in this state. Is that acceptable? If so, please send me your SSH public key. Please look at https://bugzilla.redhat.com/attachment.cgi?id=1276539 ? Check out Stacktrace, StacktraceSource, and ThreadStacktrace.
To make it easier to debug, please install the `glusterfs-dbg` package, which should provide better information in the backtraces. Also, try to start glusterd with debug logs, either directly by running `glusterd -LDEBUG` or by modifying the init script. Doing the above should help get better logs and stacktraces, which will help you get to the cause faster.
Kaushal, 'glusterfs-dbg' is already installed and I've already modified the init scripts (upstart job in this case) to use DEBUG level logging.
Ben, Logs attached by you doesn't help much. I am not able to see the proper backtrace. In order to do RCA I need either reproducer or your host. Here is my SSH public key ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDWFZqzFVo7orVZx2ODZyok46VI6EqLg16uP2Z1pkMrEQGu50i3Ye16V5I63UMrHjDwdr4hxtvkW9UfhckQpgBwjsVg9xoyl9tuYt1h9au8G0hH2UL1XYWmbQt82N9VbeYGStg3n0VoefHNZ4LH/VINg0gBWtIK7iTQxWR6XOvs2QqOJnUnM+Fgu5b9kS9vPoDr93BxGLya2ijASkRxsi5dUN4qm7LgFX7Hsyh14G+BBouF5wDZ6frR/UPpqocBVJ5/n4f9OkhwMOShlkWm0m/JDcu6L0phL+Dqm9KxPHBEA/PFW3atjvJW70Iun+j1i72SCcMccQjHSPB6J5QYSeQb gyadav.eng.blr.redhat.com
Gaurav, I've provided the connection info to you in a direct email.
Running this command before peer probe reproduces this problem (leading to the backtrace handler problem) in all cases: sysctl net.ipv4.ip_local_reserved_ports="49152-49156" or sysctl net.ipv4.ip_local_reserved_ports="24007-24008,32765-32768,49152-49156" The issue appears to be with parsing the contents of '/proc/sys/net/ipv4/ip_local_reserved_ports' here: https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/common-utils.c#L3038 This option appears to defer to the kernel for source port selection. Is there a known issue with kernel port selection? https://github.com/gluster/glusterfs/blob/master/configure.ac#L312-L320 I'm going to build and test with the above configure option.
(In reply to Ben Werthmann from comment #22) > Running this command before peer probe reproduces this problem (leading to > the backtrace handler problem) in all cases: > > sysctl net.ipv4.ip_local_reserved_ports="49152-49156" > > or > > sysctl net.ipv4.ip_local_reserved_ports="24007-24008,32765-32768,49152-49156" > > The issue appears to be with parsing the contents of > '/proc/sys/net/ipv4/ip_local_reserved_ports' here: > > https://github.com/gluster/glusterfs/blob/master/libglusterfs/src/common- > utils.c#L3038 > > This option appears to defer to the kernel for source port selection. Is > there a known issue with kernel port selection? > > https://github.com/gluster/glusterfs/blob/master/configure.ac#L312-L320 This option is not in 3.8. > > I'm going to build and test with the above configure option.
Thanks Ben for providing the additional info, It helped me in finding the root cause of the issue. While parsing the ports we are not handling MIN MAX range properly hence glusterd is crashing.
Please backport this patch to to 3.8.
REVIEW: https://review.gluster.org/17505 (libglusterfs : Fix crash in glusterd while peer probing) posted (#1) for review on release-3.8 by Gaurav Yadav (gyadav)
COMMIT: https://review.gluster.org/17505 committed in release-3.8 by jiffin tony Thottan (jthottan) ------ commit ba5115bccdc08f395a88d682d3df4af0e4bbfb0a Author: Gaurav Yadav <gyadav> Date: Mon May 22 23:25:47 2017 +0530 libglusterfs : Fix crash in glusterd while peer probing glusterd crashes when port is being set explcitly to a range which is outside greater than short data type range. Eg. sysctl net.ipv4.ip_local_reserved_ports="49152-49156" In above case glusterd crashes while parsing the port. With this fix glusterd will be able to handle port range between INT_MIN to INT_MAX > Reviewed-on: https://review.gluster.org/17359 > Smoke: Gluster Build System <jenkins.org> > NetBSD-regression: NetBSD Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> > Reviewed-by: Samikshan Bairagya <samikshan> > Reviewed-by: Atin Mukherjee <amukherj> > Reviewed-by: Niels de Vos <ndevos> > Reviewed-by: Jeff Darcy <jeff.us> Change-Id: I7c75ee67937b0e3384502973d96b1c36c89e0fe1 BUG: 1447523 Signed-off-by: Gaurav Yadav <gyadav> Reviewed-on: https://review.gluster.org/17505 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Atin Mukherjee <amukherj>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.13, please open a new bug report. glusterfs-3.8.13 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2017-June/000075.html [2] https://www.gluster.org/pipermail/gluster-users/