Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1774712

Summary:	Brick process crashes after few days
Product:	[Community] GlusterFS	Reporter:	Sud <sudsingh>
Component:	glusterd	Assignee:	bugs <bugs>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6	CC:	bugs, pasik, srakonde, sudsingh
Target Milestone:	---	Flags:	sudsingh: needinfo+ sudsingh: needinfo+
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-12-26 05:52:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sud 2019-11-20 18:55:25 UTC

Description of problem:

Brick process crashed after few days.

Version-Release number of selected component (if applicable): 6.5
Brick Logs:
[2019-11-15 14:25:01.769891] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2019-11-15 17:33:39.466467] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:22077ebe-81d2-4afd-8fb2-84b350864033-GRAPH_ID:0-PID:632477-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 17:33:39.466682] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:22077ebe-81d2-4afd-8fb2-84b350864033-GRAPH_ID:0-PID:632477-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 17:33:49.173936] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.74"
[2019-11-15 17:33:49.173993] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:8972c9ce-edc9-4e79-94bd-f75919cabe2b-GRAPH_ID:0-PID:43417-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 17:33:49.228481] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:8972c9ce-edc9-4e79-94bd-f75919cabe2b-GRAPH_ID:0-PID:43417-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 17:33:49.228682] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:8972c9ce-edc9-4e79-94bd-f75919cabe2b-GRAPH_ID:0-PID:43417-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 17:33:49.321262] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.74"
[2019-11-15 17:33:49.321303] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:8032f65a-c523-4210-bce8-c9da8137c88e-GRAPH_ID:0-PID:43529-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 17:45:56.960791] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:8032f65a-c523-4210-bce8-c9da8137c88e-GRAPH_ID:0-PID:43529-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 17:45:56.960995] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:8032f65a-c523-4210-bce8-c9da8137c88e-GRAPH_ID:0-PID:43529-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 19:55:02.663281] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2019-11-15 19:55:02.665054] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2019-11-15 19:55:02.894968] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2019-11-15 19:55:02.921734] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2019-11-15 19:55:03.154098] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2019-11-15 19:55:03.179969] I [glusterfsd-mgmt.c:2019:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2019-11-15 19:55:03.330235] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:c9e49fb9-ca04-46cb-a722-191e2ea1f3d9-GRAPH_ID:0-PID:646084-HOST:gfs1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 19:55:03.330360] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:c9e49fb9-ca04-46cb-a722-191e2ea1f3d9-GRAPH_ID:0-PID:646084-HOST:gfs1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 19:55:05.354991] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.141"
[2019-11-15 19:55:05.355022] I [login.c:110:gf_auth] 0-auth/login: allowed user names: b79a4434-e9b8-468b-a5e8-b16f13e51525
[2019-11-15 19:55:05.355047] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:35a321ce-ad83-44a2-a796-6deead4f95be-GRAPH_ID:0-PID:684492-HOST:gfs1-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 19:55:05.398300] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:8494fe30-7113-4760-a559-649e9f9908a0-GRAPH_ID:0-PID:53600-HOST:gfs2-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 19:55:05.398439] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:8494fe30-7113-4760-a559-649e9f9908a0-GRAPH_ID:0-PID:53600-HOST:gfs2-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 19:55:05.399677] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:0a66757e-6d29-486f-b5ad-2c082cddf990-GRAPH_ID:0-PID:788227-HOST:gfs0-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 19:55:05.399764] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:0a66757e-6d29-486f-b5ad-2c082cddf990-GRAPH_ID:0-PID:788227-HOST:gfs0-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 19:55:07.421749] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.181"
[2019-11-15 19:55:07.421786] I [login.c:110:gf_auth] 0-auth/login: allowed user names: b79a4434-e9b8-468b-a5e8-b16f13e51525
[2019-11-15 19:55:07.421824] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:135504a8-877f-43a0-83d8-e9e79c71fa07-GRAPH_ID:0-PID:843611-HOST:gfs0-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 19:55:07.423987] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.121"
[2019-11-15 19:55:07.424014] I [login.c:110:gf_auth] 0-auth/login: allowed user names: b79a4434-e9b8-468b-a5e8-b16f13e51525
[2019-11-15 19:55:07.424030] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:f0c092a8-117b-4169-b159-17f462340638-GRAPH_ID:0-PID:106832-HOST:gfs2-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 20:01:17.797092] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.74"
[2019-11-15 20:01:17.797155] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:063f7a90-4dee-47ca-9bee-9eccb810b83c-GRAPH_ID:0-PID:64959-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 20:01:17.828340] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:063f7a90-4dee-47ca-9bee-9eccb810b83c-GRAPH_ID:0-PID:64959-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 20:01:17.828547] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:063f7a90-4dee-47ca-9bee-9eccb810b83c-GRAPH_ID:0-PID:64959-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 20:01:17.926493] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.74"
[2019-11-15 20:01:17.926581] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:6112e3eb-b22f-4fc0-8fb5-28781c5ea22b-GRAPH_ID:0-PID:65092-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 20:01:48.269565] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:6112e3eb-b22f-4fc0-8fb5-28781c5ea22b-GRAPH_ID:0-PID:65092-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 20:01:48.269782] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:6112e3eb-b22f-4fc0-8fb5-28781c5ea22b-GRAPH_ID:0-PID:65092-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 20:02:48.213821] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.74"
[2019-11-15 20:02:48.213895] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:5887ed16-8f13-4f2a-ad8e-6fe971c6f51e-GRAPH_ID:0-PID:77965-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-15 20:02:48.256466] I [MSGID: 115036] [server.c:499:server_rpc_notify] 0-test-server: disconnecting connection from CTX_ID:5887ed16-8f13-4f2a-ad8e-6fe971c6f51e-GRAPH_ID:0-PID:77965-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 20:02:48.256667] I [MSGID: 101055] [client_t.c:436:gf_client_unref] 0-test-server: Shutting down connection CTX_ID:5887ed16-8f13-4f2a-ad8e-6fe971c6f51e-GRAPH_ID:0-PID:77965-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0
[2019-11-15 20:02:48.351022] I [addr.c:54:compare_addr_and_update] 0-/var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick: allowed = "*", received addr = "10.180.72.74"
[2019-11-15 20:02:48.351069] I [MSGID: 115029] [server-handshake.c:550:server_setvolume] 0-test-server: accepted client from CTX_ID:37cb0888-7e3a-4bcc-98f4-fb0d6df068d8-GRAPH_ID:0-PID:78096-HOST:storage-datap-1-PC_NAME:test-client-0-RECON_NO:-0 (version: 6.5) with subvol /var/lib/heketi/mounts/vg_1281b98d059c2dbde3b0379fb0b284a6/brick_5419fbc483676eb04e13efaed26f55fb/brick
[2019-11-18 23:35:15.014118] E [MSGID: 101064] [event-epoll.c:618:event_dispatch_epoll_handler] 0-epoll: generation mismatch on idx=9, gen=10, slot->gen=11, slot->fd=16
[2019-11-18 23:35:15.116207] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong MSG-TYPE (9) received from 10.176.0.139:58950
[2019-11-18 23:35:15.255633] E [socket.c:1303:socket_event_poll_err] (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7f62c) [0x7f791f05e62c] -->/usr/lib/x86_64-linux-gnu/glusterfs/6.5/rpc-transport/socket.so(+0x9bbf) [0x7f7919a7abbf] -->/usr/lib/x86_64-linux-gnu/glusterfs/6.5/rpc-transport/socket.so(+0x79cb) [0x7f7919a789cb] ) 0-socket: invalid argument: this->private [Invalid argument]
[2019-11-18 23:35:24.203114] E [MSGID: 101064] [event-epoll.c:618:event_dispatch_epoll_handler] 0-epoll: generation mismatch on idx=9, gen=19, slot->gen=20, slot->fd=16
[2019-11-18 23:35:24.217318] E [MSGID: 101064] [event-epoll.c:618:event_dispatch_epoll_handler] 0-epoll: generation mismatch on idx=9, gen=22, slot->gen=23, slot->fd=16
[2019-11-18 23:35:28.497697] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong MSG-TYPE (-66911361) received from 10.176.0.139:58594
[2019-11-18 23:35:36.326252] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong MSG-TYPE (9109504) received from 10.176.0.139:51188
[2019-11-18 23:38:17.784843] C [rpcsvc.c:1029:rpcsvc_notify] 0-rpcsvc: got MAP_XID event, which should have not come
pending frames:
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash: 
2019-11-18 23:40:40
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 6.5
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x224ea)[0x7f791f0014ea]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2e7)[0x7f791f00bd17]
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0)[0x7f791e3cf4b0]
/usr/lib/x86_64-linux-gnu/glusterfs/6.5/rpc-transport/socket.so(+0x9c04)[0x7f7919a7ac04]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7f62c)[0x7f791f05e62c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f791e76b6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f791e4a141d]
---------
.

Comment 1 Sanju 2019-11-22 10:03:25 UTC

Please share output of "bt" and "t a a bt". If possible, please share the coredump with us.

Thanks,
Sanju

Comment 2 Sanju 2019-12-26 05:52:01 UTC

As there is sufficient data provided, closing this bug as insufficient data. Please, feel free to re-open the bug if you happen to hit this issue .