Bug 811206

can you paste the output of the 'gluster volume info' ?

# gluster volume info

Volume Name: xxxxxxx02
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: xxxxstorage102c.xx.xxxxxxx.co.jp:/brick02
Brick2: xxxxstorageb102c.xx.xxxxxxx.co.jp:/brick02
Options Reconfigured:
network.frame-timeout: 30
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
diagnostics.dump-fd-stats: off
performance.stat-prefetch: off
performance.io-thread-count: 64
performance.cache-size: 6GB
network.ping-timeout: 5
nfs.disable: on

Volume Name: xxxxxxx01
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: xxxxstorage101c.xx.xxxxxxx.co.jp:/brick01
Brick2: xxxxstorageb101c.xx.xxxxxxx.co.jp:/brick01
Options Reconfigured:
network.frame-timeout: 30
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
diagnostics.dump-fd-stats: off
performance.stat-prefetch: off
performance.io-thread-count: 64
performance.cache-size: 6GB
network.ping-timeout: 5
nfs.disable: on

I will send you a backtrace log.

(*) In case of normal processing (Server-A, Server-B)
(gdb) bt
#0  dict_unserialize (orig_buf=0x0, size=0, fill=0x7fffffffe118) at dict.c:2443
#1  0x000000000040f088 in gf_cli3_1_profile_volume_cbk (req=<value optimized out>, iov=0x2aaaabf5a064,
    count=<value optimized out>, myframe=<value optimized out>) at cli-rpc-ops.c:3039
#2  0x00002aaaaaf523b2 in rpc_clnt_handle_reply (clnt=0x622780, pollin=<value optimized out>) at rpc-clnt.c:736
#3  0x00002aaaaaf525ad in rpc_clnt_notify (trans=<value optimized out>, mydata=0x6227b0, event=<value optimized out>,
    data=0x36253529e8) at rpc-clnt.c:849
#4  0x00002aaaaaf4d287 in rpc_transport_notify (this=0x0, event=4294959384, data=0x36253529e8) at rpc-transport.c:918
#5  0x00002aaaac21feef in socket_event_poll_in (this=0x622940) at socket.c:1647
#6  0x00002aaaac220098 in socket_event_handler (fd=<value optimized out>, idx=0, data=0x622940, poll_in=1, poll_out=0, poll_err=0)
    at socket.c:1762
#7  0x00002aaaaaae9391 in event_dispatch_epoll_handler (event_pool=0x621350) at event.c:794
#8  event_dispatch_epoll (event_pool=0x621350) at event.c:856
#9  0x0000000000407602 in main (argc=5, argv=0x7fffffffea28) at cli.c:648

(*) In case of normal processing (Server-C, Server-D)
(gdb) bt
#0  dict_unserialize (orig_buf=0x8069c48 "", size=2123, fill=0xbfffcc48) at dict.c:2443
#1  0x0805608e in gf_cli3_1_profile_volume_cbk (req=0xb720e014, iov=0xb720e034, count=1, myframe=0xb748f088) at cli-rpc-ops.c:3039
#2  0x00f4d132 in rpc_clnt_handle_reply (clnt=0x8066de0, pollin=0x8068948) at rpc-clnt.c:736
#3  0x00f4d352 in rpc_clnt_notify (trans=0x8066ed0, mydata=0x8066dfc, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x8068b40) at rpc-clnt.c:849
#4  0x00f47421 in rpc_transport_notify (this=0x8068b40, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x8068948) at rpc-transport.c:918
#5  0x0011bc40 in socket_event_poll_in (this=0x8066ed0) at socket.c:1647
#6  0x0011bdc2 in socket_event_handler (fd=7, idx=0, data=0x8066ed0, poll_in=1, poll_out=0, poll_err=0) at socket.c:1762
#7  0x007acae7 in event_dispatch_epoll_handler (event_pool=0x80661e8) at event.c:794
#8  event_dispatch_epoll (event_pool=0x80661e8) at event.c:856
#9  0x007ab881 in event_dispatch (event_pool=0x8068b40) at event.c:956
#10 0x0804dd37 in main (argc=5, argv=0xbfffd494) at cli.c:648

(*) Comment
It seems orig_buf is null.
It seems not to get a fd at #6.
Please give us some advice for more investigation.

some of the memory tuning suggestions to check...

* reduce io-cache cache-size to 2GB or lesser (testing purpose)
* reduce the io-threads count to 8


The logs anyways suggests that the system's memory manager (malloc/calloc/free) itself returning ENOMEM (Failed to Allocate Memory) error.

Other good feedback would be to take 'statedump' of glusterfs/d (by sending signal SIGUSR1 to the process, which will be captured at /tmp/glusterdump.<pid>)

Hello.

Thank you for your advice.

This error is still continue.
We want to challenge reduce setting value but we can not be changed currently gluster's setting because of this error.
So we restarted OS and glusterd (server A) .
After that we could mount glusterfs but we still couldn't modify gluster's setting.

If we restart all server (at same time), we can solved this ?

Nakai, sorry for delay in getting back to you. Can you please check if this error happens now for you with 3.3.0 release?

I will be closing this bug as WORKSFORME, as in our testing in recent times, we never faced this issue. Re-open the bug if it still happens for you.

Hi,

We restarted all of the configuration node.
After that GFS didn't output same error messages.
Maybe resolved.
But we didn't retry execute profile command every 5 min,
Because I thought this profile command include some memory issue. 

Anyway thank you for your support.