Bug 811206
Summary: | Unable to allocate memory | ||
---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Nakai Kanako <nakai.kanako> |
Component: | cli | Assignee: | Amar Tumballi <amarts> |
Status: | CLOSED WORKSFORME | QA Contact: | |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 3.2.3 | CC: | gluster-bugs, vraman, yasuya.ichikawa |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2012-09-18 08:45:47 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Nakai Kanako
2012-04-10 12:24:14 UTC
can you paste the output of the 'gluster volume info' ? # gluster volume info Volume Name: xxxxxxx02 Type: Replicate Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: xxxxstorage102c.xx.xxxxxxx.co.jp:/brick02 Brick2: xxxxstorageb102c.xx.xxxxxxx.co.jp:/brick02 Options Reconfigured: network.frame-timeout: 30 diagnostics.latency-measurement: on diagnostics.count-fop-hits: on diagnostics.dump-fd-stats: off performance.stat-prefetch: off performance.io-thread-count: 64 performance.cache-size: 6GB network.ping-timeout: 5 nfs.disable: on Volume Name: xxxxxxx01 Type: Replicate Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: xxxxstorage101c.xx.xxxxxxx.co.jp:/brick01 Brick2: xxxxstorageb101c.xx.xxxxxxx.co.jp:/brick01 Options Reconfigured: network.frame-timeout: 30 diagnostics.latency-measurement: on diagnostics.count-fop-hits: on diagnostics.dump-fd-stats: off performance.stat-prefetch: off performance.io-thread-count: 64 performance.cache-size: 6GB network.ping-timeout: 5 nfs.disable: on I will send you a backtrace log. (*) In case of normal processing (Server-A, Server-B) (gdb) bt #0 dict_unserialize (orig_buf=0x0, size=0, fill=0x7fffffffe118) at dict.c:2443 #1 0x000000000040f088 in gf_cli3_1_profile_volume_cbk (req=<value optimized out>, iov=0x2aaaabf5a064, count=<value optimized out>, myframe=<value optimized out>) at cli-rpc-ops.c:3039 #2 0x00002aaaaaf523b2 in rpc_clnt_handle_reply (clnt=0x622780, pollin=<value optimized out>) at rpc-clnt.c:736 #3 0x00002aaaaaf525ad in rpc_clnt_notify (trans=<value optimized out>, mydata=0x6227b0, event=<value optimized out>, data=0x36253529e8) at rpc-clnt.c:849 #4 0x00002aaaaaf4d287 in rpc_transport_notify (this=0x0, event=4294959384, data=0x36253529e8) at rpc-transport.c:918 #5 0x00002aaaac21feef in socket_event_poll_in (this=0x622940) at socket.c:1647 #6 0x00002aaaac220098 in socket_event_handler (fd=<value optimized out>, idx=0, data=0x622940, poll_in=1, poll_out=0, poll_err=0) at socket.c:1762 #7 0x00002aaaaaae9391 in event_dispatch_epoll_handler (event_pool=0x621350) at event.c:794 #8 event_dispatch_epoll (event_pool=0x621350) at event.c:856 #9 0x0000000000407602 in main (argc=5, argv=0x7fffffffea28) at cli.c:648 (*) In case of normal processing (Server-C, Server-D) (gdb) bt #0 dict_unserialize (orig_buf=0x8069c48 "", size=2123, fill=0xbfffcc48) at dict.c:2443 #1 0x0805608e in gf_cli3_1_profile_volume_cbk (req=0xb720e014, iov=0xb720e034, count=1, myframe=0xb748f088) at cli-rpc-ops.c:3039 #2 0x00f4d132 in rpc_clnt_handle_reply (clnt=0x8066de0, pollin=0x8068948) at rpc-clnt.c:736 #3 0x00f4d352 in rpc_clnt_notify (trans=0x8066ed0, mydata=0x8066dfc, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x8068b40) at rpc-clnt.c:849 #4 0x00f47421 in rpc_transport_notify (this=0x8068b40, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x8068948) at rpc-transport.c:918 #5 0x0011bc40 in socket_event_poll_in (this=0x8066ed0) at socket.c:1647 #6 0x0011bdc2 in socket_event_handler (fd=7, idx=0, data=0x8066ed0, poll_in=1, poll_out=0, poll_err=0) at socket.c:1762 #7 0x007acae7 in event_dispatch_epoll_handler (event_pool=0x80661e8) at event.c:794 #8 event_dispatch_epoll (event_pool=0x80661e8) at event.c:856 #9 0x007ab881 in event_dispatch (event_pool=0x8068b40) at event.c:956 #10 0x0804dd37 in main (argc=5, argv=0xbfffd494) at cli.c:648 (*) Comment It seems orig_buf is null. It seems not to get a fd at #6. Please give us some advice for more investigation. some of the memory tuning suggestions to check... * reduce io-cache cache-size to 2GB or lesser (testing purpose) * reduce the io-threads count to 8 The logs anyways suggests that the system's memory manager (malloc/calloc/free) itself returning ENOMEM (Failed to Allocate Memory) error. Other good feedback would be to take 'statedump' of glusterfs/d (by sending signal SIGUSR1 to the process, which will be captured at /tmp/glusterdump.<pid>) Hello. Thank you for your advice. This error is still continue. We want to challenge reduce setting value but we can not be changed currently gluster's setting because of this error. So we restarted OS and glusterd (server A) . After that we could mount glusterfs but we still couldn't modify gluster's setting. If we restart all server (at same time), we can solved this ? Nakai, sorry for delay in getting back to you. Can you please check if this error happens now for you with 3.3.0 release? I will be closing this bug as WORKSFORME, as in our testing in recent times, we never faced this issue. Re-open the bug if it still happens for you. Hi, We restarted all of the configuration node. After that GFS didn't output same error messages. Maybe resolved. But we didn't retry execute profile command every 5 min, Because I thought this profile command include some memory issue. Anyway thank you for your support. |