| Summary: | nfs clients cause glusterfs-crash in 3.2.0 | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Christopher <christopher.anderlik> |
| Component: | glusterd | Assignee: | Vijay Bellur <vbellur> |
| Status: | CLOSED DUPLICATE | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.2.0 | CC: | christopher.anderlik, gluster-bugs, vijay |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | --- | Mount Type: | nfs |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
the traces were made line this: gdb /opt/glusterfs/3.2.0/sbin/glusterfsd --core core.33941 --batch --quiet -ex "thread apply all bt full" -ex "quit" > gfs-server-crash-full-stacktrace-3.txt please see also: http://gluster.org/pipermail/gluster-users/2011-June/007933.html http://gluster.org/pipermail/gluster-users/2011-June/007953.html additional information: glusterfs-server: ----------------- SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 0 glusterfs-core-3.2.0-1 glusterfs-rdma-3.2.0-1 glusterfs-fuse-3.2.0-1 nfs-client1: ----------- SUSE LINUX Enterprise Server 9 (x86_64) VERSION = 9 PATCHLEVEL = 3 yast2-nfs-client-2.9.11-23.2 nfs-client2: ------------ SUSE Linux Enterprise Server 10 (x86_64) VERSION = 10 PATCHLEVEL = 2 yast2-nfs-client-2.13.4-0.11 nfs clients cause glusterfs-crash in 3.2.0
-> all clients (also the glusterfs-clients) then see "transport endpoint not connected"
then you have to stop and start glusterserver.
volume community-client-0
type protocol/client
option remote-host 10.0.1.152
option remote-subvolume /gluster-storage/community
option transport-type tcp
end-volume
volume community-client-1
type protocol/client
option remote-host 10.0.1.153
option remote-subvolume /gluster-storage/community
option transport-type tcp
end-volume
volume community-replicate-0
type cluster/replicate
subvolumes community-client-0 community-client-1
end-volume
volume community-write-behind
type performance/write-behind
subvolumes community-replicate-0
end-volume
volume community-read-ahead
type performance/read-ahead
subvolumes community-write-behind
end-volume
volume community-io-cache
type performance/io-cache
subvolumes community-read-ahead
end-volume
volume community-quick-read
type performance/quick-read
subvolumes community-io-cache
end-volume
volume community-stat-prefetch
type performance/stat-prefetch
subvolumes community-quick-read
end-volume
volume community
type debug/io-stats
option latency-measurement off
option count-fop-hits off
subvolumes community-stat-prefetch
end-volume
here are the nfs-logs from glusterfs-server when crashing:
---------------------------------------
[2011-06-10 08:54:14.900049] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: f8851fc2, ACCESS: NFS: 0(Call completed successfully.), POSIX: 0(Success)
[2011-06-10 08:54:14.902002] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: f9851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:14.902037] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:14.902062] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: f9851fc2, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465
[2011-06-10 08:54:14.920579] D [afr-transaction.c:976:afr_post_nonblocking_inodelk_cbk] 0-ksc-replicate-0: Non blocking inodelks done. Proceeding to FOP
[2011-06-10 08:54:14.921099] D [client-lk.c:442:delete_granted_locks_fd] 0-ksc-client-0: Number of locks cleared=0
[2011-06-10 08:54:14.921155] D [client-lk.c:442:delete_granted_locks_fd] 0-ksc-client-1: Number of locks cleared=0
[2011-06-10 08:54:14.932629] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: f9851fc2, GETATTR: NFS: 0(Call completed successfully.), POSIX: 0(Success)
[2011-06-10 08:54:14.932863] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fa851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:14.932890] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:14.932907] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fa851fc2, ACCESS: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465
[2011-06-10 08:54:14.961700] D [socket.c:193:__socket_rwv] 0-ksc-client-0: EOF from peer 10.0.1.198:24031
[2011-06-10 08:54:14.961741] W [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.1.198:24031)
[2011-06-10 08:54:14.961757] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now
[2011-06-10 08:54:14.961858] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-0: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-10 08:54:14.920644
[2011-06-10 08:54:14.961880] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-0: remote operation failed: Transport endpoint is not connected
[2011-06-10 08:54:14.961880] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-0: remote operation failed: Transport endpoint is not connected
[2011-06-10 08:54:14.961940] I [client.c:1883:client_rpc_notify] 0-ksc-client-0: disconnected
[2011-06-10 08:54:14.988468] W [socket.c:204:__socket_rwv] 0-ksc-client-1: readv failed (Connection reset by peer)
[2011-06-10 08:54:14.988490] W [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-1: reading from socket failed. Error (Connection reset by peer), peer (10.0.1.199:24027)
[2011-06-10 08:54:14.988501] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now
[2011-06-10 08:54:14.988501] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now
[2011-06-10 08:54:14.988551] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(SETATTR(38)) called at 2011-06-10 08:54:14.920655
[2011-06-10 08:54:14.988568] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected
[2011-06-10 08:54:14.988568] I [client3_1-fops.c:1640:client3_1_setattr_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected
[2011-06-10 08:54:14.988599] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state
[2011-06-10 08:54:14.988630] W [client3_1-fops.c:4379:client3_1_xattrop] 0-ksc-client-0: failed to send the fop: Transport endpoint is not connected
[2011-06-10 08:54:14.988657] D [name.c:157:client_fill_address_family] 0-ksc-client-1: address-family not specified, guessing it to be inet/inet6
[2011-06-10 08:54:14.991719] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.0.1.199 (port-24007) for hostname: 10.0.1.199 and port: 24007
[2011-06-10 08:54:14.991786] I [socket.c:2272:socket_submit_request] 0-ksc-client-1: not connected (priv->connected = 0)
[2011-06-10 08:54:14.991803] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-ksc-client-1: failed to submit rpc-request (XID: 0x70357x Program: GlusterFS 3.1, ProgVers: 310, Proc: 33) to rpc-transport (ksc-client-1)
[2011-06-10 08:54:14.991819] D [afr-lk-common.c:409:transaction_lk_op] 0-ksc-replicate-0: lk op is for a transaction
[2011-06-10 08:54:14.991833] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state
[2011-06-10 08:54:14.991845] W [client3_1-fops.c:4735:client3_1_inodelk] 0-ksc-client-0: failed to send the fop: Transport endpoint is not connected
[2011-06-10 08:54:14.991862] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-ksc-client-1: failed to submit rpc-request (XID: 0x70358x Program: GlusterFS 3.1, ProgVers: 310, Proc: 29) to rpc-transport (ksc-client-1)
[2011-06-10 08:54:14.991876] I [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected
[2011-06-10 08:54:14.991904] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: c71a69ff, CREATE: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 4, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid f3e21f99-0aed-4042-8b81-16a804497b5b
[2011-06-10 08:54:14.991945] D [client.c:129:client_submit_request] 0-ksc-client-1: rpc_clnt_submit failed
[2011-06-10 08:54:14.991967] D [client.c:129:client_submit_request] 0-ksc-client-1: rpc_clnt_submit failed
[2011-06-10 08:54:14.992014] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(RELEASE(41)) called at 2011-06-10 08:54:14.921180
[2011-06-10 08:54:14.992046] E [rpc-clnt.c:338:saved_frames_unwind] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_notify+0x158) [0x7f140e8c0acc] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x101) [0x7f140e8c006a] (-->/opt/glusterfs/3.2.0/lib64/libgfrpc.so.0(saved_frames_destroy+0x1c) [0x7f140e8bfb78]))) 0-ksc-client-1: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2011-06-10 08:54:14.932952
[2011-06-10 08:54:14.992060] I [client3_1-fops.c:411:client3_1_stat_cbk] 0-ksc-client-1: remote operation failed: Transport endpoint is not connected
[2011-06-10 08:54:14.992073] D [client.c:77:client_submit_request] 0-ksc-client-0: connection in disconnected state
[2011-06-10 08:54:14.992091] W [client3_1-fops.c:2658:client3_1_stat] 0-ksc-client-0: failed to send the fop Transport endpoint is not connected
[2011-06-10 08:54:14.992101] D [afr-inode-read.c:204:afr_stat_cbk] 0-ksc-replicate-0: /pimp/htdocs/mountpoints.sh: all subvolumes tried, going out
[2011-06-10 08:54:14.992115] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fa851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:14.992181] I [client.c:1883:client_rpc_notify] 0-ksc-client-1: disconnected
[2011-06-10 08:54:14.992197] E [afr-common.c:2546:afr_notify] 0-ksc-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2011-06-10 08:54:15.5285] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fb851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.5312] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.5312] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.5344] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fb851fc2, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 74c48fb3-d065-462b-83a9-e4558b042465
[2011-06-10 08:54:15.5381] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /pimp/htdocs/mountpoints.sh: no child is up
[2011-06-10 08:54:15.5399] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fb851fc2, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.5783] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fc851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2011-06-10 08:54:15.5810] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP
[2011-06-10 08:54:15.5833] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: fc851fc2, LOOKUP: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid ff18fc56-c352-4caf-b98d-4eff71494acc, name: mountpoints.sh
[2011-06-10 08:54:15.5897] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup
[2011-06-10 08:54:15.5954] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: fc851fc2, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000
[2011-06-10 08:54:15.11858] E [socket.c:1685:socket_connect_finish] 0-ksc-client-1: connection to 10.0.1.199:24027 failed (Connection refused)
[2011-06-10 08:54:15.11886] D [socket.c:289:__socket_disconnect] 0-ksc-client-1: shutdown() returned -1. Transport endpoint is not connected
[2011-06-10 08:54:15.11921] D [socket.c:193:__socket_rwv] 0-ksc-client-1: EOF from peer 10.0.1.199:24027
[2011-06-10 08:54:15.11938] D [socket.c:1494:__socket_proto_state_machine] 0-ksc-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.1.199:24027)
[2011-06-10 08:54:15.11954] D [socket.c:1768:socket_event_handler] 0-transport: disconnecting now
[2011-06-10 08:54:15.158995] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e1f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.159038] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.159038] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.159065] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: e1f47a24, GETATTR: args: FH: hashcount 6, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 9b62d8e0-cf19-487e-b56a-9ac980294801
[2011-06-10 08:54:15.159105] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /vertrieb/htdocs/vertrieb_imrich/fetischtopliste.de/www/button.php: no child is up
[2011-06-10 08:54:15.159126] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: e1f47a24, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.159501] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e2f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2011-06-10 08:54:15.159528] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP
[2011-06-10 08:54:15.159550] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: e2f47a24, LOOKUP: args: FH: hashcount 5, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid dd60aa24-78fa-4509-abee-acf32f12542f, name: button.php
[2011-06-10 08:54:15.159604] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup
[2011-06-10 08:54:15.159654] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: e2f47a24, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000
[2011-06-10 08:54:15.160014] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e3f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.160041] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.160080] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: e3f47a24, GETATTR: args: FH: hashcount 5, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid dd60aa24-78fa-4509-abee-acf32f12542f
[2011-06-10 08:54:15.160113] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /vertrieb/htdocs/vertrieb_imrich/fetischtopliste.de/www: no child is up
[2011-06-10 08:54:15.160130] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: e3f47a24, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.160532] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: e4f47a24, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2011-06-10 08:54:15.160554] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP
[2011-06-10 08:54:15.160554] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP
[2011-06-10 08:54:15.160569] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: e4f47a24, LOOKUP: args: FH: hashcount 4, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 434fc86c-5bed-4a38-b919-8822d38bd1a3, name: www
[2011-06-10 08:54:15.160611] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup
[2011-06-10 08:54:15.160657] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: e4f47a24, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000
[2011-06-10 08:54:15.192962] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: c81a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.193000] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.193019] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: c81a69ff, GETATTR: args: FH: hashcount 3, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 61273392-07cb-4427-bb48-2cf869031802
[2011-06-10 08:54:15.193050] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com/typo3temp/locks: no child is up
[2011-06-10 08:54:15.193069] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: c81a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.193526] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: c91a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2011-06-10 08:54:15.193547] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP
[2011-06-10 08:54:15.193564] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: c91a69ff, LOOKUP: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 96012827-9101-40a1-b540-b6c47fe49e59, name: locks
[2011-06-10 08:54:15.193607] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup
[2011-06-10 08:54:15.193654] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: c91a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000
[2011-06-10 08:54:15.193987] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: ca1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.194015] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.194037] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: ca1a69ff, GETATTR: args: FH: hashcount 2, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 96012827-9101-40a1-b540-b6c47fe49e59
[2011-06-10 08:54:15.194070] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com/typo3temp: no child is up
[2011-06-10 08:54:15.194086] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: ca1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.194566] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cb1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2011-06-10 08:54:15.194587] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP
[2011-06-10 08:54:15.194614] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: cb1a69ff, LOOKUP: args: FH: hashcount 1, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 7fecc27e-a8de-4fe5-adce-c5e00d35c7ac, name: typo3temp
[2011-06-10 08:54:15.194659] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup
[2011-06-10 08:54:15.194703] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: cb1a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000
[2011-06-10 08:54:15.396934] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cc1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.396989] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.396989] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.397017] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: cc1a69ff, GETATTR: args: FH: hashcount 1, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 7fecc27e-a8de-4fe5-adce-c5e00d35c7ac
[2011-06-10 08:54:15.397064] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /tomorrowwinners.com: no child is up
[2011-06-10 08:54:15.397087] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: cc1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.397441] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cd1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 3
[2011-06-10 08:54:15.397468] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - LOOKUP
[2011-06-10 08:54:15.397492] D [nfs3-helpers.c:2304:nfs3_log_fh_entry_call] 0-nfs-nfsv3: XID: cd1a69ff, LOOKUP: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001, name: tomorrowwinners.com
[2011-06-10 08:54:15.397551] D [nfs3.c:1080:nfs3_fresh_lookup] 0-nfs-nfsv3: inode needs fresh lookup
[2011-06-10 08:54:15.397600] D [nfs3-helpers.c:2477:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: cd1a69ff, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: hashcount 0, exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000
[2011-06-10 08:54:15.397953] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: ce1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:15.397981] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:15.398004] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: ce1a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:15.398038] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:15.398055] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: ce1a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.600726] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: cf1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.600769] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.600769] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.600789] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: cf1a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:15.600857] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: cf1a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.601525] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d01a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:15.601553] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:15.601576] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d01a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:15.601627] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:15.601646] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d01a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.725647] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fd851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:15.725689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:15.725689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:15.725719] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fd851fc2, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:15.725749] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:15.725769] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fd851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.726209] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: fe851fc2, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:15.726236] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:15.726252] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: fe851fc2, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:15.726276] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:15.726292] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: fe851fc2, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.804546] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d11a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:15.804580] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.804580] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:15.804606] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d11a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:15.804658] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d11a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:15.805060] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d21a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:15.805088] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:15.805111] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d21a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:15.805143] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:15.805159] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d21a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.8639] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d31a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:16.8698] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:16.20711] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d31a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:16.20821] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d31a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.21517] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d41a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:16.21556] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:16.21575] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d41a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:16.21606] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:16.21625] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d41a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.225038] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d51a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:16.225096] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:16.225096] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:16.225118] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d51a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:16.225193] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d51a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.225595] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d61a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:16.225622] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:16.225639] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d61a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:16.225669] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:16.225688] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d61a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.243210] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d71a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:16.243232] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:16.243232] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:16.243249] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d71a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:16.243274] I [afr-inode-read.c:270:afr_stat] 0-ksc-replicate-0: /: no child is up
[2011-06-10 08:54:16.243291] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d71a69ff, ACCESS: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.243670] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d81a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:16.243697] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:16.243717] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d81a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:16.243756] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d81a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.428628] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: d91a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 1
[2011-06-10 08:54:16.428689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:16.428689] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - GETATTR
[2011-06-10 08:54:16.428711] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: d91a69ff, GETATTR: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
[2011-06-10 08:54:16.428787] D [nfs3-helpers.c:2424:nfs3_log_common_res] 0-nfs-nfsv3: XID: d91a69ff, GETATTR: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected)
[2011-06-10 08:54:16.429129] D [rpcsvc.c:1940:nfs_rpcsvc_request_create] 0-nfsrpc: RPC XID: da1a69ff, Ver: 2, Program: 100003, ProgVers: 3, Proc: 4
[2011-06-10 08:54:16.429157] D [rpcsvc.c:1357:nfs_rpcsvc_program_actor] 0-nfsrpc: Actor found: NFS3 - ACCESS
[2011-06-10 08:54:16.429182] D [nfs3-helpers.c:2292:nfs3_log_common_call] 0-nfs-nfsv3: XID: da1a69ff, ACCESS: args: FH: hashcount 0, exportid ea50df7c-ff08-4416-8fb3-59d09667cc51, gfid 00000000-0000-0000-0000-000000000001
do you have some feedback? do you need further information? (In reply to comment #4) > do you have some feedback? > do you need further information? Thanks for the information provided. This crash is the same as the one seen in bug 764584. Fix for this is available in 3.2.1. *** This bug has been marked as a duplicate of bug 2852 *** thank you for feedback. we will upgrade and test it :-) thanks! |
here are some traces from core-dumps: ------------------------------------- [New Thread 33941] [New Thread 33944] [New Thread 33984] [New Thread 33942] [New Thread 34996] [New Thread 19224] [New Thread 34360] [New Thread 33986] [New Thread 33983] Core was generated by `/opt/glusterfs/3.2.0/sbin/glusterfsd --xlator-option community-server.listen-po'. Program terminated with signal 11, Segmentation fault. #0 0x00007f24be8fbaa8 in marker_setattr_cbk (frame=0x7f24c0b3c7d0, cookie=0x7f24c0b3d958, this=0x63c090, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at marker.c:1590 in marker.c Thread 10 (Thread 33983): #0 0x00007f24c18acfdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f24beb104e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f24c0870a00 sleep_till = {tv_sec = 1307701232, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 9 (Thread 33986): #0 0x00007f24c18acfdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f24beb104e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f24c0870a00 sleep_till = {tv_sec = 1307701232, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 8 (Thread 34360): #0 0x00007f24c18acfdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f24beb104e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f24c0875d04 sleep_till = {tv_sec = 1307701232, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 7 (Thread 19224): #0 0x00007f24c18acfdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f24beb104e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f24c0875d04 sleep_till = {tv_sec = 1307701232, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 6 (Thread 34996): #0 0x00007f24c18acfdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f24beb104e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f24c0875d04 sleep_till = {tv_sec = 1307701232, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 5 (Thread 33942): #0 0x00007f24c18b0767 in do_sigwait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f24c18b080d in sigwait () from /lib64/libpthread.so.0 No symbol table info available. #2 0x0000000000406b52 in glusterfs_sigwaiter (arg=0x7fff74f3ac20) at glusterfsd.c:1229 set = {__val = {139795842624003, 139795853955993, 5, 0, 0, 0, 139795839132544, 139795825609040, 0, 139795853980290, 139795825609040, 139795842633727, 18446744073709551520, 24, 139795825607120, 139795825609040}} ret = 0 sig = 0 #3 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 4 (Thread 33984): #0 0x00007f24c18acfdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f24bf154c32 in janitor_get_next_fd (this=0x637c30) at posix.c:1219 priv = (struct posix_private *) 0x641e00 pfd = (struct posix_fd *) 0x0 timeout = {tv_sec = 1307701712, tv_nsec = 0} #2 0x00007f24bf154da9 in posix_janitor_thread_proc (data=0x637c30) at posix.c:1265 this = (xlator_t *) 0x637c30 priv = (struct posix_private *) 0x641e00 pfd = (struct posix_fd *) 0x671070 now = 1307701112 __FUNCTION__ = "posix_janitor_thread_proc" #3 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 3 (Thread 33944): #0 0x00007f24c15eacb1 in nanosleep () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f24c1615e64 in usleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f24c21294e8 in gf_timer_proc (ctx=0x62e010) at timer.c:181 now = 1307701111980806 now_tv = {tv_sec = 1307701111, tv_usec = 980806} event = (gf_timer_t *) 0x7f24b003c160 reg = (gf_timer_registry_t *) 0x631d20 __FUNCTION__ = "gf_timer_proc" #3 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 2 (Thread 33941): #0 0x00007f24c161c728 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f24c21413dc in event_dispatch_epoll (event_pool=0x62e320) at event.c:839 events = (struct epoll_event *) 0x631ef0 size = 1 i = 1 ret = 0 __FUNCTION__ = "event_dispatch_epoll" #2 0x00007f24c214179c in event_dispatch (event_pool=0x62e320) at event.c:956 ret = -1 __FUNCTION__ = "event_dispatch" #3 0x00000000004072a1 in main (argc=17, argv=0x7fff74f3adf8) at glusterfsd.c:1476 ctx = (glusterfs_ctx_t *) 0x62e010 ret = 0 __FUNCTION__ = "main" Thread 1 (Thread 65177): #0 0x00007f24be8fbaa8 in marker_setattr_cbk (frame=0x7f24c0b3c7d0, cookie=0x7f24c0b3d958, this=0x63c090, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at marker.c:1590 local = (marker_local_t *) 0x0 priv = (marker_conf_t *) 0x0 __FUNCTION__ = "marker_setattr_cbk" #1 0x00007f24beb10d66 in iot_setattr_cbk (frame=0x7f24c0b3d958, cookie=0x7f24c0b41294, this=0x63aff0, op_ret=-1, op_errno=1, preop=0x0, postop=0x0) at io-threads.c:251 fn = (fop_setattr_cbk_t) 0x7f24be8fba1e <marker_setattr_cbk> _parent = (call_frame_t *) 0x7f24c0b3c7d0 old_THIS = (xlator_t *) 0x63aff0 __FUNCTION__ = "iot_setattr_cbk" #2 0x00007f24c211b1aa in default_setattr_cbk (frame=0x7f24c0b41294, cookie=0x7f24c0b3dc70, this=0x639fb0, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at defaults.c:405 fn = (fop_setattr_cbk_t) 0x7f24beb10c3b <iot_setattr_cbk> _parent = (call_frame_t *) 0x7f24c0b3d958 old_THIS = (xlator_t *) 0x639fb0 __FUNCTION__ = "default_setattr_cbk" #3 0x00007f24bef476d7 in ac_setattr_stat_cbk (frame=0x7f24c0b3dc70, cookie=0x7f24c0b393bc, this=0x638e40, op_ret=-1, op_errno=1, buf=0x7f24bd189e90) at access-control.c:1843 fn = (fop_setattr_cbk_t) 0x7f24c211b07f <default_setattr_cbk> _parent = (call_frame_t *) 0x7f24c0b41294 old_THIS = (xlator_t *) 0x638e40 stub = (call_stub_t *) 0x7f24c0870a00 valid = 0 setbuf = (struct iatt *) 0x0 __FUNCTION__ = "ac_setattr_stat_cbk" #4 0x00007f24bf1517d9 in posix_stat (frame=0x7f24c0b393bc, this=0x637c30, loc=0x7f24c0875d3c) at posix.c:518 fn = (fop_stat_cbk_t) 0x7f24bef473b1 <ac_setattr_stat_cbk> _parent = (call_frame_t *) 0x7f24c0b3dc70 old_THIS = (xlator_t *) 0x637c30 buf = {ia_ino = 390291458, ia_gfid = "ÜØ\aÂÀ\032C\202\205×râ\\Ú®\a", ia_dev = 2081, ia_type = IA_IFDIR, ia_prot = {suid = 0 '\0', sgid = 0 '\0', sticky = 0 '\0', owner = {read = 1 '\001', write = 1 '\001', exec = 1 '\001'}, group = {read = 1 '\001', write = 0 '\0', exec = 1 '\001'}, other = {read = 1 '\001', write = 0 '\0', exec = 1 '\001'}}, ia_nlink = 3, ia_uid = 1005, ia_gid = 100, ia_rdev = 0, ia_size = 4096, ia_blksize = 4096, ia_blocks = 8, ia_atime = 1307700690, ia_atime_nsec = 0, ia_mtime = 1306417107, ia_mtime_nsec = 0, ia_ctime = 1307700774, ia_ctime_nsec = 0} real_path = 0x7f24bd189dd0 "/gluster-storage/community/flirty/rotlichkartei_base/static/images" op_ret = 0 op_errno = 0 priv = (struct posix_private *) 0x641e00 __FUNCTION__ = "posix_stat" #5 0x00007f24bef47958 in ac_setattr (frame=0x7f24c0b3dc70, this=0x638e40, loc=0x7f24c0875d3c, buf=0x7f24c0875d64, valid=48) at access-control.c:1870 _new = (call_frame_t *) 0x7f24c0b393bc old_THIS = (xlator_t *) 0x638e40 tmp_cbk = (fop_stat_cbk_t) 0x7f24bef473b1 <ac_setattr_stat_cbk> stub = (call_stub_t *) 0x7f24c0870a00 ret = -14 __FUNCTION__ = "ac_setattr" #6 0x00007f24c2124967 in default_setattr (frame=0x7f24c0b41294, this=0x639fb0, loc=0x7f24c0875d3c, stbuf=0x7f24c0875d64, valid=48) at defaults.c:1131 _new = (call_frame_t *) 0x7f24c0b3dc70 old_THIS = (xlator_t *) 0x639fb0 tmp_cbk = (fop_setattr_cbk_t) 0x7f24c211b07f <default_setattr_cbk> __FUNCTION__ = "default_setattr" #7 0x00007f24beb10f5e in iot_setattr_wrapper (frame=0x7f24c0b3d958, this=0x63aff0, loc=0x7f24c0875d3c, stbuf=0x7f24c0875d64, valid=48) at io-threads.c:260 _new = (call_frame_t *) 0x7f24c0b41294 old_THIS = (xlator_t *) 0x63aff0 tmp_cbk = (fop_setattr_cbk_t) 0x7f24beb10c3b <iot_setattr_cbk> __FUNCTION__ = "iot_setattr_wrapper" #8 0x00007f24c2137296 in call_resume_wind (stub=0x7f24c0875d04) at call-stub.c:2467 __FUNCTION__ = "call_resume_wind" #9 0x00007f24c213d020 in call_resume (stub=0x7f24c0875d04) at call-stub.c:3861 old_THIS = (xlator_t *) 0x63aff0 __FUNCTION__ = "call_resume" #10 0x00007f24beb105c6 in iot_worker (data=0x641bd0) at io-threads.c:129 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f24c0875d04 sleep_till = {tv_sec = 1307701232, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #11 0x00007f24c18a9070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #12 0x00007f24c161c13d in clone () from /lib64/libc.so.6 No symbol table info available. #13 0x0000000000000000 in ?? () No symbol table info available. [New Thread 43505] [New Thread 43467] [New Thread 44539] [New Thread 43466] [New Thread 43576] [New Thread 43469] [New Thread 43545] [New Thread 44504] Core was generated by `/opt/glusterfs/3.2.0/sbin/glusterfsd --xlator-option community-server.listen-po'. Program terminated with signal 11, Segmentation fault. #0 0x00007f2d77ab6aa8 in marker_setattr_cbk (frame=0x7f2d79cf56d0, cookie=0x7f2d79cf53b8, this=0x63c090, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at marker.c:1590 in marker.c Thread 9 (Thread 44504): #0 0x00007f2d7aa67fdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f2d77ccb4e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f2d79a296c8 sleep_till = {tv_sec = 1307703021, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 8 (Thread 43545): #0 0x00007f2d7aa67fdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f2d77ccb4e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f2d79a296c8 sleep_till = {tv_sec = 1307703021, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 7 (Thread 43469): #0 0x00007f2d7a7a5cb1 in nanosleep () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f2d7a7d0e64 in usleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f2d7b2e44e8 in gf_timer_proc (ctx=0x62e010) at timer.c:181 now = 1307702901372768 now_tv = {tv_sec = 1307702901, tv_usec = 372768} event = (gf_timer_t *) 0x632b10 reg = (gf_timer_registry_t *) 0x631d20 __FUNCTION__ = "gf_timer_proc" #3 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 6 (Thread 43576): #0 0x00007f2d7aa67fdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f2d77ccb4e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f2d79a296c8 sleep_till = {tv_sec = 1307703021, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 5 (Thread 43466): #0 0x00007f2d7a7d7728 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f2d7b2fc3dc in event_dispatch_epoll (event_pool=0x62e320) at event.c:839 events = (struct epoll_event *) 0x631ef0 size = 1 i = 1 ret = 0 __FUNCTION__ = "event_dispatch_epoll" #2 0x00007f2d7b2fc79c in event_dispatch (event_pool=0x62e320) at event.c:956 ret = -1 __FUNCTION__ = "event_dispatch" #3 0x00000000004072a1 in main (argc=17, argv=0x7fff38de6e88) at glusterfsd.c:1476 ctx = (glusterfs_ctx_t *) 0x62e010 ret = 0 __FUNCTION__ = "main" Thread 4 (Thread 44539): #0 0x00007f2d7aa67fdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f2d77ccb4e3 in iot_worker (data=0x641bd0) at io-threads.c:101 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f2d79a296c8 sleep_till = {tv_sec = 1307703021, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #2 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. Thread 3 (Thread 43467): #0 0x00007f2d7aa6b767 in do_sigwait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f2d7aa6b80d in sigwait () from /lib64/libpthread.so.0 No symbol table info available. #2 0x0000000000406b52 in glusterfs_sigwaiter (arg=0x7fff38de6cb0) at glusterfsd.c:1229 set = {__val = {139833307945475, 139833319293849, 5, 0, 0, 0, 139833304470400, 139833290946896, 0, 139833319318146, 139833290946896, 139833307971583, 18446744073709551520, 24, 139833290944976, 139833290946896}} ret = 0 sig = 0 #3 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 2 (Thread 43505): #0 0x00007f2d7aa67fdd in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f2d7830fc32 in janitor_get_next_fd (this=0x637c30) at posix.c:1219 priv = (struct posix_private *) 0x7f2d700008e0 pfd = (struct posix_fd *) 0x0 timeout = {tv_sec = 1307703494, tv_nsec = 0} #2 0x00007f2d7830fda9 in posix_janitor_thread_proc (data=0x637c30) at posix.c:1265 this = (xlator_t *) 0x637c30 priv = (struct posix_private *) 0x7f2d700008e0 pfd = (struct posix_fd *) 0x7f2d68046ef0 now = 1307702894 __FUNCTION__ = "posix_janitor_thread_proc" #3 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 1 (Thread 43504): #0 0x00007f2d77ab6aa8 in marker_setattr_cbk (frame=0x7f2d79cf56d0, cookie=0x7f2d79cf53b8, this=0x63c090, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at marker.c:1590 local = (marker_local_t *) 0x0 priv = (marker_conf_t *) 0x0 __FUNCTION__ = "marker_setattr_cbk" #1 0x00007f2d77ccbd66 in iot_setattr_cbk (frame=0x7f2d79cf53b8, cookie=0x7f2d79cf4d04, this=0x63aff0, op_ret=-1, op_errno=1, preop=0x0, postop=0x0) at io-threads.c:251 fn = (fop_setattr_cbk_t) 0x7f2d77ab6a1e <marker_setattr_cbk> _parent = (call_frame_t *) 0x7f2d79cf56d0 old_THIS = (xlator_t *) 0x63aff0 __FUNCTION__ = "iot_setattr_cbk" #2 0x00007f2d7b2d61aa in default_setattr_cbk (frame=0x7f2d79cf4d04, cookie=0x7f2d79cf4230, this=0x639fb0, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at defaults.c:405 fn = (fop_setattr_cbk_t) 0x7f2d77ccbc3b <iot_setattr_cbk> _parent = (call_frame_t *) 0x7f2d79cf53b8 old_THIS = (xlator_t *) 0x639fb0 __FUNCTION__ = "default_setattr_cbk" #3 0x00007f2d781026d7 in ac_setattr_stat_cbk (frame=0x7f2d79cf4230, cookie=0x7f2d79cf50a0, this=0x638e40, op_ret=-1, op_errno=1, buf=0x7f2d773b3e90) at access-control.c:1843 fn = (fop_setattr_cbk_t) 0x7f2d7b2d607f <default_setattr_cbk> _parent = (call_frame_t *) 0x7f2d79cf4d04 old_THIS = (xlator_t *) 0x638e40 stub = (call_stub_t *) 0x7f2d79a2a35c valid = 0 setbuf = (struct iatt *) 0x0 __FUNCTION__ = "ac_setattr_stat_cbk" #4 0x00007f2d7830c7d9 in posix_stat (frame=0x7f2d79cf50a0, this=0x637c30, loc=0x7f2d79a29700) at posix.c:518 fn = (fop_stat_cbk_t) 0x7f2d781023b1 <ac_setattr_stat_cbk> _parent = (call_frame_t *) 0x7f2d79cf4230 old_THIS = (xlator_t *) 0x637c30 buf = {ia_ino = 219389988, ia_gfid = "'¬Õ\220\tRJ¦\221£\016\023üªÜl", ia_dev = 2081, ia_type = IA_IFREG, ia_prot = {suid = 0 '\0', sgid = 0 '\0', sticky = 0 '\0', owner = {read = 1 '\001', write = 1 '\001', exec = 0 '\0'}, group = {read = 1 '\001', write = 0 '\0', exec = 0 '\0'}, other = {read = 1 '\001', write = 0 '\0', exec = 0 '\0'}}, ia_nlink = 1, ia_uid = 1005, ia_gid = 100, ia_rdev = 0, ia_size = 4440, ia_blksize = 4096, ia_blocks = 24, ia_atime = 1307702643, ia_atime_nsec = 0, ia_mtime = 1307702901, ia_mtime_nsec = 0, ia_ctime = 1307702901, ia_ctime_nsec = 0} real_path = 0x7f2d773b3dc0 "/gluster-storage/community/amateurseite/image_data/static/resources/video/js/_video.js" op_ret = 0 op_errno = 0 priv = (struct posix_private *) 0x7f2d700008e0 __FUNCTION__ = "posix_stat" #5 0x00007f2d78102958 in ac_setattr (frame=0x7f2d79cf4230, this=0x638e40, loc=0x7f2d79a29700, buf=0x7f2d79a29728, valid=8) at access-control.c:1870 _new = (call_frame_t *) 0x7f2d79cf50a0 old_THIS = (xlator_t *) 0x638e40 tmp_cbk = (fop_stat_cbk_t) 0x7f2d781023b1 <ac_setattr_stat_cbk> stub = (call_stub_t *) 0x7f2d79a2a35c ret = -14 __FUNCTION__ = "ac_setattr" #6 0x00007f2d7b2df967 in default_setattr (frame=0x7f2d79cf4d04, this=0x639fb0, loc=0x7f2d79a29700, stbuf=0x7f2d79a29728, valid=8) at defaults.c:1131 _new = (call_frame_t *) 0x7f2d79cf4230 old_THIS = (xlator_t *) 0x639fb0 tmp_cbk = (fop_setattr_cbk_t) 0x7f2d7b2d607f <default_setattr_cbk> __FUNCTION__ = "default_setattr" #7 0x00007f2d77ccbf5e in iot_setattr_wrapper (frame=0x7f2d79cf53b8, this=0x63aff0, loc=0x7f2d79a29700, stbuf=0x7f2d79a29728, valid=8) at io-threads.c:260 _new = (call_frame_t *) 0x7f2d79cf4d04 old_THIS = (xlator_t *) 0x63aff0 tmp_cbk = (fop_setattr_cbk_t) 0x7f2d77ccbc3b <iot_setattr_cbk> __FUNCTION__ = "iot_setattr_wrapper" #8 0x00007f2d7b2f2296 in call_resume_wind (stub=0x7f2d79a296c8) at call-stub.c:2467 __FUNCTION__ = "call_resume_wind" #9 0x00007f2d7b2f8020 in call_resume (stub=0x7f2d79a296c8) at call-stub.c:3861 old_THIS = (xlator_t *) 0x63aff0 __FUNCTION__ = "call_resume" #10 0x00007f2d77ccb5c6 in iot_worker (data=0x641bd0) at io-threads.c:129 conf = (iot_conf_t *) 0x641bd0 this = (xlator_t *) 0x63aff0 stub = (call_stub_t *) 0x7f2d79a296c8 sleep_till = {tv_sec = 1307703021, tv_nsec = 0} ret = 0 timeout = 0 '\0' bye = 0 '\0' __FUNCTION__ = "iot_worker" #11 0x00007f2d7aa64070 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #12 0x00007f2d7a7d713d in clone () from /lib64/libc.so.6 No symbol table info available. #13 0x0000000000000000 in ?? () No symbol table info available. [New Thread 58484] [New Thread 58453] [New Thread 58455] [New Thread 58461] [New Thread 58452] [New Thread 58485] Core was generated by `/opt/glusterfs/3.2.0/sbin/glusterfsd --xlator-option ksc-server.listen-port=240'. Program terminated with signal 11, Segmentation fault. #0 0x00007f156afe2aa8 in marker_setattr_cbk (frame=0x7f156d21f96c, cookie=0x7f156d21f4c8, this=0x63c120, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at marker.c:1590 1590 gf_log (this->name, ((op_errno == ENOENT) ? GF_LOG_DEBUG : Thread 7 (Thread 58485): #0 0x00007f156dccf23a in __getdents64 () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f156dccebe7 in readdir64 () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f156b849f9a in posix_do_readdir (frame=0x7f156d21f1b0, this=0x637c20, fd=0x7f1569dd6468, size=130960, off=0, whichop=28) at posix.c:4082 tmp_pfd = 139729714597552 pfd = 0x7f155c00ceb0 dir = 0x7f155c057600 ret = 0 filled = 0 count = 0 op_ret = -1 op_errno = 0 this_entry = 0x0 entries = {{list = {next = 0x7f1569cd3b70, prev = 0x7f1569cd3b70}, {next = 0x7f1569cd3b70, prev = 0x7f1569cd3b70}}, d_ino = 6662912, d_off = 1677858592, d_len = 1775058160, d_type = 32533, d_stat = {ia_ino = 139729974822883, ia_gfid = "\360<\315i\025\177\000\000\000\000\000\000\000\000\000", ia_dev = 8243122732211660591, ia_type = 1919906931, ia_prot = {suid = 1 '\001', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 1 '\001', write = 1 '\001', exec = 1 '\001'}, group = {read = 1 '\001', write = 0 '\000', exec = 1 '\001'}, other = {read = 1 '\001', write = 1 '\001', exec = 1 '\001'}}, ia_nlink = 1785688688, ia_uid = 1702128485, ia_gid = 1668508463, ia_rdev = 112, ia_size = 139729974822273, ia_blksize = 6, ia_blocks = 5925466990, ia_atime = 1714254453, ia_atime_nsec = 1701080681, ia_mtime = 14, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0}, d_name = 0x7f1569cd3c08 "P<\315i\025\177"} entry = 0x0 in_case = 0 this_size = -1 real_path = 0x7f155c00b410 "/glusterstorage/projekte/ksc/vertrieb/htdocs/vertrieb_imrich/hirtermadl.com/www" real_path_len = 79 entry_path = 0x7f1569cd1860 "/glusterstorage/projekte/ksc/vertrieb/htdocs/vertrieb_imrich/hirtermadl.com/www/" entry_path_len = 334 priv = 0x641de0 stbuf = {ia_ino = 0, ia_gfid = '\000' <repeats 15 times>, ia_dev = 0, ia_type = IA_INVAL, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0} base_path = "/glusterstorage/projekte/ksc/", '\000' <repeats 4066 times> tmp_entry = 0x0 statbuf = {st_dev = 0, st_ino = 0, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 0, st_blksize = 0, st_blocks = 0, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 0, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __unused = {0, 0, 0}} hidden_path = '\000' <repeats 4095 times> __FUNCTION__ = "posix_do_readdir" #3 0x00007f156b84a4ff in posix_readdir (frame=0x7f156d21f1b0, this=0x637c20, fd=0x7f1569dd6468, size=130960, off=0) at posix.c:4164 No locals. #4 0x00007f156e811576 in default_readdir (frame=0x7f156d21f5d0, this=0x638e90, fd=0x7f1569dd6468, size=130960, off=0) at defaults.c:1112 _new = 0x7f156d21f1b0 old_THIS = 0x638e90 tmp_cbk = 0x7f156e807e0b <default_readdir_cbk> __FUNCTION__ = "default_readdir" #5 0x00007f156e811576 in default_readdir (frame=0x7f156d21f54c, this=0x63a020, fd=0x7f1569dd6468, size=130960, off=0) at defaults.c:1112 _new = 0x7f156d21f5d0 old_THIS = 0x63a020 tmp_cbk = 0x7f156e807e0b <default_readdir_cbk> __FUNCTION__ = "default_readdir" #6 0x00007f156b20210f in iot_readdir_wrapper (frame=0x7f156d220020, this=0x63b010, fd=0x7f1569dd6468, size=130960, offset=0) at io-threads.c:1808 _new = 0x7f156d21f54c old_THIS = 0x63b010 tmp_cbk = 0x7f156b201df7 <iot_readdir_cbk> __FUNCTION__ = "iot_readdir_wrapper" #7 0x00007f156e824182 in call_resume_wind (stub=0x7f156cf547b0) at call-stub.c:2427 __FUNCTION__ = "call_resume_wind" #8 0x00007f156e82a020 in call_resume (stub=0x7f156cf547b0) at call-stub.c:3861 old_THIS = 0x63b010 __FUNCTION__ = "call_resume" #9 0x00007f156b1f75c6 in iot_worker (data=0x641b90) at io-threads.c:129 conf = 0x641b90 this = 0x63b010 stub = 0x7f156cf547b0 sleep_till = {tv_sec = 1307691192, tv_nsec = 0} ret = 0 timeout = 0 '\000' bye = 0 '\000' __FUNCTION__ = "iot_worker" #10 0x00007f156df955f0 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #11 0x00007f156dd0487d in clone () from /lib64/libc.so.6 No symbol table info available. #12 0x0000000000000000 in ?? () No symbol table info available. Thread 6 (Thread 58452): #0 0x00007f156dd04f03 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f156e82e3dc in event_dispatch_epoll (event_pool=0x62e320) at event.c:839 events = 0x631eb0 size = 1 i = 1 ret = 0 __FUNCTION__ = "event_dispatch_epoll" #2 0x00007f156e82e79c in event_dispatch (event_pool=0x62e320) at event.c:956 ret = -1 __FUNCTION__ = "event_dispatch" #3 0x00000000004072a1 in main (argc=17, argv=0x7fffa9d57d48) at glusterfsd.c:1476 ctx = 0x62e010 ret = 0 __FUNCTION__ = "main" Thread 5 (Thread 58461): #0 0x00007f156df99799 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f156b83bc32 in janitor_get_next_fd (this=0x637c20) at posix.c:1219 priv = 0x641de0 pfd = 0x0 timeout = {tv_sec = 1307691672, tv_nsec = 0} #2 0x00007f156b83bda9 in posix_janitor_thread_proc (data=0x637c20) at posix.c:1265 this = 0x637c20 priv = 0x641de0 pfd = 0x65a740 now = 1307691072 __FUNCTION__ = "posix_janitor_thread_proc" #3 0x00007f156df955f0 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f156dd0487d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 4 (Thread 58455): #0 0x00007f156dcd307d in nanosleep () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f156dcfe464 in usleep () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f156e8164e8 in gf_timer_proc (ctx=0x62e010) at timer.c:181 now = 1307691072241487 now_tv = {tv_sec = 1307691072, tv_usec = 241487} event = 0x632b30 reg = 0x631c00 __FUNCTION__ = "gf_timer_proc" #3 0x00007f156df955f0 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f156dd0487d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 3 (Thread 58453): #0 0x00007f156df9d2a7 in do_sigwait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f156df9d34d in sigwait () from /lib64/libpthread.so.0 No symbol table info available. #2 0x0000000000406b52 in glusterfs_sigwaiter (arg=0x7fffa9d57b70) at glusterfsd.c:1229 set = {__val = {18951, 0, 0, 0, 139730012575144, 139729999050512, 0, 139730016138912, 139729999048704, 139730027448453, 139729999050512, 139730016097663, 18446744073709551520, 24, 139729999048560, 139729999050512}} ret = 0 sig = 0 #3 0x00007f156df955f0 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f156dd0487d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 2 (Thread 58484): #0 0x00007f156df9c57d in read () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f156b836a73 in _posix_xattr_get_set (xattr_req=0x7f15640041b0, key=0x7f1564006370 "glusterfs.content", data=0x7f15640062f0, xattrargs=0x7f1569dd4a60) at posix.c:138 filler = 0x7f1569dd4a60 value = 0x0 xattr_size = -1 ret = -1 databuf = 0x677ec0 "" _fd = 17 loc = 0x0 req_size = 65536 __FUNCTION__ = "_posix_xattr_get_set" #2 0x00007f156e7f6d9c in dict_foreach (dict=0x7f15640041b0, fn=0x7f156b8368f0 <_posix_xattr_get_set>, data=0x7f1569dd4a60) at dict.c:1198 pairs = 0x7f1564006340 next = 0x7f1564006290 __FUNCTION__ = "dict_foreach" #3 0x00007f156b837785 in posix_lookup_xattr_fill (this=0x637c20, real_path=0x7f1569dd4ac0 "/glusterstorage/projekte/ksc/maxposter.at/administration/index.php", loc=0x7f156cf54f74, xattr_req=0x7f15640041b0, buf=0x7f1569dd4c10) at posix.c:318 xattr = 0x65ab50 filler = {this = 0x637c20, real_path = 0x7f1569dd4ac0 "/glusterstorage/projekte/ksc/maxposter.at/administration/index.php", xattr = 0x65ab50, stbuf = 0x7f1569dd4c10, loc = 0x7f156cf54f74} #4 0x00007f156b837f96 in posix_lookup (frame=0x7f156d21ff9c, this=0x637c20, loc=0x7f156cf54f74, xattr_req=0x7f15640041b0) at posix.c:445 buf = {ia_ino = 355033535, ia_gfid = "\227\304\354\367\241\fLL\201]\216\024l\357\023\273", ia_dev = 2097, ia_type = IA_IFREG, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 1 '\001', write = 1 '\001', exec = 1 '\001'}, group = {read = 1 '\001', write = 1 '\001', exec = 1 '\001'}, other = {read = 1 '\001', write = 1 '\001', exec = 1 '\001'}}, ia_nlink = 1, ia_uid = 1019, ia_gid = 100, ia_rdev = 0, ia_size = 2834, ia_blksize = 4096, ia_blocks = 16, ia_atime = 1297772584, ia_atime_nsec = 0, ia_mtime = 1304057179, ia_mtime_nsec = 0, ia_ctime = 1304057179, ia_ctime_nsec = 0} real_path = 0x7f1569dd4ac0 "/glusterstorage/projekte/ksc/maxposter.at/administration/index.php" op_ret = 0 entry_ret = 0 op_errno = 22 xattr = 0x0 pathdup = 0x0 parentpath = 0x0 postparent = {ia_ino = 0, ia_gfid = '\000' <repeats 15 times>, ia_dev = 0, ia_type = IA_INVAL, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0} __FUNCTION__ = "posix_lookup" #5 0x00007f156e811f1e in default_lookup (frame=0x7f156d21f024, this=0x638e90, loc=0x7f156cf54f74, xattr_req=0x7f15640041b0) at defaults.c:1156 _new = 0x7f156d21ff9c old_THIS = 0x638e90 tmp_cbk = 0x7f156e805118 <default_lookup_cbk> __FUNCTION__ = "default_lookup" #6 0x00007f156b41472f in pl_lookup (frame=0x7f156d21fa74, this=0x63a020, loc=0x7f156cf54f74, xattr_req=0x7f15640041b0) at posix.c:1490 _new = 0x7f156d21f024 old_THIS = 0x63a020 tmp_cbk = 0x7f156b414082 <pl_lookup_cbk> local = 0x65b760 ret = -1 __FUNCTION__ = "pl_lookup" #7 0x00007f156b1f7a13 in iot_lookup_wrapper (frame=0x7f156d21f7e0, this=0x63b010, loc=0x7f156cf54f74, xattr_req=0x7f15640041b0) at io-threads.c:209 _new = 0x7f156d21fa74 old_THIS = 0x63b010 tmp_cbk = 0x7f156b1f76ed <iot_lookup_cbk> __FUNCTION__ = "iot_lookup_wrapper" #8 0x00007f156e8240f9 in call_resume_wind (stub=0x7f156cf54f3c) at call-stub.c:2408 __FUNCTION__ = "call_resume_wind" #9 0x00007f156e82a020 in call_resume (stub=0x7f156cf54f3c) at call-stub.c:3861 old_THIS = 0x63b010 __FUNCTION__ = "call_resume" #10 0x00007f156b1f75c6 in iot_worker (data=0x641b90) at io-threads.c:129 conf = 0x641b90 this = 0x63b010 stub = 0x7f156cf54f3c sleep_till = {tv_sec = 1307691192, tv_nsec = 0} ret = 0 timeout = 0 '\000' bye = 0 '\000' __FUNCTION__ = "iot_worker" #11 0x00007f156df955f0 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #12 0x00007f156dd0487d in clone () from /lib64/libc.so.6 No symbol table info available. #13 0x0000000000000000 in ?? () No symbol table info available. Thread 1 (Thread 58460): #0 0x00007f156afe2aa8 in marker_setattr_cbk (frame=0x7f156d21f96c, cookie=0x7f156d21f4c8, this=0x63c120, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at marker.c:1590 local = 0x0 priv = 0x0 __FUNCTION__ = "marker_setattr_cbk" #1 0x00007f156b1f7d66 in iot_setattr_cbk (frame=0x7f156d21f4c8, cookie=0x7f156d21fc84, this=0x63b010, op_ret=-1, op_errno=1, preop=0x0, postop=0x0) at io-threads.c:251 fn = 0x7f156afe2a1e <marker_setattr_cbk> _parent = 0x7f156d21f96c old_THIS = 0x63b010 __FUNCTION__ = "iot_setattr_cbk" #2 0x00007f156e8081aa in default_setattr_cbk (frame=0x7f156d21fc84, cookie=0x7f156d2201ac, this=0x63a020, op_ret=-1, op_errno=1, statpre=0x0, statpost=0x0) at defaults.c:405 fn = 0x7f156b1f7c3b <iot_setattr_cbk> _parent = 0x7f156d21f4c8 old_THIS = 0x63a020 __FUNCTION__ = "default_setattr_cbk" #3 0x00007f156b62e6d7 in ac_setattr_stat_cbk (frame=0x7f156d2201ac, cookie=0x7f156d21f0a8, this=0x638e90, op_ret=-1, op_errno=1, buf=0x7f156a987c30) at access-control.c:1843 fn = 0x7f156e80807f <default_setattr_cbk> _parent = 0x7f156d21fc84 old_THIS = 0x638e90 stub = 0x7f156cf54cb8 valid = 0 setbuf = 0x0 __FUNCTION__ = "ac_setattr_stat_cbk" #4 0x00007f156b8387d9 in posix_stat (frame=0x7f156d21f0a8, this=0x637c20, loc=0x7f156cf5547c) at posix.c:518 fn = 0x7f156b62e3b1 <ac_setattr_stat_cbk> _parent = 0x7f156d2201ac old_THIS = 0x637c20 buf = {ia_ino = 355345969, ia_gfid = "\207O\263f\335bG@\217F\002\354\021\263\063\061", ia_dev = 2097, ia_type = IA_IFREG, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = { read = 1 '\001', write = 0 '\000', exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 1 '\001'}}, ia_nlink = 1, ia_uid = 30, ia_gid = 8, ia_rdev = 0, ia_size = 0, ia_blksize = 4096, ia_blocks = 0, ia_atime = 1307691072, ia_atime_nsec = 0, ia_mtime = 1307691072, ia_mtime_nsec = 0, ia_ctime = 1307691072, ia_ctime_nsec = 0} real_path = 0x7f156a987b50 "/glusterstorage/projekte/ksc/tomorrowwinners.com/typo3temp/locks/8b2d56cc0865323e768e2f7ff9764c72" op_ret = 0 op_errno = 0 priv = 0x641de0 __FUNCTION__ = "posix_stat" #5 0x00007f156b62e958 in ac_setattr (frame=0x7f156d2201ac, this=0x638e90, loc=0x7f156cf5547c, buf=0x7f156cf554a4, valid=48) at access-control.c:1870 _new = 0x7f156d21f0a8 old_THIS = 0x638e90 tmp_cbk = 0x7f156b62e3b1 <ac_setattr_stat_cbk> stub = 0x7f156cf54cb8 ret = -14 __FUNCTION__ = "ac_setattr" #6 0x00007f156e811967 in default_setattr (frame=0x7f156d21fc84, this=0x63a020, loc=0x7f156cf5547c, stbuf=0x7f156cf554a4, valid=48) at defaults.c:1131 _new = 0x7f156d2201ac old_THIS = 0x63a020 tmp_cbk = 0x7f156e80807f <default_setattr_cbk> __FUNCTION__ = "default_setattr" #7 0x00007f156b1f7f5e in iot_setattr_wrapper (frame=0x7f156d21f4c8, this=0x63b010, loc=0x7f156cf5547c, stbuf=0x7f156cf554a4, valid=48) at io-threads.c:260 _new = 0x7f156d21fc84 old_THIS = 0x63b010 tmp_cbk = 0x7f156b1f7c3b <iot_setattr_cbk> __FUNCTION__ = "iot_setattr_wrapper" #8 0x00007f156e824296 in call_resume_wind (stub=0x7f156cf55444) at call-stub.c:2467 __FUNCTION__ = "call_resume_wind" #9 0x00007f156e82a020 in call_resume (stub=0x7f156cf55444) at call-stub.c:3861 old_THIS = 0x63b010 __FUNCTION__ = "call_resume" #10 0x00007f156b1f75c6 in iot_worker (data=0x641b90) at io-threads.c:129 conf = 0x641b90 this = 0x63b010 stub = 0x7f156cf55444 sleep_till = {tv_sec = 1307691192, tv_nsec = 0} ret = 0 timeout = 0 '\000' bye = 0 '\000' __FUNCTION__ = "iot_worker" #11 0x00007f156df955f0 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #12 0x00007f156dd0487d in clone () from /lib64/libc.so.6 No symbol table info available. #13 0x0000000000000000 in ?? () No symbol table info available.