Bug 1370683
| Summary: | "rpc_clnt_ping_timer_expired" when copying big file (reproducible) | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Christopher Pereira <kripper> |
| Component: | fuse | Assignee: | bugs <bugs> |
| Status: | CLOSED EOL | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.7.11 | CC: | bugs, kaushal, rgowdapp |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | rpc-ping-timeout | ||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-03-08 10:52:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Christopher Pereira
2016-08-27 01:04:00 UTC
Hi, It would help us if you could provide the following additional information. - The volume type of the volume to which the file is being copied to. The output of `gluster volume info` will be good. - Is the timer expiry always for the same server? In the logs you've pasted, <host-9> has timed out. - The brick logs from <host-9> or the server which timed-out will be helpful. It can help identify why the brick is blocked. - Just to clarify again, it's just this one file which causes problems, and nothing else. Thanks. Hi Kaushal, 1) Volume Type: Volume Name: backups-h9 Type: Distribute Volume ID: 782b8005-6db7-4b91-9854-a9a7ae326fef Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: <host-9>:/home/datacenter/gluster-bricks/backups-h9 Options Reconfigured: performance.readdir-ahead: on 2) Yes, <host-9> is the gluster server. 3) Brick logs (mount point is on <host-8>): [2016-08-27 00:27:00.792426] W [socket.c:1236:__socket_read_simple_msg] 0-tcp.backups-h9-server: reading from socket failed. Error (Connection timed out), peer (200.63.97.100:65526) [2016-08-27 00:27:00.792708] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-backups-h9-server: disconnecting connection from <host-8>-4531-2016/05/18-13:54:19:979576-backups-h9-client-0-0-115 [2016-08-27 00:27:00.792761] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-backups-h9-server: fd cleanup on /ixdc/images/snapshots/ixdc.vda.dOUjqL.tmp2 [2016-08-27 00:27:00.792807] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-backups-h9-server: fd cleanup on /ixdc/images/snapshots/ixdc.vda.dOUjqL.tmp [2016-08-27 00:27:00.792931] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-backups-h9-server: Shutting down connection <host-8>-4531-2016/05/18-13:54:19:979576-backups-h9-client-0-0-115 [2016-08-27 00:27:44.561842] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-backups-h9-server: accepted client from <host-8>-4531-2016/05/18-13:54:19:979576-backups-h9-client-0-0-116 (version: 3.7.11) [2016-08-27 00:27:00.792794] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-backups-h9-server: fd cleanup on /ixdc/images/snapshots/ixdc.vda.dOUjqL.tmp2 [2016-08-27 00:45:48.792359] W [socket.c:1236:__socket_read_simple_msg] 0-tcp.backups-h9-server: reading from socket failed. Error (Connection timed out), peer (200.63.97.100:65525) [2016-08-27 00:45:48.792446] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-backups-h9-server: disconnecting connection from <host-8>-4531-2016/05/18-13:54:19:979576-backups-h9-client-0-0-116 [2016-08-27 00:45:48.792500] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-backups-h9-server: fd cleanup on /ixdc/images/snapshots/ixdc.vda.dOUjqL.tmp2 [2016-08-27 00:45:48.792542] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-backups-h9-server: fd cleanup on /ixdc/images/snapshots/ixdc.vda.dOUjqL.tmp3 [2016-08-27 00:45:48.792663] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-backups-h9-server: Shutting down connection <host-8>-4531-2016/05/18-13:54:19:979576-backups-h9-client-0-0-116 [2016-08-27 00:45:48.792531] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-backups-h9-server: fd cleanup on /ixdc/images/snapshots/ixdc.vda.dOUjqL.tmp2 [2016-08-27 00:45:58.712780] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-backups-h9-server: accepted client from <host-8>-4531-2016/05/18-13:54:19:979576-backups-h9-client-0-0-117 (version: 3.7.11) [2016-08-29 06:07:02.214318] I [MSGID: 100011] [glusterfsd.c:1323:reincarnate] 0-glusterfsd: Fetching the volume file from server... 4) Yes, file transfers have been working for months without issues and this is the only conflicting file. It fails each time I retry to 'cp' it. Do you see something suspicious? PS: I have no control or information about the firewalls in the middle. Maybe there is some software detecting a given byte sequence and dropping the connection. This bug is getting closed because GlusteFS-3.7 has reached its end-of-life. Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS. If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release. |