+++ This bug was initially created as a clone of Bug #1221605 +++
Description of problem:
=======================
Scrub.log increases rapidly and grows upto 24GB in a span of about 10 hours and root filesystem gets full
[root@rhs-arch-srv3 glusterfs]# du -sch scrub.log
24G scrub.log
24G total
Version-Release number of selected component (if applicable):
=============================================================
gluster --version
glusterfs 3.7.0beta2 built on May 11 2015 01:27:46
How reproducible:
================
1/1
Steps to Reproduce:
==================
1.Created some EC and tiered volumes and had USS, quota and bitrot enabled.
2.Scheduled some snapshots on the volumes
3.Some bricks crashed - tracked by BZ 1221577
4.[root@rhs-arch-srv3 glusterfs]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_rhsarchsrv3-lv_root
26G 26G 0 100% /
[root@rhs-arch-srv3 glusterfs]# du -sch *
3.6M bitd.log
940K bricks
8.0K cli.log
4.0K cmd_history.log
208K etc-glusterfs-glusterd.vol.log
4.0K gcron.log
4.0K geo-replication
8.0K geo-replication-slaves
448K glustershd.log
680K nfs.log
476K quotad.log
24G scrub.log
72K snaps
4.0K snap_scheduler.log
16K var-run-gluster-shared_storage.log
28K vol0-quota-crawl.log
24K vol0-rebalance.log
24K vol1-quota-crawl.log
12K vol2-quota-crawl.log
24G total
----------------Part of scrub.log-----------------
[2015-05-14 08:42:50.132193] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132209] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132216] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132239] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e18 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132250] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132262] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132269] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132280] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e19 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132301] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132310] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132317] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132327] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1a Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132336] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132345] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132351] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132361] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1b Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132371] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132379] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132386] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132396] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1c Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132405] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132414] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132420] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132430] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1d Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132440] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132459] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132466] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132477] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132486] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132495] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132501] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132511] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1f Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132521] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132530] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132536] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132546] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e20 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132555] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132564] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132570] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132580] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e21 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132590] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132598] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132605] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132615] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e22 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132624] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132633] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132639] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132649] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e23 P
Actual results:
Expected results:
Additional info:
Unfortunately glusterfs-3.7.2 did not contain a code change that was associated with this bug report. This bug is now proposed to be a blocker for glusterfs-3.7.3.