+++ This bug was initially created as a clone of Bug #1221605 +++ Description of problem: ======================= Scrub.log increases rapidly and grows upto 24GB in a span of about 10 hours and root filesystem gets full [root@rhs-arch-srv3 glusterfs]# du -sch scrub.log 24G scrub.log 24G total Version-Release number of selected component (if applicable): ============================================================= gluster --version glusterfs 3.7.0beta2 built on May 11 2015 01:27:46 How reproducible: ================ 1/1 Steps to Reproduce: ================== 1.Created some EC and tiered volumes and had USS, quota and bitrot enabled. 2.Scheduled some snapshots on the volumes 3.Some bricks crashed - tracked by BZ 1221577 4.[root@rhs-arch-srv3 glusterfs]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_rhsarchsrv3-lv_root 26G 26G 0 100% / [root@rhs-arch-srv3 glusterfs]# du -sch * 3.6M bitd.log 940K bricks 8.0K cli.log 4.0K cmd_history.log 208K etc-glusterfs-glusterd.vol.log 4.0K gcron.log 4.0K geo-replication 8.0K geo-replication-slaves 448K glustershd.log 680K nfs.log 476K quotad.log 24G scrub.log 72K snaps 4.0K snap_scheduler.log 16K var-run-gluster-shared_storage.log 28K vol0-quota-crawl.log 24K vol0-rebalance.log 24K vol1-quota-crawl.log 12K vol2-quota-crawl.log 24G total ----------------Part of scrub.log----------------- [2015-05-14 08:42:50.132193] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132209] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132216] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132239] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e18 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132250] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132262] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132269] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132280] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e19 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132301] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132310] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132317] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132327] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1a Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132336] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132345] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132351] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132361] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1b Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132371] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132379] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132386] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132396] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1c Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132405] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132414] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132420] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132430] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1d Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132440] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132459] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132466] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132477] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132486] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132495] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132501] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132511] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1f Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132521] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132530] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132536] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132546] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e20 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132555] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132564] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132570] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132580] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e21 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132590] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132598] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132605] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132615] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e22 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132624] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132633] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132639] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132649] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e23 P Actual results: Expected results: Additional info:
The required changes to fix this bug have not made it into glusterfs-3.7.1. This bug is now getting tracked for glusterfs-3.7.2.
Unfortunately glusterfs-3.7.2 did not contain a code change that was associated with this bug report. This bug is now proposed to be a blocker for glusterfs-3.7.3.