Bug 1221605

Summary: Scrub.log grows rapidly and the size increases upto 24GB in a span of 10 hours
Product: [Community] GlusterFS Reporter: senaik
Component: bitrotAssignee: Venky Shankar <vshankar>
Status: CLOSED EOL QA Contact:
Severity: high Docs Contact: bugs <bugs>
Priority: unspecified    
Version: 3.7.0CC: bugs, smohan, vshankar
Target Milestone: ---Keywords: Triaged
Target Release: ---Flags: ndevos: needinfo?
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1224193 (view as bug list) Environment:
Last Closed: 2017-03-08 10:56:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1207134, 1226146    
Bug Blocks: 1224193, 1233025    

Description senaik 2015-05-14 12:33:00 UTC 
Description of problem:
=======================
Scrub.log increases rapidly and grows upto 24GB in a span of about 10 hours and root filesystem gets full 

[root@rhs-arch-srv3 glusterfs]# du -sch scrub.log 
24G	scrub.log
24G	total


Version-Release number of selected component (if applicable):
=============================================================
gluster --version
glusterfs 3.7.0beta2 built on May 11 2015 01:27:46


How reproducible:
================
1/1

Steps to Reproduce:
==================
1.Created some EC and tiered volumes and had USS, quota and bitrot enabled.

2.Scheduled some snapshots on the volumes 

3.Some bricks crashed - tracked by BZ 1221577

4.[root@rhs-arch-srv3 glusterfs]# df -h 
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhsarchsrv3-lv_root
                       26G   26G     0 100% /


[root@rhs-arch-srv3 glusterfs]# du -sch *
3.6M	bitd.log
940K	bricks
8.0K	cli.log
4.0K	cmd_history.log
208K	etc-glusterfs-glusterd.vol.log
4.0K	gcron.log
4.0K	geo-replication
8.0K	geo-replication-slaves
448K	glustershd.log
680K	nfs.log
476K	quotad.log
24G	scrub.log 
72K	snaps
4.0K	snap_scheduler.log
16K	var-run-gluster-shared_storage.log
28K	vol0-quota-crawl.log
24K	vol0-rebalance.log
24K	vol1-quota-crawl.log
12K	vol2-quota-crawl.log
24G	total


----------------Part of scrub.log-----------------

[2015-05-14 08:42:50.132193] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132209] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132216] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132239] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e18 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132250] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132262] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132269] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132280] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e19 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132301] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132310] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132317] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132327] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1a Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132336] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132345] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132351] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132361] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1b Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132371] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132379] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132386] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132396] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1c Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132405] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132414] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132420] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132430] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1d Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132440] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132459] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132466] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132477] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132486] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132495] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132501] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132511] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1f Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132521] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132530] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132536] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132546] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e20 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132555] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132564] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132570] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132580] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e21 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132590] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132598] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132605] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132615] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e22 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2)
[2015-05-14 08:42:50.132624] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2015-05-14 08:42:50.132633] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected]
[2015-05-14 08:42:50.132639] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2)
[2015-05-14 08:42:50.132649] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e23 P


Actual results:


Expected results:


Additional info:
Comment 1 Niels de Vos 2015-06-02 08:20:18 UTC 
The required changes to fix this bug have not made it into glusterfs-3.7.1. This bug is now getting tracked for glusterfs-3.7.2.
Comment 2 Niels de Vos 2015-06-20 10:08:16 UTC 
Unfortunately glusterfs-3.7.2 did not contain a code change that was associated with this bug report. This bug is now proposed to be a blocker for glusterfs-3.7.3.
Comment 3 Niels de Vos 2015-06-20 10:22:51 UTC 
This bug has status ON_QA, but there is no patch associated with it. If a patch has been posted in relation to an other bug, please close this one as a duplicate. Note that ON_QA only makes sense when "Fixed in version" is set. Furtermore, we only use ON_QA for alpha/beta releases.

Please mark this bug as CLOSED in case all required patches are included in glusterfs-3.7.2. If that is not the case, you will need to move the status back to MODIFIED, POST or ASSIGNED. Thanks!
Comment 5 Kaushal 2017-03-08 10:56:10 UTC 
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life.

Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS.
If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.