Description of problem: ======================= Scrub.log increases rapidly and grows upto 24GB in a span of about 10 hours and root filesystem gets full [root@rhs-arch-srv3 glusterfs]# du -sch scrub.log 24G scrub.log 24G total Version-Release number of selected component (if applicable): ============================================================= gluster --version glusterfs 3.7.0beta2 built on May 11 2015 01:27:46 How reproducible: ================ 1/1 Steps to Reproduce: ================== 1.Created some EC and tiered volumes and had USS, quota and bitrot enabled. 2.Scheduled some snapshots on the volumes 3.Some bricks crashed - tracked by BZ 1221577 4.[root@rhs-arch-srv3 glusterfs]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_rhsarchsrv3-lv_root 26G 26G 0 100% / [root@rhs-arch-srv3 glusterfs]# du -sch * 3.6M bitd.log 940K bricks 8.0K cli.log 4.0K cmd_history.log 208K etc-glusterfs-glusterd.vol.log 4.0K gcron.log 4.0K geo-replication 8.0K geo-replication-slaves 448K glustershd.log 680K nfs.log 476K quotad.log 24G scrub.log 72K snaps 4.0K snap_scheduler.log 16K var-run-gluster-shared_storage.log 28K vol0-quota-crawl.log 24K vol0-rebalance.log 24K vol1-quota-crawl.log 12K vol2-quota-crawl.log 24G total ----------------Part of scrub.log----------------- [2015-05-14 08:42:50.132193] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132209] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132216] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132239] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e18 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132250] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132262] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132269] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132280] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e19 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132301] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132310] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132317] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132327] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1a Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132336] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132345] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132351] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132361] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1b Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132371] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132379] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132386] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132396] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1c Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132405] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132414] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132420] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132430] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1d Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132440] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132459] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132466] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132477] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132486] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132495] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132501] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132511] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e1f Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132521] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132530] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132536] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132546] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e20 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132555] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132564] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132570] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132580] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e21 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132590] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132598] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132605] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132615] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e22 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol0-client-2) [2015-05-14 08:42:50.132624] W [client-rpc-fops.c:2826:client3_3_lookup_cbk] 0-vol0-client-2: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) [2015-05-14 08:42:50.132633] E [bit-rot.c:1252:br_brick_connect] 0-vol0-bit-rot-0: lookup on root failed [Reason: Transport endpoint is not connected] [2015-05-14 08:42:50.132639] E [bit-rot.c:1331:br_handle_events] 0-vol0-bit-rot-0: failed to connect to the child (subvolume: vol0-client-2) [2015-05-14 08:42:50.132649] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vol0-client-2: failed to submit rpc-request (XID: 0x2218e23 P Actual results: Expected results: Additional info:
The required changes to fix this bug have not made it into glusterfs-3.7.1. This bug is now getting tracked for glusterfs-3.7.2.
Unfortunately glusterfs-3.7.2 did not contain a code change that was associated with this bug report. This bug is now proposed to be a blocker for glusterfs-3.7.3.
This bug has status ON_QA, but there is no patch associated with it. If a patch has been posted in relation to an other bug, please close this one as a duplicate. Note that ON_QA only makes sense when "Fixed in version" is set. Furtermore, we only use ON_QA for alpha/beta releases. Please mark this bug as CLOSED in case all required patches are included in glusterfs-3.7.2. If that is not the case, you will need to move the status back to MODIFIED, POST or ASSIGNED. Thanks!
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life. Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS. If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.