Bug 1360317
Summary: | [GSS] glusterfs doesn't respect cluster.min-free-disk on remove-brick operation | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Mateusz Mazur <mmazurx> |
Component: | distribute | Assignee: | Susant Kumar Palai <spalai> |
Status: | CLOSED ERRATA | QA Contact: | Prasad Desala <tdesala> |
Severity: | urgent | Docs Contact: | |
Priority: | urgent | ||
Version: | rhgs-3.1 | CC: | agk, amukherj, asrivast, atumball, bkunal, nbalacha, rcyriac, rgowdapp, rhinduja, rhs-bugs, rnalakka, spalai, tdesala, yjog |
Target Milestone: | --- | ||
Target Release: | RHGS 3.3.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | glusterfs-3.8.4-25 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-09-21 04:28:23 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1441508, 1473132, 1473133 | ||
Bug Blocks: | 1369781, 1417145 |
Description
Mateusz Mazur
2016-07-26 12:26:01 UTC
__dht_check_free_space (dht-rebalance.c), as can be seen below checks whether dst has enough space to accomodate the file that is being migrated. Nowhere it is checking whether dst has free space configured in 'min-free-disk'. Now, since parallel migrations can happen to same brick (either from same rebalance process - multithreaded rebalance - or from multiple rebalance processes), we can end up saying yes to more than one parallel migrations of different files when there is only enough free space to hold the largest of files, which can result in ENOSPC during migration. check_avail_space: if (((dst_statfs.f_bavail * dst_statfs.f_bsize) / GF_DISK_SECTOR_SIZE) < stbuf->ia_blocks) { gf_msg (this->name, GF_LOG_ERROR, 0, DHT_MSG_MIGRATE_FILE_FAILED, "data movement attempted from node (%s) to node (%s) " "which does not have required free space for (%s)", from->name, to->name, loc->path); ret = -1; goto out; } We should either: * use a buffer space to account for parallel migrations (like a min-free-disk relatively larger than largest file size) * make check_free_space resilient against parallel migrations (may be by atomically decrementing free space in statfs by the size of file being migrated etc). regards, Raghavendra upstream patch : https://review.gluster.org/#/c/17034/ downstream patches: https://code.engineering.redhat.com/gerrit/#/c/103914/ https://code.engineering.redhat.com/gerrit/#/c/103915/ Verified this BZ on glusterfs version: 3.8.4-36.el7rhgs.x86_64. This issue is fixed, now remove-brick is considering cluster.min-free-disk values during file migration. Moving this BZ to Verified state. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774 |