Bug 1549714
| Summary: | On sharded tiered volume, only first shard of new file goes on hot tier. | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Jeff Byers <jbyers> |
| Component: | tiering | Assignee: | bugs <bugs> |
| Status: | CLOSED WONTFIX | QA Contact: | bugs <bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | mainline | CC: | bugs, hgichon, vnosov |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-11-02 08:13:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions. Patch https://review.gluster.org/#/c/glusterfs/+/21331/ removes tier functionality from GlusterFS. https://bugzilla.redhat.com/show_bug.cgi?id=1642807 is used as the tracking bug for this. Recommendation is to convert your tier volume to regular volume (either replicate, ec, or plain distribute) with "tier detach" command before upgrade, and use backend features like dm-cache etc to utilize the caching from backend to provide better performance and functionality. |
On sharded tiered volume, only first shard of new file goes on hot tier. On a sharded tiered volume, only the first shard of a new file goes on the hot tier, the rest are written to the cold tier. This is unfortunate for archival applications where the hot tier is fast, but the cold tier is very slow. After the tier- promote-frequency (default 120 seconds), all of the shards do migrate to hot tier, but for archival applications, this migration is not helpful since the file is likely not be accessed again just after being copied on, and it will later just migrate back to the cold tier. Sharding should be, and really needs to be used with very large archive files because of bug: Bug 1277112 - Data Tiering:File create and new writes to existing file fails when the hot tier is full instead of redirecting/flushing the data to cold tier which sharding with tiering helps mitigate. This problem occurs in GlusterFS 3.7.18 and 3.12.3, at least. I/O size doesn't make any difference, I tried multiple sizes. I didn't find any volume configuration options that helped in getting all of the new shards to go directly to the hot tier. # dd if=/dev/root bs=64M of=/samba/tiered-sharded-vol/file-1 count=6 402653184 bytes (403 MB) copied, 1.31154 s, 307 MB/s # ls -lrtdh /samba/tiered-sharded-vol/* /exports/brick-*/tiered-sharded-vol/* /exports/brick-*/tiered-sharded-vol/.shard/* 2>/dev/null ---------T 0 Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/file-1 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/file-1 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4 -rw-r--r-- 384M Feb 27 08:58 /samba/tiered-sharded-vol/file-1 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5 # sleep 120 # ls -lrtdh /samba/tiered-sharded-vol/* /exports/brick-*/tiered-sharded-vol/* /exports/brick-*/tiered-sharded-vol/.shard/* 2>/dev/null ---------T 0 Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/file-1 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/file-1 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4 -rw-r--r-- 64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5 -rw-r--r-- 384M Feb 27 08:58 /samba/tiered-sharded-vol/file-1 ---------T 0 Feb 27 09:00 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5 ---------T 0 Feb 27 09:00 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1 ---------T 0 Feb 27 09:00 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2 ---------T 0 Feb 27 09:00 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3 ---------T 0 Feb 27 09:00 /exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4 Volume Name: tiered-sharded-vol Type: Tier Volume ID: 8c09077a-371e-4d30-9faa-c9c76d7b1b57 Status: Started Number of Bricks: 2 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: 10.10.60.169:/exports/brick-hot/tiered-sharded-vol Cold Tier: Cold Tier Type : Distribute Number of Bricks: 1 Brick2: 10.10.60.169:/exports/brick-cold/tiered-sharded-vol Options Reconfigured: cluster.tier-mode: cache features.ctr-enabled: on features.shard: on features.shard-block-size: 64MB server.allow-insecure: on performance.quick-read: off performance.stat-prefetch: off nfs.disable: on nfs.addr-namelookup: off performance.readdir-ahead: on snap-activate-on-create: enable cluster.enable-shared-storage: disable