Bug 1479528
| Summary: | Rebalance estimate(ETA) shows wrong details(as intial message of 10min wait reappears) when still in progress | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Nithya Balachandran <nbalacha> | |
| Component: | distribute | Assignee: | Nithya Balachandran <nbalacha> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | mainline | CC: | bugs, nchilaka, rhs-bugs, storage-qa-internal, tdesala | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-4.0.0 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1479446 | |||
| : | 1511271 (view as bug list) | Environment: | ||
| Last Closed: | 2018-03-15 11:17:56 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1479446 | |||
| Bug Blocks: | 1511271, 1511274 | |||
|
Comment 1
Worker Ant
2017-08-08 17:46:05 UTC
Description of problem:
==============================
I did a removebrick operation to convert 2x2 to 1x2 , while IOs were going on from 3 different ganesha mounts.
I noticed that at a later stage(may be >80% completed), the message of "The estimated time for rebalance to complete will be unavailable for the first 10 minutes." appears again.
I thinks this comes when the rebalance estimated time is over, but rebalance as such is not yet completed
Last login: Tue Aug 8 19:32:38 2017 from 10.70.35.77
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 5145 7.4MB 10594 0 0 in progress 0:06:38
server2 4142 21.7MB 8722 0 0 in progress 0:06:38
The estimated time for rebalance to complete will be unavailable for the first 10 minutes.
volume rebalance: nrep2: success
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 5993 31.3MB 11970 0 0 in progress 0:08:38
server2 5050 26.6MB 10415 0 0 in progress 0:08:38
The estimated time for rebalance to complete will be unavailable for the first 10 minutes.
volume rebalance: nrep2: success
[root@server1 ~]# gluster v rebal nrep2 status Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 8059 62.0MB 16022 0 0 in progress 0:13:13
server2 7208 76.2MB 14071 0 0 in progress 0:13:13
Estimated time left for rebalance to complete : 0:47:28
volume rebalance: nrep2: success
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 10699 110.9MB 21188 0 0 in progress 0:19:58
server2 9949 119.4MB 16739 0 0 in progress 0:19:58
Estimated time left for rebalance to complete : 0:47:25
volume rebalance: nrep2: success
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 16839 151.7MB 28114 0 0 in progress 0:33:23
server2 16754 184.3MB 27528 0 0 in progress 0:33:23
Estimated time left for rebalance to complete : 0:00:48
volume rebalance: nrep2: success
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 20687 192.2MB 32058 0 0 in progress 0:39:16
server2 20965 189.6MB 32669 0 0 in progress 0:39:16
Estimated time left for rebalance to complete : 0:00:06
volume rebalance: nrep2: success
============== SEE FROM BELOW ==================
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 21521 192.8MB 33069 0 0 in progress 0:40:28
server2 22456 189.6MB 35708 0 0 in progress 0:40:28
The estimated time for rebalance to complete will be unavailable for the first 10 minutes.
volume rebalance: nrep2: success
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 21669 192.8MB 33372 0 0 in progress 0:40:36
server2 22614 189.6MB 35708 0 0 in progress 0:40:36
The estimated time for rebalance to complete will be unavailable for the first 10 minutes.
volume rebalance: nrep2: success
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 21718 192.8MB 33372 0 0 in progress 0:40:40
server2 22667 189.6MB 36020 0 0 in progress 0:40:40
The estimated time for rebalance to complete will be unavailable for the first 10 minutes.
volume rebalance: nrep2: success
[root@server1 ~]#
[root@server1 ~]# gluster v rebal nrep2 status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 23842 194.1MB 37488 0 0 in progress 0:43:47
server2 23440 285.5MB 39635 0 0 completed 0:43:29
The estimated time for rebalance to complete will be unavailable for the first 10 minutes.
volume rebalance: nrep2: success
Version-Release number of selected component (if applicable):
[root@server1 ~]# rpm -qa|grep gluster
glusterfs-api-3.8.4-38.el7rhgs.x86_64
python-gluster-3.8.4-34.el7rhgs.noarch
glusterfs-server-3.8.4-38.el7rhgs.x86_64
gluster-nagios-addons-0.2.9-1.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-16.el7rhgs.x86_64
glusterfs-3.8.4-38.el7rhgs.x86_64
glusterfs-cli-3.8.4-38.el7rhgs.x86_64
glusterfs-rdma-3.8.4-38.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.2.x86_64
vdsm-gluster-4.17.33-1.2.el7rhgs.noarch
glusterfs-libs-3.8.4-38.el7rhgs.x86_64
glusterfs-fuse-3.8.4-38.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-38.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-38.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-38.el7rhgs.x86_64
Steps to Reproduce:
1.had a 1x2 volume add-brick to convert 2x2 and rebalance was done(with some files skipped)
2.did linux untar from one client, lookups from another client(going on till end)
rename,move,chmod,chgrp from another client , but for only sometime, that too these operations were complete much before the rebalance was at this state.
3.observed rebalance eta
Actual results:
==========
again eta starts to show the initial 10 min wait message
COMMIT: https://review.gluster.org/18000 committed in master by ------------- cli: correct rebalance status elapsed check Check that elapsed time has crossed 10 mins for at least one rebalance process before displaying the estimates. Change-Id: Ib357a6f0d0125a178e94ede1e31514fdc6ce3593 BUG: 1479528 Signed-off-by: N Balachandran <nbalacha> REVIEW: https://review.gluster.org/18698 (cli: correct rebalance status elapsed check) posted (#1) for review on release-3.12 by N Balachandran REVISION POSTED: https://review.gluster.org/18698 (cli: correct rebalance status elapsed check) posted (#2) for review on release-3.12 by N Balachandran This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.0.0, please open a new bug report. glusterfs-4.0.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2018-March/000092.html [2] https://www.gluster.org/pipermail/gluster-users/ |