Bug 1012898 - [RHSC] - Monitoring rebalance from CLI does not work.
[RHSC] - Monitoring rebalance from CLI does not work.
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhsc (Show other bugs)
2.1
Unspecified Unspecified
medium Severity medium
: ---
: RHGS 2.1.2
Assigned To: Sahina Bose
RamaKasturi
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-27 06:49 EDT by RamaKasturi
Modified: 2015-05-13 12:33 EDT (History)
9 users (show)

See Also:
Fixed In Version: cb3
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-25 02:45:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Attaching vdsm log (8.67 MB, text/x-log)
2013-09-30 05:10 EDT, RamaKasturi
no flags Details
Attaching rebalance log for the volume (7.89 KB, text/x-log)
2013-09-30 05:16 EDT, RamaKasturi
no flags Details
Attaching engine log (117.50 KB, text/x-log)
2013-09-30 05:23 EDT, RamaKasturi
no flags Details
Attaching engine log (1.13 MB, text/x-log)
2013-09-30 10:19 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (418.29 KB, text/x-log)
2013-09-30 10:20 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (418.73 KB, text/x-log)
2013-09-30 10:20 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (376.57 KB, text/x-log)
2013-09-30 10:21 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (422.98 KB, text/x-log)
2013-09-30 10:21 EDT, RamaKasturi
no flags Details
Attaching new engine log (1.13 MB, text/x-log)
2013-09-30 10:22 EDT, RamaKasturi
no flags Details
Attaching new set of engine logs (1.46 MB, text/x-log)
2013-10-04 05:18 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (972.81 KB, text/x-log)
2013-10-04 05:19 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (1018.07 KB, text/x-log)
2013-10-04 05:20 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (1018.07 KB, text/x-log)
2013-10-04 05:20 EDT, RamaKasturi
no flags Details
Attaching new set of vdsm logs (962.03 KB, text/x-log)
2013-10-04 05:21 EDT, RamaKasturi
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 19863 None None None Never
Red Hat Product Errata RHEA-2014:0208 normal SHIPPED_LIVE Red Hat Storage 2.1 enhancement and bug fix update #2 2014-02-25 07:20:30 EST

  None (edit)
Description RamaKasturi 2013-09-27 06:49:15 EDT
Description of problem:
Monitoring rebalance from CLI does not work.

Version-Release number of selected component (if applicable):
rhsc-2.1.1-0.0.1.master.el6ev.noarch
vdsm-4.12.0-142.git6137a58.el6.x86_64
glusterfs-3.4.1rc1-1.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Login to rhsc.
2. Create volume from cli and start the volume.
3. Start rebalance on the volume from CLI using the command "gluster vol rebalance <volName> start"

Actual results:
1)rebalance starts sucessfully from CLI but does not get updated in GUI.
2)Reblance start icon does not come up in activities column.
3)No event messages coming up.
4)No tasks present in the task pane.
5)when clicked on rebalance status button it shows "No rebalance has ever run on this volume:vol1"
6)From cli when queried for the status it gives the following output.
gluster vol rebalance vol_dis status
                                    Node Rebalanced-files          size       scanned      failures       skipped         status run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------   ------------   --------------
                               localhost                0        0Bytes             0             0             0      completed             0.00
                             10.70.37.51                0        0Bytes             0             0             0      completed             0.00


Expected results:
1) Rebalance should start sucessfully and shown up in the UI.
2) Rebalance start icon should come up in activities column.
3) Event message should come saying "Rebalance has started on this volume".
4) Tasks pane should have the details of rebalance started.
5)When clicked on rebalance status it should give the same output as CLI.

Additional info:
Comment 2 Sahina Bose 2013-09-30 01:44:41 EDT
Please attach vdsm and engine logs
Comment 3 Sahina Bose 2013-09-30 01:47:45 EDT
Also, please note that if rebalance has been completed, this task will not be added to engine to monitor. The task polling is currently set up to be 1 minute
So if the rebalance has completed within that 1 minute, this will not be monitored as task has been completed.
Is this the case?
Comment 4 RamaKasturi 2013-09-30 05:10:31 EDT
Created attachment 805010 [details]
Attaching vdsm log
Comment 5 RamaKasturi 2013-09-30 05:16:36 EDT
Created attachment 805024 [details]
Attaching rebalance log for the volume
Comment 6 RamaKasturi 2013-09-30 05:19:54 EDT
Rebalanced a file which takes more than 1 minute. But i am still not able to see UI getting synced. Please find the status output from gluster cli.

[root@localhost ~]# gluster vol rebalance vol_dis status
                                    Node Rebalanced-files          size       scanned      failures       skipped         status run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------   ------------   --------------
                               localhost                0        0Bytes            20             0             2      completed             0.00
                             10.70.37.51                1      1000.0MB            20             0             0      completed            72.00
Comment 7 RamaKasturi 2013-09-30 05:23:31 EDT
Created attachment 805025 [details]
Attaching engine log
Comment 8 Sahina Bose 2013-09-30 05:33:12 EDT
I don't see any exception in log.
From engine log, it looks like a Job has been created.
The vdsm log is only partial - so I see only the tasksList command returning with completed status.

I suspect, the Job is marked completed and cleaned up. Could you try with a longer running task?
Comment 9 Sahina Bose 2013-09-30 05:33:39 EDT
I don't see any exception in log.
From engine log, it looks like a Job has been created.
The vdsm log is only partial - so I see only the tasksList command returning with completed status.

I suspect, the Job is marked completed and cleaned up. Could you try with a longer running task?
Comment 10 RamaKasturi 2013-09-30 10:19:03 EDT
Created attachment 805229 [details]
Attaching engine log
Comment 11 RamaKasturi 2013-09-30 10:20:04 EDT
Created attachment 805230 [details]
Attaching new set of vdsm logs
Comment 12 RamaKasturi 2013-09-30 10:20:42 EDT
Created attachment 805231 [details]
Attaching new set of vdsm logs
Comment 13 RamaKasturi 2013-09-30 10:21:09 EDT
Created attachment 805233 [details]
Attaching new set of vdsm logs
Comment 14 RamaKasturi 2013-09-30 10:21:45 EDT
Created attachment 805234 [details]
Attaching new set of vdsm logs
Comment 15 RamaKasturi 2013-09-30 10:22:20 EDT
Created attachment 805237 [details]
Attaching new engine log
Comment 16 RamaKasturi 2013-09-30 10:32:52 EDT
Hi Sahina,

   I tried  a long running task. The following are the issues i have observed:

1) If the tasks runs for more than a minute, then only the task is getting updated in the tasks pane . what would happen to the tasks which are running less than a minute? I think we should take care of this too.
2) For example, there was a task which ran for 70 sec or less than 100 sec. If engine starts sycning after one min then the tasks which ran for 70/80 secs should also be updated in UI, IMO.
2) Rebalance start icon is not getting updated in the activities column.
3) There is no events message which says rebalance is started.
4) Rebalance start is not getting disabled and stop is not getting enabled.
5) Rebalance status says rebalance has never happened.
6) And the task in the tasks pane is always executing even if the task is completed. just curious to know if there is a time limit even for this? If so, could you please let me know.
7) And is there a way to clear the task or does it get cleared after some time limit?
8) The tasks pane does not get updated with the correct number of files scanned,moved,failed and Total size moved.
Comment 17 Sahina Bose 2013-10-04 04:30:20 EDT
(In reply to RamaKasturi from comment #16)
> Hi Sahina,
> 
>    I tried  a long running task. The following are the issues i have
> observed:
> 
> 1) If the tasks runs for more than a minute, then only the task is getting
> updated in the tasks pane . what would happen to the tasks which are running
> less than a minute? I think we should take care of this too.

This can be taken care of by reducing the monitoring interval for tasks. Currently it's set to 1 minute. It can be reduced to 10 secs or lesser

> 2) For example, there was a task which ran for 70 sec or less than 100 sec.
> If engine starts sycning after one min then the tasks which ran for 70/80
> secs should also be updated in UI, IMO.

Same as above

> 2) Rebalance start icon is not getting updated in the activities column.

This was a bug which has been addressed with patch http://gerrit.ovirt.org/#/c/19616/ and fixed in cb2 build 

> 3) There is no events message which says rebalance is started.

Please log a separate bug on this - I assume this is for when rebalance is started from cli

> 4) Rebalance start is not getting disabled and stop is not getting enabled.

Fixed in cb2

> 5) Rebalance status says rebalance has never happened.

Fixed in cb2

> 6) And the task in the tasks pane is always executing even if the task is
> completed. just curious to know if there is a time limit even for this? If
> so, could you please let me know.

Fixed in cb2

> 7) And is there a way to clear the task or does it get cleared after some
> time limit?

The tasks get cleared after a time limit - 10 mins for succeeded jobs and 60 mins for failed jobs

> 8) The tasks pane does not get updated with the correct number of files
> scanned,moved,failed and Total size moved.

Related to point 6 and fixed in cb2
Comment 18 RamaKasturi 2013-10-04 05:17:28 EDT
(In reply to Sahina Bose from comment #17)
> (In reply to RamaKasturi from comment #16)
> > Hi Sahina,
> > 
> >    I tried  a long running task. The following are the issues i have
> > observed:
> > 
> > 1) If the tasks runs for more than a minute, then only the task is getting
> > updated in the tasks pane . what would happen to the tasks which are running
> > less than a minute? I think we should take care of this too.
> 
> This can be taken care of by reducing the monitoring interval for tasks.
> Currently it's set to 1 minute. It can be reduced to 10 secs or lesser
> 
> > 2) For example, there was a task which ran for 70 sec or less than 100 sec.
> > If engine starts sycning after one min then the tasks which ran for 70/80
> > secs should also be updated in UI, IMO.
> 
> Same as above
> 
> > 2) Rebalance start icon is not getting updated in the activities column.
> 
> This was a bug which has been addressed with patch
> http://gerrit.ovirt.org/#/c/19616/ and fixed in cb2 build 

Works in cb2.
> 
> > 3) There is no events message which says rebalance is started.
> 
> Please log a separate bug on this - I assume this is for when rebalance is
> started from cli

Will log a separate BZ on the same.
> 
> > 4) Rebalance start is not getting disabled and stop is not getting enabled.
> 
> Fixed in cb2

works in cb2.
> 
> > 5) Rebalance status says rebalance has never happened.
> 
> Fixed in cb2

In cb2 status dialog hungs.
> 
> > 6) And the task in the tasks pane is always executing even if the task is
> > completed. just curious to know if there is a time limit even for this? If
> > so, could you please let me know.
> 
> Fixed in cb2
> 
This is still not working.
> > 7) And is there a way to clear the task or does it get cleared after some
> > time limit?
> 
> The tasks get cleared after a time limit - 10 mins for succeeded jobs and 60
> mins for failed jobs
> 
> > 8) The tasks pane does not get updated with the correct number of files
> > scanned,moved,failed and Total size moved.
> 
> Related to point 6 and fixed in cb2

This is still not working.
Comment 19 RamaKasturi 2013-10-04 05:18:59 EDT
Created attachment 807507 [details]
Attaching new set of engine logs
Comment 20 RamaKasturi 2013-10-04 05:19:32 EDT
Created attachment 807508 [details]
Attaching new set of vdsm logs
Comment 21 RamaKasturi 2013-10-04 05:20:08 EDT
Created attachment 807509 [details]
Attaching new set of vdsm logs
Comment 22 RamaKasturi 2013-10-04 05:20:41 EDT
Created attachment 807510 [details]
Attaching new set of vdsm logs
Comment 23 RamaKasturi 2013-10-04 05:21:09 EDT
Created attachment 807511 [details]
Attaching new set of vdsm logs
Comment 24 RamaKasturi 2013-10-04 05:22:00 EDT
My replies are in line and attached new set of logs.
Comment 25 Sahina Bose 2013-10-04 06:31:57 EDT
Point 6 & point 8 are not working due to overflow errors returned from vdsm.
The patch linked here fixes that.
Comment 26 Sahina Bose 2013-10-04 06:36:15 EDT
Status dialog - I think there's a separate bug filed on that - Bug 1012329
So will not be addressing that fix here.
Comment 27 RamaKasturi 2013-10-17 08:33:50 EDT
Verified in RHS console rhsc-2.1.2-0.0.scratch.beta1.el6_4.noarch (cb4). Logged a separate bug for point no.3 and all the remaining steps mentioned above are working.
Comment 29 errata-xmlrpc 2014-02-25 02:45:29 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0208.html

Note You need to log in before you can comment on or make changes to this bug.