Description of problem: Monitoring rebalance from CLI does not work. Version-Release number of selected component (if applicable): rhsc-2.1.1-0.0.1.master.el6ev.noarch vdsm-4.12.0-142.git6137a58.el6.x86_64 glusterfs-3.4.1rc1-1.el6.x86_64 How reproducible: Always Steps to Reproduce: 1. Login to rhsc. 2. Create volume from cli and start the volume. 3. Start rebalance on the volume from CLI using the command "gluster vol rebalance <volName> start" Actual results: 1)rebalance starts sucessfully from CLI but does not get updated in GUI. 2)Reblance start icon does not come up in activities column. 3)No event messages coming up. 4)No tasks present in the task pane. 5)when clicked on rebalance status button it shows "No rebalance has ever run on this volume:vol1" 6)From cli when queried for the status it gives the following output. gluster vol rebalance vol_dis status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 0 0 0 completed 0.00 10.70.37.51 0 0Bytes 0 0 0 completed 0.00 Expected results: 1) Rebalance should start sucessfully and shown up in the UI. 2) Rebalance start icon should come up in activities column. 3) Event message should come saying "Rebalance has started on this volume". 4) Tasks pane should have the details of rebalance started. 5)When clicked on rebalance status it should give the same output as CLI. Additional info:
Please attach vdsm and engine logs
Also, please note that if rebalance has been completed, this task will not be added to engine to monitor. The task polling is currently set up to be 1 minute So if the rebalance has completed within that 1 minute, this will not be monitored as task has been completed. Is this the case?
Created attachment 805010 [details] Attaching vdsm log
Created attachment 805024 [details] Attaching rebalance log for the volume
Rebalanced a file which takes more than 1 minute. But i am still not able to see UI getting synced. Please find the status output from gluster cli. [root@localhost ~]# gluster vol rebalance vol_dis status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 20 0 2 completed 0.00 10.70.37.51 1 1000.0MB 20 0 0 completed 72.00
Created attachment 805025 [details] Attaching engine log
I don't see any exception in log. From engine log, it looks like a Job has been created. The vdsm log is only partial - so I see only the tasksList command returning with completed status. I suspect, the Job is marked completed and cleaned up. Could you try with a longer running task?
Created attachment 805229 [details] Attaching engine log
Created attachment 805230 [details] Attaching new set of vdsm logs
Created attachment 805231 [details] Attaching new set of vdsm logs
Created attachment 805233 [details] Attaching new set of vdsm logs
Created attachment 805234 [details] Attaching new set of vdsm logs
Created attachment 805237 [details] Attaching new engine log
Hi Sahina, I tried a long running task. The following are the issues i have observed: 1) If the tasks runs for more than a minute, then only the task is getting updated in the tasks pane . what would happen to the tasks which are running less than a minute? I think we should take care of this too. 2) For example, there was a task which ran for 70 sec or less than 100 sec. If engine starts sycning after one min then the tasks which ran for 70/80 secs should also be updated in UI, IMO. 2) Rebalance start icon is not getting updated in the activities column. 3) There is no events message which says rebalance is started. 4) Rebalance start is not getting disabled and stop is not getting enabled. 5) Rebalance status says rebalance has never happened. 6) And the task in the tasks pane is always executing even if the task is completed. just curious to know if there is a time limit even for this? If so, could you please let me know. 7) And is there a way to clear the task or does it get cleared after some time limit? 8) The tasks pane does not get updated with the correct number of files scanned,moved,failed and Total size moved.
(In reply to RamaKasturi from comment #16) > Hi Sahina, > > I tried a long running task. The following are the issues i have > observed: > > 1) If the tasks runs for more than a minute, then only the task is getting > updated in the tasks pane . what would happen to the tasks which are running > less than a minute? I think we should take care of this too. This can be taken care of by reducing the monitoring interval for tasks. Currently it's set to 1 minute. It can be reduced to 10 secs or lesser > 2) For example, there was a task which ran for 70 sec or less than 100 sec. > If engine starts sycning after one min then the tasks which ran for 70/80 > secs should also be updated in UI, IMO. Same as above > 2) Rebalance start icon is not getting updated in the activities column. This was a bug which has been addressed with patch http://gerrit.ovirt.org/#/c/19616/ and fixed in cb2 build > 3) There is no events message which says rebalance is started. Please log a separate bug on this - I assume this is for when rebalance is started from cli > 4) Rebalance start is not getting disabled and stop is not getting enabled. Fixed in cb2 > 5) Rebalance status says rebalance has never happened. Fixed in cb2 > 6) And the task in the tasks pane is always executing even if the task is > completed. just curious to know if there is a time limit even for this? If > so, could you please let me know. Fixed in cb2 > 7) And is there a way to clear the task or does it get cleared after some > time limit? The tasks get cleared after a time limit - 10 mins for succeeded jobs and 60 mins for failed jobs > 8) The tasks pane does not get updated with the correct number of files > scanned,moved,failed and Total size moved. Related to point 6 and fixed in cb2
(In reply to Sahina Bose from comment #17) > (In reply to RamaKasturi from comment #16) > > Hi Sahina, > > > > I tried a long running task. The following are the issues i have > > observed: > > > > 1) If the tasks runs for more than a minute, then only the task is getting > > updated in the tasks pane . what would happen to the tasks which are running > > less than a minute? I think we should take care of this too. > > This can be taken care of by reducing the monitoring interval for tasks. > Currently it's set to 1 minute. It can be reduced to 10 secs or lesser > > > 2) For example, there was a task which ran for 70 sec or less than 100 sec. > > If engine starts sycning after one min then the tasks which ran for 70/80 > > secs should also be updated in UI, IMO. > > Same as above > > > 2) Rebalance start icon is not getting updated in the activities column. > > This was a bug which has been addressed with patch > http://gerrit.ovirt.org/#/c/19616/ and fixed in cb2 build Works in cb2. > > > 3) There is no events message which says rebalance is started. > > Please log a separate bug on this - I assume this is for when rebalance is > started from cli Will log a separate BZ on the same. > > > 4) Rebalance start is not getting disabled and stop is not getting enabled. > > Fixed in cb2 works in cb2. > > > 5) Rebalance status says rebalance has never happened. > > Fixed in cb2 In cb2 status dialog hungs. > > > 6) And the task in the tasks pane is always executing even if the task is > > completed. just curious to know if there is a time limit even for this? If > > so, could you please let me know. > > Fixed in cb2 > This is still not working. > > 7) And is there a way to clear the task or does it get cleared after some > > time limit? > > The tasks get cleared after a time limit - 10 mins for succeeded jobs and 60 > mins for failed jobs > > > 8) The tasks pane does not get updated with the correct number of files > > scanned,moved,failed and Total size moved. > > Related to point 6 and fixed in cb2 This is still not working.
Created attachment 807507 [details] Attaching new set of engine logs
Created attachment 807508 [details] Attaching new set of vdsm logs
Created attachment 807509 [details] Attaching new set of vdsm logs
Created attachment 807510 [details] Attaching new set of vdsm logs
Created attachment 807511 [details] Attaching new set of vdsm logs
My replies are in line and attached new set of logs.
Point 6 & point 8 are not working due to overflow errors returned from vdsm. The patch linked here fixes that.
Status dialog - I think there's a separate bug filed on that - Bug 1012329 So will not be addressing that fix here.
Verified in RHS console rhsc-2.1.2-0.0.scratch.beta1.el6_4.noarch (cb4). Logged a separate bug for point no.3 and all the remaining steps mentioned above are working.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0208.html