Created attachment 1643843 [details] Migration plan in progress video Description of problem: In Migration plans in progress page, the "Datastores" displays the VMs disks total size in phases, instead of at once. In the attached video, 20 VMs migration, with 1 16GB disk each. 1 conversion host, that is set to 10 Max concurrent tasks, so the Migration should happen in 2 phases: 10 VMs, and then the next 10 VMs. At first Datastores display "0.00B out of 154GB". After a few seconds "0.00B out of 343GB". Mike Turley 2019-12-10 20:51:21 UTC: " @Fabien, the logic for displaying the total disk space is just based on the data in those miq_request_tasks. If we don't want to display the incorrect total, we'll need some way of inferring from that data that the sum is still being prepared and shouldn't be displayed yet. If you can help me identify that criteria I can maybe just have it display "Calculating..." or something there. Otherwise, we would just need to make sure the totals in there are always accurate. The code just sums the values of miq_request_tasks[].virtv2v_disks[].size across all disks of all tasks in the request: https://github.com/ManageIQ/manageiq-v2v/blob/master/app/javascript/react/screens/App/Overview/components/Migrations/helpers/inProgressHelpers.js#L91-L106 " Fabien Dupont 2019-12-11 08:21:36 UTC " When we create the ServiceTemplateTransformationPlanRequest, it creates the tasks for the virtual machines that have been successfully migrated. Once the tasks have been created, it creates the InfraConversionJob's that handle the workflow: https://github.com/ManageIQ/manageiq/blob/master/app/models/service_template_transformation_plan_request.rb#L49. The miq_request_task.virtv2v_disks array is populated during preflight check that happens when the throttler tries to allocate a conversion host. It takes a few seconds to happen and the tasks are treated serially, so if you look at the tasks before that have all been throttled at least once, you will have an incomplete data set. Even if we move the preflight check earlier in the process, it might still take some time to process and you will still have that X-phases. IMO, the UI should check that all the tasks have a virtv2v_disks attribute and display "Calculating..." in the meantime. " Version-Release number of selected component (if applicable): CFME-5.11.1.1 How reproducible: 100%
@Fabien, > IMO, the UI should check that all the tasks have a virtv2v_disks attribute and display "Calculating..." in the meantime. That sounds good to me. Is it sufficient for the UI to check that every task has a defined virtv2v_disks attribute, or could that attribute be defined but incomplete? In the "Calculating..." case, is virtv2v_disks null/undefined, or an empty array? If possible I'd like to grab a sample database dump where I can reproduce that case.
Opened an issue for this in GitHub: https://github.com/ManageIQ/manageiq-v2v/issues/1078
Ilanit, do you have an appliance where Mike could reproduce this behavior and look at the DB?
Mike, I tried to reproduce this on CFME-5.11.1.2, and it did not reproduce. btw, this CFME version already contains the change done in bug 1767902 (GB -> GiB). For 20 VMs, of 16 GB disk each, I tracked the UI: Right from the start it displayed 320 GiB (20 x 16 GiB). Could it be that this was solved by some recent code change? "cfme-5.11.1.2-migration-screenshot" attached.
Created attachment 1650964 [details] cfme-5.11.1.2-migration-screenshot
Ilanit, I think it's possible that is a coincidence. If I understand Fabien's description of the problem correctly, the bug happens when the disk data takes longer to populate than the UI takes to switch views and load the data. So it could just be that when you tried to reproduce it, the data populated fast enough that you didn't see the bug. I still think it's worth writing the fix just to be sure, although that does make it tricky to verify... If it is fixed, it wouldn't have been fixed by the GB -> GiB change. Fabien, do you think it's possible this bug was fixed by some other recent code change?
I don't think it's been fixed by any change. It's probably because the preflight check happened fast enough that all disks were in the tasks before UI refresh.
I did several trials of 20 VMs, 16 GB disk, vddk migration trials. I found that only when I created a new migration plan, using a new name, it was possible to see the total size in phases. Cancel migration, and then retry, will always display the over all disks size (320 GiB in this case). also creating a migration plan, with the same name, as the previous one, also displayed the over all disks size (320 GiB). Attached "IMS-on-cfme-5.11.1.2-2020-01-13_11.07.02" video, where on time 6:48 this was displayed: Datastores size 0.00 B out of 80 GiB, on time 7:03 this was displayed: Datastores size 0.00 B out of 320 GiB The first total size displayed (80 GiB above), is not always the same, and is not necessarily half of the overall disks size. On another trial, I did yesterday, it was 96 GiB.
Created attachment 1651798 [details] IMS-on-cfme-5.11.1.2-2020-01-13_11.07.02.mkv
Perfect, thanks for figuring out those cases Ilanit. I'll go ahead and reproduce the issue in a few minutes to capture the data I need to test a fix. > also creating a migration plan, with the same name, as the previous one, also displayed the over all disks size (320 GiB). This is strange to me, I wonder if we're doing some kind of caching based on the plan name that we shouldn't be doing, and if so whether this is happening in the UI or the backend. Maybe we should make sure that we can create a new plan with the same name but with different VMs, and get the correct size for those new VMs instead of the old plan's data.
Ilanit, I apologize for not realizing this sooner: in the latest version on master (and soon on ivanchuk, when warm migration is backported) the Migrations In Progress view has changed from a card view to a list view, and it no longer even displays the total disk size of the migration at all. Vince decided that it was a misleading representation of the full progress of a migration (since customers were seeing 100% of the disk size migrated while playbooks and other processes were still completing, and wondering why they were not finished). Instead, the progress bar just shows the number of VMs completed, and you have to click through to the plan details view to see the math on the individual disks. I dumped the database right when the incorrect size was displaying on your appliance, and I'm looking at my local UI (master) with that data: disk sizes on the plan details view are accurate, and we no longer display an inaccurate total anywhere. With this in mind, do we need to fix this at all? I imagine we can't/won't release a hotfix for gaprindashvili, so the question is, is it worth fixing it on ivanchuk now even though the issue will disappear when the changes I describe above are released with IMS 1.3 in CloudForms 5.0.4? I'm sorry I didn't realize this was the case until I went to reproduce the issue locally.
Mike, Thanks for investing time on this! As this bug is not critical, and will disappear on IMS 1.3, I think there is no need to fix it on ivanchuk.
Thanks Ilanit! Not sure what status we need to put on this BZ then. I'll close the GitHub issue and JIRA ticket.
Ah, Ilanit I think Fabien wants me to fix this as a hammer-only PR (I meant hammer when I said gaprindashvili earlier).
Mike, Add the PR here then, and once it will move to ON_QA we will verify it on next CFME-5.10 release to QE.