Created attachment 1407715 [details] active_vm_request.rb Please ask the customer to run the attached script from an appliance console using "bin/rails r <script_name>" The output should display Requests and specifically invalid ones that no longer have a relationship to the vm_template. The invalid requests are likely the ones causing the issue and will need to be deleted.
Hi Niladri, Thanks for the update. I'm glad to hear that provisioning is working properly. The reason for the behavior change was that quota calculations for active provisions was changed to query the MiqRequest table instead of the MiqQueue table to provide more accurate quota values. I'll modify the active provision code to ignore these invalid VM provision requests. Thanks, Tina
https://github.com/ManageIQ/manageiq/pull/17158
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/0fa6f8b40b354d209172abdad77ef5cdc4f29235 commit 0fa6f8b40b354d209172abdad77ef5cdc4f29235 Author: Tina Fitzgerald <tfitzger> AuthorDate: Wed Mar 14 16:57:34 2018 -0400 Commit: Tina Fitzgerald <tfitzger> CommitDate: Wed Mar 14 16:57:34 2018 -0400 Change active_provision query to exclude requests with invalid sources. Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1554989 app/models/mixins/miq_provision_quota_mixin.rb | 4 + spec/models/miq_provision_request_spec.rb | 6 + 2 files changed, 10 insertions(+)
Hi Dmitry, You'll need to create an invalid VM provisioning request to reproduce this issue. The starting point is an appliance with a provider configured, that can successfully provision a VM using lifecycle provisioning. 1. Add a second provider to use for VM lifecycle provisioning. 2. Add a 2nd zone called "test_zone". (Don't add a second appliance for this zone) 3. Set the zone of the second provider to be "test_zone". 4. Provision a VM for the second provider, using VM lifecycle provisioning. (The provisioning request should remain in pending/active status and should not get processed because there is no appliance/workers for the "test_zone".) 5. Delete the template used in step 4.(Through the UI when you navigate to virtual machines, templates is on the left nav bar, select the template used in step 4 and select: "Remove from Inventory" 6. Provisioning a VM for the first provider, using VM lifecycle provisioning should produce the reported error. Let me know if you have any questions. Thanks, Tina
Thank you very much Tina, I've tested these steps on 5.9.2.2.20180410215257_6961422 and 5.8.3.4.20180221165627_8efa813. I'm not sure about "VM lifecycle provisioning". I provisioned vms using the UI ("Lifecycle" button and then "Provision VMs"). There were no errors. A vm for the first provider was provisioned successfully.
Hi Dmitry, Can I have access to your environments? Thanks, Tina
Hi Dmitry, There are a few additional steps required to reproduce this issue. 4a The Provision request from step 4 above needs to be approved. You can do a manual approval. 4b Using the Automate Explorer, Copy the /Infrastructure/VM/Provisioning/Placement/Default instance to a writable domain. 4c Modify the copied instance in step 4b, change the vmware value from: vmware_best_fit_least_utilized to: vmware_best_fit_with_scope (I added a tina domain with the instance change on 10.8.198.46, but couldn't test because the request needs to be approved BEFORE you remove the template. So, you'll have to create a new request for this) Let me know if you have any questions. Thanks, Tina
Hi Dmitry, I checked out the 10.8.198.185 appliance and it has the code fix. The reported issue is that the vmware_best_fit_with_scope method was throwing this error: ----] I, [2018-03-13T05:06:48.324012 #19065:c7513c] INFO -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> <AEMethod [/PitneyBowes/Infrastructure/VM/Provisioning/Placement/vmware_best_fit_with_scope]> Starting [----] I, [2018-03-13T05:06:48.951487 #19065:d0efbc] INFO -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> <AEMethod vmware_best_fit_with_scope> VM=<Template-CentOS-7.2-x64-Server>, Space Required=<45097156608>, group=<S [----] I, [2018-03-13T05:06:48.954712 #19065:d0efbc] INFO -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> <AEMethod vmware_best_fit_with_scope> storage_max_vms:<0> storage_max_pct_used:<100> [----] E, [2018-03-13T05:06:49.198116 #19065:d0efbc] ERROR -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> The following error occurred during instance method <check_quota> for AR object <#<ManageIQ::Providers::Vmware::I [----] E, [2018-03-13T05:06:49.201764 #19065:d0efbc] ERROR -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> MiqAeServiceModelBase.ar_method raised: <NoMethodError>: <undefined method `provisioned_storage' for nil:NilClass [----] E, [2018-03-13T05:06:49.201944 #19065:d0efbc] ERROR -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> /var/www/miq/vmdb/app/models/mixins/miq_provision_quota_mixin.rb:378:in `storage' [----] E, [2018-03-13T05:06:49.204481 #19065:6749c70] ERROR -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> <AEMethod vmware_best_fit_with_scope> The following error occurred during method evaluation: [----] E, [2018-03-13T05:06:49.205527 #19065:6749c70] ERROR -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> <AEMethod vmware_best_fit_with_scope> NoMethodError: undefined method `provisioned_storage' for nil:NilClass [----] E, [2018-03-13T05:06:49.206780 #19065:6749c70] ERROR -- : Q-task_id([miq_provision_6000000002384]) <AutomationEngine> <AEMethod vmware_best_fit_with_scope> (druby://127.0.0.1:34153) /var/www/miq/vmdb/app/models/mixins/miq_provis This log snippet from your test appliance shows the method completed successfully: [----] I, [2018-04-19T03:05:48.766596 #11992:bf5108] INFO -- : Q-task_id([miq_provision_1]) Updated namespace [Infrastructure/VM/Provisioning/Placement/vmware_best_fit_with_scope RedHat/Infrastructure/VM/Provisioning] [----] I, [2018-04-19T03:05:48.815964 #11992:bf5108] INFO -- : Q-task_id([miq_provision_1]) Invoking [inline] method [/RedHat/Infrastructure/VM/Provisioning/Placement/vmware_best_fit_with_scope] with inputs [{}] [----] I, [2018-04-19T03:05:48.817325 #11992:bf5108] INFO -- : Q-task_id([miq_provision_1]) <AEMethod [/RedHat/Infrastructure/VM/Provisioning/Placement/vmware_best_fit_with_scope]> Starting [----] I, [2018-04-19T03:05:49.315571 #11992:be4f10] INFO -- : <AEMethod vmware_best_fit_with_scope> VM=<damnSmallLinux>, Space Required=<201326592>, group=<EvmGroup-super_administrator> [----] I, [2018-04-19T03:05:49.317612 #11992:be4f10] INFO -- : <AEMethod vmware_best_fit_with_scope> storage_max_vms:<0> storage_max_pct_used:<100> [----] I, [2018-04-19T03:05:52.821293 #11992:be4f10] INFO -- : <AEMethod vmware_best_fit_with_scope> Sorted host Order:<[:active_provioning_memory, :current_memory_headroom, :random]> Results:<[[[0, -9404, 107], "env-nvc60-h01.cfme2.lab.eng.rdu2.redhat.com"], [[0, -5402, 131], "env-nvc60-h03.cfme2.lab.eng.rdu2.redhat.com"], [[0, -4311, 385], "env-nvc60-h02.cfme2.lab.eng.rdu2.redhat.com"]]> [----] I, [2018-04-19T03:05:52.843933 #11992:be4f10] INFO -- : <AEMethod vmware_best_fit_with_scope> Selected Host:<nil> [----] I, [2018-04-19T03:05:52.844617 #11992:be4f10] INFO -- : <AEMethod vmware_best_fit_with_scope> Selected Datastore:<nil> [----] I, [2018-04-19T03:05:52.846659 #11992:be4f10] INFO -- : <AEMethod vmware_best_fit_with_scope> vm=<damnSmallLinux> host=<> storage=<> [----] I, [2018-04-19T03:05:52.871070 #11992:bf5108] INFO -- : Q-task_id([miq_provision_1]) <AEMethod [/RedHat/Infrastructure/VM/Provisioning/Placement/vmware_best_fit_with_scope]> There is additional setup required(tagging datastores) to use the best_fit_with_scope method, so the error you saw was expected. I'm satisfied that the fix resolved the issue. You should be able to reproduce the issue on an appliance that does not have the code fix. Let me know if you have any questions. Thanks, Tina
Verified on 5.10.0.1.20180619163011_900fdc4. vmware_best_fit_with_scope method works as expected.