Bug 1758216
| Summary: | [scale] Engine fails to create multiple pools of vms | ||
|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Ilan Zuckerman <izuckerm> |
| Component: | BLL.Virt | Assignee: | Arik <ahadas> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Tzahi Ashkenazi <tashkena> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.3.6 | CC: | ahadas, bugs, mperina |
| Target Milestone: | ovirt-4.4.4 | Flags: | pm-rhel:
ovirt-4.4+
|
| Target Release: | 4.4.4.1 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | ovirt-engine-4.4.4.1 | Doc Type: | Bug Fix |
| Doc Text: |
Previously, when creating a virtual machine pool with more than one virtual machine that are set with vm-init (cloud-init/sysprep/ignition), the creation of some of the virtual machines might have failed due to database violation.
Now, the creation of the requested amount of virtual machines succeeds.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-12-21 12:36:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Doesn't look like the issue is in the API layer. The API class is mentioned in the stack-trace because it reports the error, but the error doesn't originate from it, it comes from the Engine. In general API operations are discrete, so if one works, a sequence should work, and if a sequence doesn't, it's about how the Engine handles it. Tested and verified on Red-03 with the following release version:
rhv-release-4.4.4-2-001.noarch
ovirt-engine-4.4.4.1-0.1.el8ev.noarch
environment :
Hosted Engine using disk from FC storage:
4 DC & 4 Clusters
254 Hosts of which 4 real servers, 250 nested hosts
3 Storage Domains
550 VMS all are up
ansible playbook :
- name: Create VM pool from templte
ovirt.ovirt.ovirt_vmpool:
auth: "{{ ovirt_auth }}"
cluster: "{{ cluster_name }}"
name: 'BZ1758216_pool_{{item}}'
template: basic_population_rhel8_L0_Group_0
vm_count: "{{ vm_count }}"
prestarted: 0
vm_per_user: 1
wait: False
with_sequence: start=1 end=4
2020-12-02 10:39:19,633Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-57) [30bfbc2d] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_1 (containing 100 VMs) was created by admin@internal-authz.
2020-12-02 10:40:19,046Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-57) [4fb1b2b1] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_2 (containing 100 VMs) was created by admin@internal-authz.
2020-12-02 10:41:18,531Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-57) [50d7a2c3] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_3 (containing 100 VMs) was created by admin@internal-authz.
2020-12-02 10:42:14,821Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-57) [707921de] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_4
(containing 100 VMs) was created by admin@internal-authz.
2020-12-02 10:54:00,664Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [87c43d81-c84c-467b-bdc4-25f4460f5bb7] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_1 (containing 100 VMs) was created by admin@internal-authz.
2020-12-02 10:54:16,556Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [1789bdf3-49bd-472d-a00b-9b5c5a86722c] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_2 (containing 100 VMs) was created by admin@internal-authz.
2020-12-02 10:54:21,715Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [fb0deee1-effb-4c54-9d13-46d4ffa356d3] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_4 (containing 100 VMs) was created by admin@internal-authz.
2020-12-02 10:54:26,745Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-12) [0417063e-66b1-4b81-98f7-80b470f28b84] EVENT_ID: USER_ADD_VM_POOL_WITH_VMS(302), VM Pool BZ1758216_pool_3 (containing 100 VMs) was created by admin@internal-authz.
all the pools creation completed successfully without any errors or warning messages on the engine log
all the operations of the pools creations reported by INFO level on the engine log as expected
This bugzilla is included in oVirt 4.4.4 release, published on December 21st 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.4 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |
Description of problem: When creating few vms pools with Ansible in a serial way without waiting for first pool to complete and pre-started vms = 0 , the engine throws an error: 2019-10-03 12:21:18,239Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-51) [14590b32] EVENT_ID: USER_ADD_VM _POOL_WITH_VMS_FAILED(303), Failed to create VM Pool linux_hacked_pool_03_10_2 (User: admin@internal-authz). 2019-10-03 12:21:18,249Z ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-51) [] Operation Failed: [] Looking at engine ui, it looks like the creation of the pools is continuing although Ansible is failing with "HTTP response code is 400" which obviosly originates from the engine ERROR. Version-Release number of selected component (if applicable): rhv-release-4.3.6-7-001.noarch ovirt-engine-4.3.6.5-0.1.el7.noarch How reproducible: We hit this once in a while. But after restarting the engine, the issue gets resolved. Topology: Hosted Engine using disk from FC storage: 9 DC and 10 Clusters 532 Hosts of which 33 real servers, 499 nested hosts 12 Storage Domains composed of: 9 FC Domains, 1 NFS, and 2 glance instances 5000 VMS all are up Hosted Engine running on dedicated host: 16vcpu with 32 GB ram Steps to Reproduce: 1. Execute the following ansible task: - ovirt_vmpool: auth: "{{ ovirt_auth }}" cluster: "{{ cluster_name }}" name: 'linux_hacked_pool_03_10_{{item}}' template: linux_hacked_agent_18_08 vm_count: 100 prestarted: 0 vm_per_user: 1 wait: False with_sequence: start=1 end=4 Actual results: Engine is throwing an error in its log, and Causing Ansible to fail Expected results: Shouldnt throw Error or case failure. Additional info: Attaching log collector logs including the relevant SPM host: