Bug 1816951
| Summary: | [CNV&RHV] CNV VM migration failure is not handled correctly by the engine | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Pavol Brilla <pbrilla> |
| Component: | ovirt-engine | Assignee: | Arik <ahadas> |
| Status: | CLOSED ERRATA | QA Contact: | Pavol Brilla <pbrilla> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.4.0 | CC: | lleistne, masayag, michal.skrivanek, mperina |
| Target Milestone: | ovirt-4.4.2 | Keywords: | ZStream |
| Target Release: | 4.4.2 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | ovirt-engine-4.4.2.1 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-23 16:11:04 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The migration failed correctly but the error was not handled correctly in the engine. The error message in CNV was "cannot migrate VMI which does not use masquerade to connect to the pod network". With masquerade configured migration works fine. We need to make sure that the user sees why the migration failed. 2020-07-09 19:53:22,973+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [e1eea64f-cfef-4935-a64f-74ca9ce35f6a] EVENT_ID: VM_MIGRATION_NO_VDS_TO_MIGRATE_TO(166), No available host was found to migrate VM cirros-vm-notfromtemplate to.
2020-07-09 19:53:22,978+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [e1eea64f-cfef-4935-a64f-74ca9ce35f6a] EVENT_ID: VM_MIGRATION_FAILED(65), Migration failed (VM: cirros-vm-notfromtemplate, Source: talayan-pytest-8gw99-worker-wqnwd).
2020-07-09 19:53:22,984+02 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-7) [e1eea64f-cfef-4935-a64f-74ca9ce35f6a] Lock freed to object 'EngineLock:{exclusiveLocks='[08e73976-e63f-46f1-baae-0f84e9bae728=VM]', sharedLocks=''}'
2020-07-09 19:53:23,002+02 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-7) [] Operation Failed: [Fatal error during migration]
2020-07-09 19:53:44,290+02 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-7) [6b49e45e-2581-4aea-93c8-57e76a68d93f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[08e73976-e63f-46f1-baae-0f84e9bae728=VM]', sharedLocks=''}'
2020-07-09 19:53:44,391+02 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-7) [6b49e45e-2581-4aea-93c8-57e76a68d93f] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: 08e73976-e63f-46f1-baae-0f84e9bae728 Type: VMAction group MIGRATE_VM with role type USER
2020-07-09 19:53:45,053+02 ERROR [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-7) [6b49e45e-2581-4aea-93c8-57e76a68d93f] Command 'org.ovirt.engine.core.bll.MigrateVmToServerCommand' failed: EngineException: Internal error occurred: admission webhook "migration-create-validator.kubevirt.io" denied the request: Cannot migrate VMI, Reason: DisksNotLiveMigratable, Message: cannot migrate VMI with non-shared PVCs (Failed with error migrateErr and code 12)
2020-07-09 19:53:45,091+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [6b49e45e-2581-4aea-93c8-57e76a68d93f] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: cirros-vm-notfromtemplate, Source: talayan-pytest-8gw99-worker-wqnwd, Destination: talayan-pytest-8gw99-worker-rgnr5).
2020-07-09 19:53:45,099+02 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-7) [6b49e45e-2581-4aea-93c8-57e76a68d93f] Lock freed to object 'EngineLock:{exclusiveLocks='[08e73976-e63f-46f1-baae-0f84e9bae728=VM]', sharedLocks=''}'
2020-07-09 19:53:45,101+02 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-7) [] Operation Failed: [Fatal error during migration]
VM has evictionStrategy: LiveMigrate
if I have to have some more setting to be able to live migrate please specify
(In reply to Pavol Brilla from comment #4) > if I have to have some more setting to be able to live migrate please specify There's an example for how to create a migratable VM in Kubevirt: https://github.com/kubevirt/kubevirt/blob/master/examples/vmi-migratable.yaml Note that this is a VMI and in order to have it shown in oVirt, we need to create a VM. A virtual machine that should be able to migrate:
---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
labels:
special: vm-migratable
name: vm-migratable
spec:
running: false
template:
metadata:
labels:
kubevirt.io/vm: vm-migratable
spec:
domain:
devices:
disks:
- disk:
bus: virtio
name: containerdisk
interfaces:
- masquerade: {}
name: default
machine:
type: ""
resources:
requests:
memory: 64M
networks:
- name: default
pod: {}
terminationGracePeriodSeconds: 0
volumes:
- containerDisk:
image: registry:5000/kubevirt/alpine-container-disk-demo:devel
name: containerdisk
Error of migration failure is visible in engine.log: 2020-08-26 19:31:04,565+02 ERROR [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-56) [6cb613fc-b87e-475b-9d4e-9e4786200c93] Command 'org.ovirt.engine.core.bll.MigrateVmCommand' failed: EngineException: Internal error occurred: admission webhook "migration-create-validator.kubevirt.io" denied the request: Cannot migrate VMI, Reason: InterfaceNotLiveMigratable, Message: cannot migrate VMI which does not use masquerade to connect to the pod network (Failed with error migrateErr and code 12) Point of this bug - better error message for failing migration of ocp virtualization vm is verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3807 |
Description of problem: CNV VM with evictionStrategy: LiveMigrate is unable to migrate to any other host Version-Release number of selected component (if applicable): 4.4.0-26 How reproducible: 100 Steps to Reproduce: 1. have cnv with vm ( with defined evictionStrategy: LiveMigrate ) conncted to rhv 2. try to migrate such VM 3. Actual results: Migration failed (VM: cirros-vm-notfromtemplate, Source: talayan-pytest-2g7mj-worker-rgnwj). Expected results: VM should succeded. Additional info: 2020-03-25 09:41:30,218+01 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-14) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public -authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2020-03-25 09:41:30,401+01 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [62390a6e] Lock freed to object 'EngineLock:{exclusiveLocks=' [61dfa229-72c2-48d8-90ae-ecfaf51de1e1=PROVIDER]', sharedLocks=''}' 2020-03-25 09:41:49,601+01 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-7) [efba2ef7-9248-41d0-a7c2-406670e4730b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[225d7f52-620c-4ab5-bf87-a99dd60ccae5=VM]', sharedLo cks=''}' 2020-03-25 09:41:49,716+01 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-7) [efba2ef7-9248-41d0-a7c2-406670e4730b] Running command: MigrateVmCommand internal: false. Entities affected : ID: 225d7f52-620c-4ab5-bf87-a99dd 60ccae5 Type: VMAction group MIGRATE_VM with role type USER 2020-03-25 09:41:50,474+01 ERROR [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-7) [efba2ef7-9248-41d0-a7c2-406670e4730b] Command 'org.ovirt.engine.core.bll.MigrateVmCommand' failed: EngineException: failed to interact with kub evirt migrate endpoint (Failed with error unexpected and code 16) 2020-03-25 09:41:50,510+01 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [efba2ef7-9248-41d0-a7c2-406670e4730b] EVENT_ID: VM_MIGRATION_NO_VDS_TO_MIGRATE_TO(166), No available host was found to migrate VM cirros-vm-notfromtemplate to. 2020-03-25 09:41:50,517+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [efba2ef7-9248-41d0-a7c2-406670e4730b] EVENT_ID: VM_MIGRATION_FAILED(65), Migration failed (VM: cirros-vm-notfromtemplate, Source: talayan-pytest-2g7mj-worker-rgnwj). 2020-03-25 09:41:50,522+01 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (default task-7) [efba2ef7-9248-41d0-a7c2-406670e4730b] Lock freed to object 'EngineLock:{exclusiveLocks='[225d7f52-620c-4ab5-bf87-a99dd60ccae5=VM]', sharedLocks=''}' 2020-03-25 09:41:50,632+01 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-7) [] Operation Failed: [Unexpected exception] 2020-03-25 09:45:10,414+01 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedScheduledExecutorService-engineThreadMonitoringThreadPool-Thread-1) [] Thread pool 'default' is using 0 threads out of 1, 5 threads waiting for tasks.