Created attachment 1871828 [details] Engine logs during Backup and during Restore Created attachment 1871828 [details] Engine logs during Backup and during Restore Created attachment 1871828 [details] Engine logs during Backup and during Restore Description of problem: VM does not have a disk after being restored from 'Veeam' backup server Version-Release number of selected component (if applicable): RHV: 4.5.0.1-605.90f87fe14688.14.el8ev Veeam: 1.0.1488 How reproducible: 100% Steps to Reproduce: 1. Deploy Veeam backup vendor over RHV 2. Create a VM based on the guest-image template 3. Backup the VM using Veeam (with default setting) 4. Restore the VM using Veeam (with default setting) 5. Try to run the VM Actual results: Got an error: "Cannot run VM without at least one bootable disk." Expected results: VM should start successfully Additional info: - See attached logs and screen capture of the Veeam error - The same scenario works well in 4.4.10
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Yury, it looks like the backup application doesn't try to upload images when working against RHV 4.4 SP1 during "Restore" and it works fine during "Disk Restore". The same backup application works fine in both cases against RHV 4.4.10. Any idea what can go wrong?
Hello there, Unfortunately, I don't have access to the logs. Could you solve that? Thanks.
The error presented on the Veeam side is now available but maybe you can point us to a relevant log that will be more informative? :)
Could you upload a support bundle from the proxy and also engine.log ?
Created attachment 1871969 [details] the support bundle pic
Roni, can you attach it please?
(In reply to Yury.Panchenko from comment #6) > Could you upload a support bundle from the proxy and also engine.log ? Yury, The engine logs are already attached I remove the private flag (maybe that's why you didn't see it) I add the support bandle as well now! As a result, the Engine logs and the Veeam logs do not belong to the same flow If that does not help, please let me know, I will take logs again
Are there any changes in disk attach API? 2022-04-11 13:36:59,463Z WARN [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-355) [3db16e07] Validation of action 'AttachDiskToVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__ATTACH_ACTION_TO,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_NOT_EXIST 2022-04-11 13:36:59,467Z WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-355) [3db16e07] EVENT_ID: VM_IMPORT_FROM_CONFIGURATION_ATTACH_DISKS_FAILED(175), VM roni_test_4 has been imported from the given configuration but the following disk(s) failed to attach: latest-rhel-guest-image-8.6-infra.
(In reply to Yury.Panchenko from comment #12) > Are there any changes in disk attach API? That's what I suspected as well but that didn't change I saw no indication of transfer image in the log - the disks were not uploaded and so this error is correct, the disk really doesn't exist
Note that the issue is also reproduced manually, and as a WA you can run Disk Restore to restore the missing disk
Yury, Any idea what is the 'No description' error we see in the Veeam log (please see attached 'veeam error in log')
We suspected this has something to do with the changes for hybrid backup but we disabled hybrid backup on 4.7 cluster level, took a backup of a VM and we were still unable to restore it. We really need assistance from Veeam with that - it seems the backup application is affected by some change in oVirt 4.5 but hard to tell which one
Hello there, could you upload the veeam proxy support bundle? https://helpcenter.veeam.com/docs/vbrhv/userguide/export_logs.html?ver=10
Created attachment 1873825 [details] support bundle
Thank you. The app creates the vm by import it from the configuration without disks. According to the logs the app couldn't create the vm, and it got Reason: 'Operation failed', Detail: 'No description' from the RHV [2022-04-11] [13:36:59.010] [84223] [Info ] [Backup] Sent info: Creating VM... [2022-04-11] [13:36:59.010] [84223] [Info ] [RHEVClient] Creating VM. ClusterId: ce5e4aef-d9f4-4142-9cc4-863a3736317b RegenerateIds: 0 Init Data (length):9633 [2022-04-11] [13:36:59.557] [84223] [Info ] [Backup] Sent event (3): Unable to create VM: Reason: 'Operation failed', Detail: 'No description' in the engine log I see the same error, but the vm was created 2022-04-11 13:36:59,359Z INFO [org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand] (default task-355) [3e2cab0f-80de-4561-8080-ee51021b6fe9] Running command: ImportVmFromConfigurationCommand internal: false. Entities affected : ID: ce5e4aef-d9f4-4142-9cc4-863a3736317b Type: ClusterAction group CREATE_VM with role type USER, ID: 00000000-0000-0000-0000-000000000000 Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN 2022-04-11 13:36:59,408Z INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-355) [3e2cab0f-80de-4561-8080-ee51021b6fe9] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='d2be4905-d8bd-4409-aa57-c536131c2fbe', status='Down', exitStatus='Normal'}), log id: 579aec2c 2022-04-11 13:36:59,416Z INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-355) [3e2cab0f-80de-4561-8080-ee51021b6fe9] FINISH, SetVmStatusVDSCommand, return: , log id: 579aec2c 2022-04-11 13:36:59,463Z WARN [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default task-355) [3db16e07] Validation of action 'AttachDiskToVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__ATTACH_ACTION_TO,VAR__TYPE__DISK,ACTION_TYPE_FAILED_DISK_NOT_EXIST 2022-04-11 13:36:59,467Z WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-355) [3db16e07] EVENT_ID: VM_IMPORT_FROM_CONFIGURATION_ATTACH_DISKS_FAILED(175), VM roni_test_4 has been imported from the given configuration but the following disk(s) failed to attach: latest-rhel-guest-image-8.6-infra. Could you increase the app log level by editing /opt/VeeamBackupAgent/appsettings.json and setting "LogLevel" : 5 then restart the proxy and repeat the test Thank you.
Thanks for your feedback Yury This would need to wait for Roni until he's back next week One question from my side though - we've noticed that when using the same Veeam application against RHV 4.4.10, it first uploads the disks and only then calling the import-from-configuration command and so the disk attachment succeeds, do you see in the log of the Veeam application any hint why the disk is not being uploaded first?
Created attachment 1874645 [details] support_bundle log_level_5 during restore for comment #19
Created attachment 1874646 [details] support_bundle log_level_5 during backup for comment #19
(In reply to Yury.Panchenko from comment #19) > Thank you. > The app creates the vm by import it from the configuration without disks. > According to the logs the app couldn't create the vm, and it got Reason: > 'Operation failed', Detail: 'No description' from the RHV > [2022-04-11] [13:36:59.010] [84223] [Info ] [Backup] Sent info: Creating > VM... > [2022-04-11] [13:36:59.010] [84223] [Info ] [RHEVClient] Creating VM. > ClusterId: ce5e4aef-d9f4-4142-9cc4-863a3736317b > RegenerateIds: 0 > Init Data (length):9633 > [2022-04-11] [13:36:59.557] [84223] [Info ] [Backup] Sent event (3): Unable > to create VM: Reason: 'Operation failed', Detail: 'No description' > > > in the engine log I see the same error, but the vm was created > 2022-04-11 13:36:59,359Z INFO > [org.ovirt.engine.core.bll.exportimport.ImportVmFromConfigurationCommand] > (default task-355) [3e2cab0f-80de-4561-8080-ee51021b6fe9] Running command: > ImportVmFromConfigurationCommand internal: false. Entities affected : ID: > ce5e4aef-d9f4-4142-9cc4-863a3736317b Type: ClusterAction group CREATE_VM > with role type USER, ID: 00000000-0000-0000-0000-000000000000 Type: > StorageAction group IMPORT_EXPORT_VM with role type ADMIN > 2022-04-11 13:36:59,408Z INFO > [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-355) > [3e2cab0f-80de-4561-8080-ee51021b6fe9] START, SetVmStatusVDSCommand( > SetVmStatusVDSCommandParameters:{vmId='d2be4905-d8bd-4409-aa57-c536131c2fbe', > status='Down', exitStatus='Normal'}), log id: 579aec2c > 2022-04-11 13:36:59,416Z INFO > [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-355) > [3e2cab0f-80de-4561-8080-ee51021b6fe9] FINISH, SetVmStatusVDSCommand, > return: , log id: 579aec2c > 2022-04-11 13:36:59,463Z WARN > [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default > task-355) [3db16e07] Validation of action 'AttachDiskToVm' failed for user > admin@internal-authz. Reasons: > VAR__ACTION__ATTACH_ACTION_TO,VAR__TYPE__DISK, > ACTION_TYPE_FAILED_DISK_NOT_EXIST > 2022-04-11 13:36:59,467Z WARN > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (default task-355) [3db16e07] EVENT_ID: > VM_IMPORT_FROM_CONFIGURATION_ATTACH_DISKS_FAILED(175), VM roni_test_4 has > been imported from the given configuration but the following disk(s) failed > to attach: latest-rhel-guest-image-8.6-infra. > > Could you increase the app log level by editing > /opt/VeeamBackupAgent/appsettings.json and setting "LogLevel" : 5 > then restart the proxy and repeat the test > Thank you. Hi Yury Sorry for the delay just return from a long vacation Please see attached: 1. "support_bundle log_level_5 during backup for comment #19" veeam level5 & engine logs taken during backup 2. "support_bundle log_level_5 during restore for comment #19" veeam level5 & engine logs taken during restore Thx Roni
Hello Roni and Arik, We've found the problem. Before 4.4 SP1 the app used to get code "201 Created" from import vm request. Now you're returning "202 Accepted", so the app don't expect this.
Thanks Yury, that's probably a consequence of a change I've made for 4.5: https://github.com/oVirt/ovirt-engine/commit/a309569546d7cf43e5f586f6cca7180ee8e8cd85 I can argue that you should also check for "202 Accepted" but as I understand it would affect existing clients, we'll see if we can return "Created" for calls like you do (for which the import operation should really complete before you get the results)
It's ok for us to check both codes in the next version, but I'd like to ask you to keep API compatibiliy when it possible
(In reply to Yury.Panchenko from comment #26) > It's ok for us to check both codes in the next version, but I'd like to ask > you to keep API compatibiliy when it possible Yeah, so please do check both as from the next version I posted a PR that can change back the response to "201 Created" and still address the issues we had with the previous mechanism, let's see how it goes
Verified on 4.5.0.6-0.7.el8ev vdsm-4.50.0.13-1.el8ev.x86_64