Red Hat Bugzilla – Bug 1528367
stderr is not logged in the vdsm if the lvremove fails for some reason
Last modified: 2018-05-15 13:55:30 EDT
Description of problem: Currently, the exception message is not at all useful if the lvremove command fails for some reason. CannotRemoveLogicalVolume will only just pass the vg and lv name. stderr of the lvremove command is not passed in the exception message. CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\', "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],) So we may have to also pass "err" with se.CannotRemoveLogicalVolume just like we are doing in CannotCreateLogicalVolume so that the error message will be useful to understand the command's failure. Version-Release number of selected component (if applicable): vdsm-4.19.42-1.el7ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. Hold the lv using some command so that the lvremove will fail as the device is in use. Actual results: CannotRemoveLogicalVolume is not providing stderr of lvremove command. Expected results: Provide stderr of lvremove command to CannotRemoveLogicalVolume. Additional info:
Eyal - this seems like a pretty straight forward patch, and the BZ has a customer ticket. Can we have it backported to 4.2.z please?
sure
Moving back to POST until the backport will be merged.
WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{'rhevm-4.2-ga': '?'}', ] For more info please contact: rhv-devops@redhat.comINFO: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [Found non-acked flags: '{'rhevm-4.2-ga': '?'}', ] For more info please contact: rhv-devops@redhat.com
Eyal, Can you write the steps to reproduce as i'm trying to verify it and need some info. Thank you.
(In reply to Yosi Ben Shimon from comment #6) > Eyal, > Can you write the steps to reproduce as i'm trying to verify it and need > some info. > > Thank you. 1. Create a VM with ISCSI-based disk 2. Move the VM disk to a different ISCSI domain 3. while the disk migrating, try to remove the disk. Expected result should be that the operation to remove the disk failed and the VDSM log should present verbose error and not - CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\', "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],) BTW - use needinfo checkbox next time so I will get a message and response more quickly, thanks!
(In reply to Eyal Shenitzky from comment #7) > (In reply to Yosi Ben Shimon from comment #6) > > Eyal, > > Can you write the steps to reproduce as i'm trying to verify it and need > > some info. > > > > Thank you. > > 1. Create a VM with ISCSI-based disk > 2. Move the VM disk to a different ISCSI domain > 3. while the disk migrating, try to remove the disk. If we'll do it under the legs of the vdsm, we won't get the vdsm error. And we can't try to remove the disk via engine because it's being migrated and the operation is not permitted. > Expected result should be that the operation to remove the disk failed and > the VDSM log should present verbose error and not - > CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove > Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\', > "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],) > > BTW - use needinfo checkbox next time so I will get a message and response > more quickly, thanks!
(In reply to Yosi Ben Shimon from comment #8) > (In reply to Eyal Shenitzky from comment #7) > > (In reply to Yosi Ben Shimon from comment #6) > > > Eyal, > > > Can you write the steps to reproduce as i'm trying to verify it and need > > > some info. > > > > > > Thank you. > > > > 1. Create a VM with ISCSI-based disk > > 2. Move the VM disk to a different ISCSI domain > > 3. while the disk migrating, try to remove the disk. > If we'll do it under the legs of the vdsm, we won't get the vdsm error. > And we can't try to remove the disk via engine because it's being migrated > and the operation is not permitted. > > > Expected result should be that the operation to remove the disk failed and > > the VDSM log should present verbose error and not - > > CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove > > Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\', > > "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],) > > > > BTW - use needinfo checkbox next time so I will get a message and response > > more quickly, thanks! Did you tried via REST?
(In reply to Eyal Shenitzky from comment #9) > (In reply to Yosi Ben Shimon from comment #8) > > (In reply to Eyal Shenitzky from comment #7) > > > (In reply to Yosi Ben Shimon from comment #6) > > > > Eyal, > > > > Can you write the steps to reproduce as i'm trying to verify it and need > > > > some info. > > > > > > > > Thank you. > > > > > > 1. Create a VM with ISCSI-based disk > > > 2. Move the VM disk to a different ISCSI domain > > > 3. while the disk migrating, try to remove the disk. > > If we'll do it under the legs of the vdsm, we won't get the vdsm error. > > And we can't try to remove the disk via engine because it's being migrated > > and the operation is not permitted. > > > > > Expected result should be that the operation to remove the disk failed and > > > the VDSM log should present verbose error and not - > > > CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove > > > Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\', > > > "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],) > > > > > > BTW - use needinfo checkbox next time so I will get a message and response > > > more quickly, thanks! > > Did you tried via REST? Yes. I tried and got this error in the engine log: 2018-04-17 15:04:27,216+03 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-4) [] Operation Failed: [Cannot remove Virtual Disk. Related operation is currently in progress. Please try again later.] and this is the response in the RestAPI with status 409 (conflict): <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <fault> <detail>[Cannot remove Virtual Disk. Related operation is currently in progress. Please try again later.]</detail> <reason>Operation Failed</reason> </fault> But it doesn't reach VDSM.
OK, Let's meet f2f, I will customize the engine and remove the limitation to remove the disk while the VM is running or during migration. Then we will try to remove the disk and see the log.
After meeting Eyal f2f, we verified this bug using the next steps: 1. Build engine to allow VM disk remove even if the VM is up 2. Create a VM with block-based disk 3. Run the VM 4. Remove the VM disk via REST-API Returned error from VDSM: 2018-04-22 11:49:21,920+03 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-22) [] BaseAsyncTask::logEndTaskFailure: Task 'a6453706-ad2f-4f60-ae9c-e674ae2a9707' (Parent Command 'RemoveDisk', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'f81b16cc-7162-4055-9645-2ba3fdd68aaf\', "[\'cd89d322-a1c4-4c54-80a8-8ab763806f91\']", [\' Logical volume f81b16cc-7162-4055-9645-2ba3fdd68aaf/cd89d322-a1c4-4c54-80a8-8ab763806f91 in use.\'])'],)', -- Exception: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'f81b16cc-7162-4055-9645-2ba3fdd68aaf\', "[\'cd89d322-a1c4-4c54-80a8-8ab763806f91\']", [\' Logical volume f81b16cc-7162-4055-9645-2ba3fdd68aaf/cd89d322-a1c4-4c54-80a8-8ab763806f91 in use.\'])'],)' Moving to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:1489