Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1528367

Summary:	stderr is not logged in the vdsm if the lvremove fails for some reason
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	nijin ashok <nashok>
Component:	vdsm	Assignee:	Eyal Shenitzky <eshenitz>
Status:	CLOSED ERRATA	QA Contact:	Yosi Ben Shimon <ybenshim>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.1.8	CC:	ebenahar, eshenitz, lsurette, lveyde, ratamir, srevivo, tnisan, ybenshim, ycui, ykaul, ylavi
Target Milestone:	ovirt-4.2.2	Flags:	lsvaty: testing_plan_complete-
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	vdsm v4.20.19	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-05-15 17:54:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Storage	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description nijin ashok 2017-12-21 16:01:47 UTC

Description of problem:

Currently, the exception message is not at all useful if the lvremove command fails for some reason. CannotRemoveLogicalVolume will only just pass the vg and lv name. stderr of the lvremove command is not passed in the exception message.

CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\', "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],)

So we may have to also pass "err" with se.CannotRemoveLogicalVolume just like we are doing in CannotCreateLogicalVolume so that the error message will be useful to understand the command's failure.


Version-Release number of selected component (if applicable):

vdsm-4.19.42-1.el7ev.x86_64

How reproducible:

100%

Steps to Reproduce:

1. Hold the lv using some command so that the lvremove will fail as the device is in use. 
  
Actual results:

CannotRemoveLogicalVolume is not providing stderr of lvremove command.

Expected results:

Provide stderr of lvremove command to CannotRemoveLogicalVolume.

Additional info:

Comment 2 Allon Mureinik 2018-02-15 16:21:37 UTC

Eyal - this seems like a pretty straight forward patch, and the BZ has a customer ticket. Can we have it backported to 4.2.z please?

Comment 3 Eyal Shenitzky 2018-02-18 11:28:15 UTC

sure

Comment 4 Eyal Shenitzky 2018-02-18 11:29:11 UTC

Moving back to POST until the backport will be merged.

Comment 5 RHV bug bot 2018-03-16 15:03:42 UTC

WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.2-ga': '?'}', ]

For more info please contact: rhv-devops

Comment 6 Yosi Ben Shimon 2018-03-18 13:02:52 UTC

Eyal,
Can you write the steps to reproduce as i'm trying to verify it and need some info.

Thank you.

Comment 7 Eyal Shenitzky 2018-03-18 14:18:11 UTC

(In reply to Yosi Ben Shimon from comment #6)
> Eyal,
> Can you write the steps to reproduce as i'm trying to verify it and need
> some info.
> 
> Thank you.

1. Create a VM with ISCSI-based disk
2. Move the VM disk to a different ISCSI domain
3. while the disk migrating, try to remove the disk.

Expected result should be that the operation to remove the disk failed and the VDSM log should present verbose error and not - CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\', "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],)

BTW - use needinfo checkbox next time so I will get a message and response more quickly, thanks!

Comment 8 Yosi Ben Shimon 2018-04-12 14:41:25 UTC

(In reply to Eyal Shenitzky from comment #7)
> (In reply to Yosi Ben Shimon from comment #6)
> > Eyal,
> > Can you write the steps to reproduce as i'm trying to verify it and need
> > some info.
> > 
> > Thank you.
> 
> 1. Create a VM with ISCSI-based disk
> 2. Move the VM disk to a different ISCSI domain
> 3. while the disk migrating, try to remove the disk.
If we'll do it under the legs of the vdsm, we won't get the vdsm error.
And we can't try to remove the disk via engine because it's being migrated and the operation is not permitted.

> Expected result should be that the operation to remove the disk failed and
> the VDSM log should present verbose error and not -
> CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove
> Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\',
> "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],)
> 
> BTW - use needinfo checkbox next time so I will get a message and response
> more quickly, thanks!

Comment 9 Eyal Shenitzky 2018-04-15 13:31:27 UTC

(In reply to Yosi Ben Shimon from comment #8)
> (In reply to Eyal Shenitzky from comment #7)
> > (In reply to Yosi Ben Shimon from comment #6)
> > > Eyal,
> > > Can you write the steps to reproduce as i'm trying to verify it and need
> > > some info.
> > > 
> > > Thank you.
> > 
> > 1. Create a VM with ISCSI-based disk
> > 2. Move the VM disk to a different ISCSI domain
> > 3. while the disk migrating, try to remove the disk.
> If we'll do it under the legs of the vdsm, we won't get the vdsm error.
> And we can't try to remove the disk via engine because it's being migrated
> and the operation is not permitted.
> 
> > Expected result should be that the operation to remove the disk failed and
> > the VDSM log should present verbose error and not -
> > CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove
> > Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\',
> > "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],)
> > 
> > BTW - use needinfo checkbox next time so I will get a message and response
> > more quickly, thanks!

Did you tried via REST?

Comment 10 Yosi Ben Shimon 2018-04-17 12:12:51 UTC

(In reply to Eyal Shenitzky from comment #9)
> (In reply to Yosi Ben Shimon from comment #8)
> > (In reply to Eyal Shenitzky from comment #7)
> > > (In reply to Yosi Ben Shimon from comment #6)
> > > > Eyal,
> > > > Can you write the steps to reproduce as i'm trying to verify it and need
> > > > some info.
> > > > 
> > > > Thank you.
> > > 
> > > 1. Create a VM with ISCSI-based disk
> > > 2. Move the VM disk to a different ISCSI domain
> > > 3. while the disk migrating, try to remove the disk.
> > If we'll do it under the legs of the vdsm, we won't get the vdsm error.
> > And we can't try to remove the disk via engine because it's being migrated
> > and the operation is not permitted.
> > 
> > > Expected result should be that the operation to remove the disk failed and
> > > the VDSM log should present verbose error and not -
> > > CannotRemoveLogicalVolume: Cannot remove Logical Volume: (['Cannot remove
> > > Logical Volume: (u\'72650763-cfc9-4e47-b5c5-551baaea45f8\',
> > > "[\'a33ede94-69b2-4e24-9a2d-bc5864f1cdeb\']")'],)
> > > 
> > > BTW - use needinfo checkbox next time so I will get a message and response
> > > more quickly, thanks!
> 
> Did you tried via REST?

Yes. I tried and got this error in the engine log:

2018-04-17 15:04:27,216+03 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-4) [] Operation Failed: [Cannot remove Virtual Disk. Related operation is currently in progress. Please try again later.]

and this is the response in the RestAPI with status 409 (conflict):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<fault>
    <detail>[Cannot remove Virtual Disk. Related operation is currently in progress. Please try again later.]</detail>
    <reason>Operation Failed</reason>
</fault>

But it doesn't reach VDSM.

Comment 11 Eyal Shenitzky 2018-04-18 07:06:34 UTC

OK,
Let's meet f2f, I will customize the engine and remove the limitation to remove the disk while the VM is running or during migration. 
Then we will try to remove the disk and see the log.

Comment 12 Yosi Ben Shimon 2018-04-22 08:59:35 UTC

After meeting Eyal f2f, we verified this bug using the next steps:

1. Build engine to allow VM disk remove even if the VM is up
2. Create a VM with block-based disk
3. Run the VM
4. Remove the VM disk via REST-API

Returned error from VDSM:

2018-04-22 11:49:21,920+03 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-22) [] BaseAsyncTask::logEndTaskFailure: Task 'a6453706-ad2f-4f60-ae9c-e674ae2a9707' (Parent Command 'RemoveDisk', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure:
-- Result: 'cleanSuccess'
-- Message: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'f81b16cc-7162-4055-9645-2ba3fdd68aaf\', "[\'cd89d322-a1c4-4c54-80a8-8ab763806f91\']", [\'  Logical volume f81b16cc-7162-4055-9645-2ba3fdd68aaf/cd89d322-a1c4-4c54-80a8-8ab763806f91 in use.\'])'],)',
-- Exception: 'VDSGenericException: VDSErrorException: Failed in vdscommand to HSMGetAllTasksStatusesVDS, error = Cannot remove Logical Volume: (['Cannot remove Logical Volume: (u\'f81b16cc-7162-4055-9645-2ba3fdd68aaf\', "[\'cd89d322-a1c4-4c54-80a8-8ab763806f91\']", [\'  Logical volume f81b16cc-7162-4055-9645-2ba3fdd68aaf/cd89d322-a1c4-4c54-80a8-8ab763806f91 in use.\'])'],)'


Moving to VERIFIED

Comment 17 errata-xmlrpc 2018-05-15 17:54:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1489

Comment 18 Franta Kust 2019-05-16 13:08:01 UTC

BZ<2>Jira Resync

Comment 19 Daniel Gur 2019-08-28 13:14:23 UTC

sync2jira

Comment 20 Daniel Gur 2019-08-28 13:19:26 UTC

sync2jira