875909 – [Storage] Reusing LUN that was previously used by direct lun disk and storage domain causes to GetLunsByVgIdQuery fail with null pointer exception.

Bug 875909 - [Storage] Reusing LUN that was previously used by direct lun disk and storage domain causes to GetLunsByVgIdQuery fail with null pointer exception.

Summary: [Storage] Reusing LUN that was previously used by direct lun disk and storage...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Virtualization Manager
Classification:	Red Hat
Component:	ovirt-engine
Sub Component:
Version:	unspecified
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	3.2.0
Assignee:	Liron Aravot
QA Contact:	vvyazmin@redhat.com
Docs Contact:
URL:
Whiteboard:	storage
Depends On:
Blocks:	889793 915537
TreeView+	depends on / blocked

Reported:	2012-11-12 19:37 UTC by Leonid Natapov
Modified:	2016-02-10 16:32 UTC (History)
CC List:	11 users (show)
Fixed In Version:	sf3
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	889793 (view as bug list)
Environment:
Last Closed:	2013-06-11 08:17:51 UTC
oVirt Team:	Storage
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
engine log (260.87 KB, application/x-gzip) 2012-11-12 19:37 UTC, Leonid Natapov	no flags	Details
vdsm log (120.86 KB, application/x-gzip) 2012-11-12 19:38 UTC, Leonid Natapov	no flags	Details
## Logs vdsm, rhevm (1.75 MB, application/x-gzip) 2013-02-03 13:43 UTC, vvyazmin@redhat.com	no flags	Details
View All

Description Leonid Natapov 2012-11-12 19:37:37 UTC

Created attachment 643681 [details]
engine log

GetLunsByVgIdQuery fails with null pointer exception.

For some reason LUN IDs list that was sent by vdsm to engine didn't match the LUNs IDs in the database. One LUN ID was missing in the database. The scenario is the following:

2 ISCSI storage domains. One of the domains was extended with the LUN from another Storage Domain. 

engine and vdsm logs attached.

2012-11-12 19:10:42,678 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp--127.0.0.1-8702-7) START, GetDeviceLi
stVDSCommand(HostName = purple-vds2, HostId = e0fdb2b8-2c14-11e2-bc65-97e1ef3fa41e, storageType=ISCSI), log id: 87eb108
2012-11-12 19:11:01,545 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp--127.0.0.1-8702-9) START, GetDeviceLi
stVDSCommand(HostName = purple-vds2, HostId = e0fdb2b8-2c14-11e2-bc65-97e1ef3fa41e, storageType=null), log id: 6f3e4543
2012-11-12 19:11:03,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (ajp--127.0.0.1-8702-9) Failed in GetDeviceListVD
S method, for vds: purple-vds2; host: purple-vds2
2012-11-12 19:11:03,408 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp--127.0.0.1-8702-9) Command GetDeviceListVDS execution fa
iled. Exception: NullPointerException: 
2012-11-12 19:11:03,408 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp--127.0.0.1-8702-9) FINISH, GetDeviceL
istVDSCommand, log id: 6f3e4543
2012-11-12 19:11:03,409 ERROR [org.ovirt.engine.core.bll.GetLunsByVgIdQuery] (ajp--127.0.0.1-8702-9) Query GetLunsByVgIdQuery failed. Except
ion message is VdcBLLException: java.lang.NullPointerException
2012-11-12 19:11:15,395 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp--127.0.0.1-8702-7) FINISH, GetDeviceL
istVDSCommand, return: [org.ovirt.engine.core.common.businessentities.LUNs@c8a47713, org.ovirt.engine.core.common.businessentities.LUNs@b773
8574, org.ovirt.engine.core.common.businessentities.LUNs@1ad92417, org.ovirt.engine.core.common.businessentities.LUNs@818029a, org.ovirt.eng
ine.core.common.businessentities.LUNs@eb0e5748], log id: 87eb108

Comment 1 Leonid Natapov 2012-11-12 19:38:41 UTC

Created attachment 643682 [details]
vdsm log

Comment 2 Leonid Natapov 2012-11-13 10:56:11 UTC

How to reproduce:

1. create iscsi domain with lun X
2. create disk with lun X
3. remove domain
4. remove disk
5. create domain with lun X
6. activate domain
7. edit storage domain.

Also when destroying storage domain after steps 1-7 ,the domain not removed from the database.

Comment 3 Liron Aravot 2012-11-13 17:15:44 UTC

http://gerrit.ovirt.org/9228

Comment 4 Liron Aravot 2012-11-13 17:21:25 UTC

correct patch : 
http://gerrit.ovirt.org/#/c/9229/

Comment 6 Allon Mureinik 2012-12-12 10:31:38 UTC

patches are merged upstream.

Comment 7 Simon Grinberg 2012-12-23 15:34:02 UTC

This bug has been cloned to the RHEV-M .z stream:            https://bugzilla.redhat.com/show_bug.cgi?id=889793

Comment 8 vvyazmin@redhat.com 2013-01-16 14:01:38 UTC

No issues are found on iSCSI DC environment 

Verified on RHEVM 3.2 - SF03 environment 

RHEVM: rhevm-3.2.0-4.el6ev.noarch
VDSM: vdsm-4.10.2-3.0.el6ev.x86_64
LIBVIRT: libvirt-0.10.2-13.el6.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

Comment 9 Yaniv Kaul 2013-01-30 09:35:10 UTC

(In reply to comment #8)
> No issues are found on iSCSI DC environment 
> 
> Verified on RHEVM 3.2 - SF03 environment 
> 
> RHEVM: rhevm-3.2.0-4.el6ev.noarch
> VDSM: vdsm-4.10.2-3.0.el6ev.x86_64
> LIBVIRT: libvirt-0.10.2-13.el6.x86_64
> QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
> SANLOCK: sanlock-2.6-2.el6.x86_64

Please verify on FC as well.

Comment 10 vvyazmin@redhat.com 2013-02-03 13:40:50 UTC

Failed on step 7: edit storage domain in FC DC environment

/var/log/ovirt-engine/engine.log

2013-02-03 17:36:43,580 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8702-8) START, GetDeviceListVDSCommand(HostName = green-vdsa, HostId = 82e5c19b-d738-4b00-95ad-890bd6edf0ee, storageType=FCP), log id: 79170b1e
2013-02-03 17:36:43,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8702-7) START, GetDeviceListVDSCommand(HostName = green-vdsa, HostId = 82e5c19b-d738-4b00-95ad-890bd6edf0ee, storageType=null), log id: 18484190
2013-02-03 17:36:43,613 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (ajp-/127.0.0.1:8702-7) Failed in GetDeviceListVDS method, for vds: green-vdsa; host: green-vdsa.qa.lab.tlv.redhat.com
2013-02-03 17:36:43,613 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8702-7) Command GetDeviceListVDS execution failed. Exception: NullPointerException: 
2013-02-03 17:36:43,613 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, GetDeviceListVDSCommand, log id: 18484190
2013-02-03 17:36:43,614 ERROR [org.ovirt.engine.core.bll.GetLunsByVgIdQuery] (ajp-/127.0.0.1:8702-7) Query GetLunsByVgIdQuery failed. Exception message is VdcBLLException: java.lang.NullPointerException


RHEVM 3.2 - SF05 environment:

RHEVM: rhevm-3.2.0-6.el6ev.noarch
VDSM: vdsm-4.10.2-5.0.el6ev.x86_64
LIBVIRT: libvirt-0.10.2-17.el6.x86_64
QEMU & KVM: qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
SANLOCK: sanlock-2.6-2.el6.x86_64

Comment 11 vvyazmin@redhat.com 2013-02-03 13:43:08 UTC

Created attachment 692309 [details]
## Logs vdsm, rhevm

Comment 12 Liron Aravot 2013-02-03 13:51:38 UTC

this is the bug for FC , it's not merged to 3.2 yet...
https://bugzilla.redhat.com/show_bug.cgi?id=896042

Comment 13 vvyazmin@redhat.com 2013-02-03 13:57:55 UTC

iSCSI DC environment: No issues are found
FC DC environment: Failed, open separate bug BZ896042

Comment 14 Itamar Heim 2013-06-11 08:17:51 UTC

3.2 has been released

Comment 15 Itamar Heim 2013-06-11 08:22:38 UTC

3.2 has been released

Note You need to log in before you can comment on or make changes to this bug.