Bug 701671

Summary: Removing the iSCSI storage domain with host other that created it makes it not-attachable...
Product: Red Hat Enterprise Linux 6 Reporter: Jaroslav Henner <jhenner>
Component: vdsmAssignee: Saggi Mizrahi <smizrahi>
Status: CLOSED ERRATA QA Contact: Dafna Ron <dron>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3CC: abaron, bazulay, egerman, hateya, iheim, ilvovsky, jlibosva, oramraz, sgrinber, ykaul
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: vdsm-4.9-66.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 07:17:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm log of the host A
none
vdsm log of the host B
none
rhevm.log
none
Events of the time when I removed the inflanted VM. none

Description Jaroslav Henner 2011-05-03 14:15:04 UTC
Created attachment 496539 [details]
vdsm log of the host A

Description of problem:
When I remove the iSCSI storage domain with host A and remove it then with host B, it makes the storage domain impossible to attach with host A (

Version-Release number of selected component (if applicable):
Name        : vdsm
Arch        : x86_64
Version     : 4.9
Release     : 63.el6


How reproducible:
Allways

Steps to Reproduce:
1. Have two hosts
2. Add a storage domain with host A
3. Remove it with host B
4. Try adding it with host A again.
  
Actual results:
RHEVM Error:

Storage domain is not attacheable (Physical device initialisation failed. Check that device is empty... Error code 601

On the A host:
[root@srh-01 ~]# lvs
  LV      VG      Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lv_home vg_sf02 -wi-ao 215.67g                                      
  lv_root vg_sf02 -wi-ao  50.00g                                      
  lv_swap vg_sf02 -wi-ao   5.78g                                      
[root@srh-01 ~]# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  vg_sf02   1   3   0 wz--n- 271.45g    0 
[root@srh-01 ~]# dmsetup ls
d1b93bd4--5ff3--44fa--812b--4292d00708cd-metadata	(253, 4)
d1b93bd4--5ff3--44fa--812b--4292d00708cd-leases	(253, 5)
d1b93bd4--5ff3--44fa--812b--4292d00708cd-inbox	(253, 7)
vg_sf02-lv_home	(253, 2)
d1b93bd4--5ff3--44fa--812b--4292d00708cd-master	(253, 9)
d1b93bd4--5ff3--44fa--812b--4292d00708cd-ids	(253, 6)
d1b93bd4--5ff3--44fa--812b--4292d00708cd-outbox	(253, 8)
1jhenner1	(253, 3)
vg_sf02-lv_swap	(253, 1)
vg_sf02-lv_root	(253, 0)
[root@srh-01 ~]# hexdump -C /dev/sdb | head
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  e3 0f c5 00 20 00 00 00  4c 56 4d 32 20 30 30 31  |.... ...LVM2 001|
00000220  30 68 46 78 41 69 6b 56  59 57 41 46 33 41 67 4b  |0hFxAikVYWAF3AgK|
00000230  73 77 4d 59 74 74 50 46  70 4e 68 4b 77 39 75 49  |swMYttPFpNhKw9uI|
00000240  00 00 00 00 32 00 00 00  00 00 10 08 00 00 00 00  |....2...........|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 08 00 00 00 00  00 00 00 00 00 00 00 00  |................|

On storage server:
[root@str-01 ~]# hexdump -C /dev/nase_vg_str01/jhenner1 | head
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00000210  60 86 51 0a 20 00 00 00  4c 56 4d 32 20 30 30 31  |`.Q. ...LVM2 001|
00000220  63 42 47 65 6a 31 6b 70  70 37 41 6d 41 58 44 37  |cBGej1kpp7AmAXD7|
00000230  36 30 63 53 74 55 44 68  4a 42 37 30 46 68 6e 74  |60cStUDhJB70Fhnt|
00000240  00 00 00 00 32 00 00 00  00 00 10 08 00 00 00 00  |....2...........|
00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000270  00 f0 0f 08 00 00 00 00  00 00 00 00 00 00 00 00  |................|


Expected results:
Storage can be attached again no matter which host created and removed it.


Additional info:

Comment 1 Jaroslav Henner 2011-05-03 14:15:53 UTC
Created attachment 496540 [details]
vdsm log of the host B

Comment 2 Jaroslav Henner 2011-05-03 14:16:26 UTC
Created attachment 496541 [details]
rhevm.log

Comment 3 Jaroslav Henner 2011-05-03 14:22:40 UTC
I did dmsetup remove for all of the d1b93bd4--5ff3--44fa--812b--4292d00708cd-* and 1jhenner1 to recover. Than I could attach the domain with the host A again.

I also attached the domain (without the dmsetup remove procedure) with host B and removed with host A. The error appeared when attaching with host B when attaching with host A worked.

Comment 5 Jaroslav Henner 2011-05-03 14:32:44 UTC
Sorry guys. Everywhere I used the word "attach" I meant "add".

Comment 7 Dan Kenigsberg 2011-05-03 15:06:39 UTC
*** Bug 700372 has been marked as a duplicate of this bug. ***

Comment 8 Saggi Mizrahi 2011-05-04 10:50:00 UTC
http://gerrit.usersys.redhat.com/#change,355

Comment 9 Jaroslav Henner 2011-05-09 13:37:51 UTC
Today, I saw one of my machines had its thin-provisioned disk "actual size" over the limit. There is a bug 651335 about that. I didn't try to reproduce my situation, but I had same symptoms as are in this and in 651335, so these are probably related.

Comment 10 Ayal Baron 2011-05-10 20:13:22 UTC
Beyo(In reply to comment #9)
> Today, I saw one of my machines had its thin-provisioned disk "actual size"
> over the limit. 

Over what limit?

> There is a bug 651335 about that. I didn't try to reproduce my
> situation, but I had same symptoms as are in this and in 651335, so these are
> probably related.

Comment 11 Jaroslav Henner 2011-05-11 11:42:16 UTC
There(In reply to comment #10)
> Beyo(In reply to comment #9)
> > Today, I saw one of my machines had its thin-provisioned disk "actual size"
> > over the limit. 
> 
> Over what limit?

There is `Size` And `Actual size` in the machine Virtual Discs. I saw that the `Actual Size` was way above the `Size`. My storage got filled and I saw events about that.

Comment 15 Jaroslav Henner 2011-05-12 16:53:24 UTC
Created attachment 498583 [details]
Events of the time when I removed the inflanted VM.

Comment 18 Jaroslav Henner 2011-05-13 22:43:33 UTC
I reproduced the problem with inflating VM, we should continue discussing that on:
Bug 704666 - Thin provisioned VM on iSCSI gets inflated after migrating

Comment 20 Jakub Libosvar 2011-05-19 06:32:23 UTC
Verified in vdsm-4.9-67.el6.x86_64

Comment 21 errata-xmlrpc 2011-12-06 07:17:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1782.html