Created attachment 473335 [details]
vdsm-spm.logs

Description of problem:

issue: get the following error on both spm and hsm log. 
please note that it might point meta-data corruption. 
my storage configuration is a bit special, as each storage domain is combined from 8 pvs, which exposed by 2 targets, and connected to 2 different IP addresses (8 * 2 * 2). 
I suspect that someone along the way corrupt meta data:

Thread-9636::DEBUG::2011-01-13 14:02:25,050::resource::176::irs::resource Storage/e7ea530d-fbb0-4d5c-9f9b-5e610b4a2711 after decref ref 0
Thread-9636::ERROR::2011-01-13 14:02:25,050::dispatcher::104::irs::{'status': {'message': "Meta Data parameter invalid: ('Version or spm id invalid',)", 'code': 755}, 'args': [('Version or spm id invalid',)]}
Thread-9636::ERROR::2011-01-13 14:02:25,051::dispatcher::106::irs::Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 97, in run
    result = ctask.prepare(self.func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 1283, in prepare
    raise self.error
MetaDataParamError: Meta Data parameter invalid: ('Version or spm id invalid',)

Thread-714::WARNING::2011-01-13 14:02:25,056::misc::121::irs::FAILED: <err> = '  Volume group "17d52d67-3188-4c5e-99ac-b34ecb5687f7" not found\n'; <rc> = 5
Thread-714::DEBUG::2011-01-13 14:02:25,056::misc::96::irs::'/bin/dd iflag=direct skip=0 bs=2048 if=/dev/17d52d67-3188-4c5e-99ac-b34ecb5687f7/metadata count=1' (cwd None)
Thread-714::DEBUG::2011-01-13 14:02:25,089::misc::119::irs::SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000931 seconds, 2.2 MB/s\n'; <rc> = 0
Thread-714::DEBUG::2011-01-13 14:02:25,090::misc::279::irs::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000931 seconds, 2.2 MB/s'], size: 2048
Thread-714::DEBUG::2011-01-13 14:02:25,090::blockSD::87::irs::(BlockSDMetadata._get) metadata=[]
Thread-714::DEBUG::2011-01-13 14:02:25,090::metadata::95::irs::metadata.__validate() Metadata has no embedded checksum - trust it as it is
Thread-714::WARNING::2011-01-13 14:02:25,091::sd::212::irs::Meta Data self-validation failed: ('No VERSION key in metadata',)

Scenario: 

1) 2 hosts connected to iscsi storage domain 
2) storage pool configuration: 
   - 3 domains 
   - each domain is consisted with 2 targets 
   - both targets exposed same phisical devices (2 path) 
   - each target is connected twice, to 2 different ip address.
3) happened during attach\activate storage domain, though i can't tell for sure. 

see attached logs. 


[root@nott-vds2 ~]# dd iflag=direct skip=0 bs=2048 if=/dev/17d52d67-3188-4c5e-99ac-b34ecb5687f7/metadata count=1
CLASS=Data
DESCRIPTION=hateyaA2
IOOPTIMEOUTSEC=10
LEASERETRIES=3
LEASETIMESEC=60
LOCKPOLICY=None
LOCKRENEWALINTERVALSEC=5
MASTER_VERSION=3
POOL_DESCRIPTION=rhel5.6-iscsi
POOL_SPM_ID=1
POOL_SPM_LVER=0
POOL_UUID=e7ea530d-fbb0-4d5c-9f9b-5e610b4a2711
PV0=pv:1REDHAT_SCALE9,uuid:dadrkl-Gemq-oqeT-bLdQ-SkVQ-JdAI-5fjjaN,pestart:0,pecount:95,mapoffset:0
PV1=pv:1REDHAT_SCALE8,uuid:7vhZOc-XWeY-3Uvl-VM08-YGjH-kzx4-BEbefL,pestart:0,pecount:95,mapoffset:95
PV2=pv:1REDHAT_SCALE15,uuid:HEyY1Q-xhef-f703-GFYJ-6922-u9ao-0OCPn4,pestart:0,pecount:95,mapoffset:190
PV3=pv:1REDHAT_SCALE14,uuid:jHeQu5-jlxD-OnCA-zsUm-8ctT-kWeV-Oc7nsV,pestart:0,pecount:95,mapoffset:285
PV4=pv:1REDHAT_SCALE13,uuid:sNfF6S-EEiZ-ruUQ-nA1T-wwAH-EDCj-NRqPGQ,pestart:0,pecount:95,mapoffset:380
PV5=pv:1REDHAT_SCALE12,uuid:geVjkD-vU5m-v16i-hFPn-yPPW-yrhk-rdf7TX,pestart:0,pecount:95,mapoffset:475
PV6=pv:1REDHAT_SCALE11,uuid:6rYqMb-wevF-LDzy-j2pa-SRkr-3iMA-epH4Dp,pestart:0,pecount:95,mapoffset:570
PV7=pv:1REDHAT_SCALE10,uuid:2cJ2XD-lJcE-bkEp-P5qr-ouWz-ew1w-qCCIMC,pestart:0,pecount:95,mapoffset:665
ROLE=Master
SDUUID=17d52d67-3188-4c5e-99ac-b34ecb5687f7
TYPE=ISCSI
VERSION=0
VGUUID=UjcBok-DVuX-DAdN-KPb1-iXUJ-fRGO-nNVydI
POOL_DOMAINS=17d52d67-3188-4c5e-99ac-b34ecb5687f7:Active,f4e49402-1ede-4f66-9cbc-5ac255d243ff:Active,d6b96c6b-168d-48e5-918f-6e5a7035ce0b:Active
_SHA_CKSUM=246ba16efdb002d36751699295de0baafbc75dc4
1+0 records in
1+0 records out
2048 bytes (2.0 kB) copied, 0.001076 seconds, 1.9 MB/s