the following script reproduces the problem without engine:

in order to run it, just replace lun list in createVG command with your (make sure multipath recognizes it)

vdsClient 0 createVG  d55e3a64-0615-47ba-a500-4d81cacff52c 1avihai_lun1-700G41337842,1avihai_lun2-700G41337843,1avihai_newdc_lun1-800G41 && sleep 5 && vdsClient  0 createStorageDomain 3 d55e3a64-0615-47ba-a500-4d81cacff52c avihai-domain `vgs d55e3a64-0615-47ba-a500-4d81cacff52c -o+uuid  |awk '!/VG/{print $8}'` 1 2 && sleep 5 && vdsClient 0 getVGInfo `vgs d55e3a64-0615-47ba-a500-4d81cacff52c -o+uuid  |awk '!/VG/{print $8}'`
1JOAgI-25y9-BdbI-Zfm1-NaJM-UQYi-XOtxxk

Haim and Eduardo investigated this issue further and it seems like this is cause because of lvm cache issue described in bug 

https://bugzilla.redhat.com/show_bug.cgi?id=788096

However, we need a workaround from vdsm for this issue until lvm bug is fixed since currently users will not be able to create vg's from multiple targets on our hosts without a manual login to the luns.

I just installed ic155.1 on a RHEL 6.2 GA server. I added two hosts. I then added  ISCSI storage and as soon as the SPM was determined, the other host went non-operational. 

On both hosts are installed vdsm-4.9-112.6.el6_2.x86_64.

Also the message in RHEV-M was "Host <hostname> cannot access one of the Storage Domains attached to it, or the Data Center object. Setting the Host state to Non-Operational.

*** Bug 836720 has been marked as a duplicate of this bug. ***

In getVGInfo the output of pvs is parsed.
This output can be inconsistent due to lvm BZ#836663.

When a device list is passed as an argument for pvs, and not all the devices have metadata in use (as in vdsm), the device order changes the pvs output.
(See below.)

Possible solutions:
1) Require an lvm version including BZ# 836663 fix for use with vdsm.
2) Select the pv with the mda in use as the 1st parameter in vdsm pvs cmd.
2) Parse the output of vgs -o +pv_name instead of the pvs response.

[root@derez ~]# vgs -o pv_name 07622fad-381b-4b1d-b534-f9db364032c2
  PV                                           
  /dev/mapper/3600144f09dbd050000004e1994c60005
  /dev/mapper/3600144f09dbd050000004ddbe989001b
[root@derez ~]# 
[root@derez ~]# pvs -o pv_name,vg_name,pv_attr,pv_mda_used_count /dev/mapper/3600144f09dbd050000004e1994c60005 /dev/mapper/3600144f09dbd050000004ddbe989001b
  PV                                            VG                                   Attr #PMdaUse
  /dev/mapper/3600144f09dbd050000004ddbe989001b 07622fad-381b-4b1d-b534-f9db364032c2 a--         0
  /dev/mapper/3600144f09dbd050000004e1994c60005 07622fad-381b-4b1d-b534-f9db364032c2 a--         2
[root@derez ~]# 
[root@derez ~]# pvs -o pv_name,vg_name,pv_attr,pv_mda_used_count /dev/mapper/3600144f09dbd050000004ddbe989001b /dev/mapper/3600144f09dbd050000004e1994c60005
  PV                                            VG                                   Attr #PMdaUse
  /dev/mapper/3600144f09dbd050000004ddbe989001b                                      a--         0
  /dev/mapper/3600144f09dbd050000004e1994c60005 07622fad-381b-4b1d-b534-f9db364032c2 a--         2

Change-Id: I9fdf09a2637f096201f094918654e3f52663bc2d

Checked on si24.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1508.html

*** Bug 896505 has been marked as a duplicate of this bug. ***