Bug 1413867

Created attachment 1241658 [details]
lvm pvs output

Sergei, we like to understand why the multipath device name is not using the
standard format (which does not include ":").

Can you share the output of these commands:

lsblk
multipath -ll
cat /etc/multipath.conf

 Hi, Nil.
Here are requested outputs:

[root@kvm1 ~]# lsblk
NAME                                                         MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
fd0                                                            2:0    1    4K  0 disk  
sda                                                            8:0    0   80G  0 disk  
├─sda1                                                         8:1    0  500M  0 part  /boot
└─sda2                                                         8:2    0 79.5G  0 part  
  ├─centos-root                                              253:0    0   50G  0 lvm   /
  ├─centos-swap                                              253:1    0  3.9G  0 lvm   [SWAP]
  └─centos-home                                              253:2    0 25.6G  0 lvm   /home
sdb                                                            8:16   0   10G  0 disk  
└─SioFABRICVicinity_iqn.2015-03.com.iofabric:ovirt-master-00 253:3    0   10G  0 mpath 
  ├─70e64136--e537--4d27--ac3b--e067a127a1d7-metadata        253:4    0  512M  0 lvm   
  ├─70e64136--e537--4d27--ac3b--e067a127a1d7-outbox          253:5    0  128M  0 lvm   
  ├─70e64136--e537--4d27--ac3b--e067a127a1d7-leases          253:6    0    2G  0 lvm   
  ├─70e64136--e537--4d27--ac3b--e067a127a1d7-ids             253:7    0  128M  0 lvm   
  ├─70e64136--e537--4d27--ac3b--e067a127a1d7-inbox           253:8    0  128M  0 lvm   
  └─70e64136--e537--4d27--ac3b--e067a127a1d7-master          253:9    0    1G  0 lvm   /rhev/data-center/mnt/blockSD/70e64136-e537-4d27-ac3b-e067a127a1d7/master
sr0                                                           11:0    1  603M  0 rom   
14f504e46494c4552686e4962646a2d5951514e2d3747716b            253:11   0 11.7G  0 mpath 
├─a41bca58--e9ad--4c35--bc46--95dcd160415c-metadata          253:12   0  512M  0 lvm   
├─a41bca58--e9ad--4c35--bc46--95dcd160415c-outbox            253:13   0  128M  0 lvm   
├─a41bca58--e9ad--4c35--bc46--95dcd160415c-leases            253:14   0    2G  0 lvm   
├─a41bca58--e9ad--4c35--bc46--95dcd160415c-ids               253:15   0  128M  0 lvm   
├─a41bca58--e9ad--4c35--bc46--95dcd160415c-inbox             253:16   0  128M  0 lvm   
└─a41bca58--e9ad--4c35--bc46--95dcd160415c-master            253:17   0    1G  0 lvm   
[root@kvm1 ~]# 


[root@kvm1 ~]# multipath -ll
SioFABRICVicinity_iqn.2015-03.com.iofabric:ovirt-master-00 dm-3 ioFABRIC,Vicinity        
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 9:0:0:0 sdb 8:16 active ready running
14f504e46494c4552686e4962646a2d5951514e2d3747716b dm-11 
size=12G features='0' hwhandler='0' wp=rw


[root@kvm1 ~]# cat /etc/multipath.conf
# VDSM REVISION 1.3

defaults {
    polling_interval            5
    no_path_retry               fail
    user_friendly_names         no
    flush_on_last_del           yes
    fast_io_fail_tmo            5
    dev_loss_tmo                30
    max_fds                     4096
}

# Remove devices entries when overrides section is available.
devices {
    device {
        # These settings overrides built-in devices settings. It does not apply
        # to devices without built-in settings (these use the settings in the
        # "defaults" section), or to devices defined in the "devices" section.
        # Note: This is not available yet on Fedora 21. For more info see
        # https://bugzilla.redhat.com/1253799
        all_devs                yes
        no_path_retry           fail
    }
}

# Enable when this section is available on all supported platforms.
# Options defined here override device specific options embedded into
# multipathd.
#
# overrides {
#      no_path_retry           fail
# }

[root@kvm1 ~]#

(In reply to Sergei from comment #3)
Thanks Sergei, everything seems normal except the strange lun name. I guess this is
related to the storage server configuration.

I guess so. I will try to contact them directly. 
Thanks for help anyway.

 Sergei.

Moving to 4.1.1 since 4.1 RC is out and this bug is not marked as a blocker

Hi Nir ,

I see the customer had the following target that included ":" , so to verify this I tried to do the same .

Customer Target from bug :
SioFABRICVicinity_iqn.2015-03.com.iofabric:ovirt-master-00

I created a volume with this exact name (iofabric:ovirt-master-00) & mapped it to the VDSM host .

But the host still sees it via multipath -ll  as :
"3514f0c5a51600543 dm-40 XtremIO ,XtremApp"

And not as a its volume name -> iofabric:ovirt-master-00 .

Is there any way we can change the lun-id on the server ( we're using extreamIO) or in the host itself ?

Or can we change VDSM code to trick VDSM to see a target name with ":" in it ?

I tried also with targetcli server we have 10.35.88.157  and it does not look good.

I created a target name "target iqn.2015-03.com.iofabric:ovirt-master-00" and added a lun to it with 11G in size .

when I tried to add a new iscsi storage domain it failed .

event:
Feb 12, 2017 1:17:43 PM  Failed to add Storage Domain test22. (User: admin@internal-authz)

Many errors seen in vdsm log:
2017-02-12 13:17:03,208 INFO  (jsonrpc/5) [storage.TaskManager.Task] (Task='baf14138-3e79-421e-90e0-e27920017dca') aborting: Task is aborted: u'Failed reload: /dev/mapper/360014055ce39b844c384d6dbe13f9544' - code 100 (task:1175)
2017-02-12 13:17:03,209 ERROR (jsonrpc/5) [storage.Dispatcher] Failed reload: /dev/mapper/360014055ce39b844c384d6dbe13f9544 (dispatcher:80)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/dispatcher.py", line 72, in wrapper
    result = ctask.prepare(func, *args, **kwargs)
  File "/usr/share/vdsm/storage/task.py", line 105, in wrapper
    return m(self, *a, **kw)
  File "/usr/share/vdsm/storage/task.py", line 1183, in prepare
    raise self.error

Engine log:
2017-02-12 13:17:04,262+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (default task-25) [4f3a17ac-5746-41da-a2b4-206d17be8fb5] Failed in 'GetDeviceListVDS' method

See logs attached .

More info :
lsblk output:
sdad                                                                                   65:208  0   11G  0 disk  
└─360014055ce39b844c384d6dbe13f9544 

multipath output:
360014055ce39b844c384d6dbe13f9544 dm-42 LIO-ORG ,FILEIO          
size=11G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 81:0:0:0 sdad 65:208 active ready running

Engine : ovirt-engine-4.1.0.4-0.1.el7.noarch
VDSM :4.19.2-2

Created attachment 1249529 [details]
engine & vdsm logs

Hi,

Can you provide the pvs output ?

Thanks

(In reply to Fred Rolland from comment #11)
> Hi,
> 
> Can you provide the pvs output ?
> 
> Thanks

Lucky I saved it last time around .

I used :
lvm pvs -vvvv --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/SioFABRICVicinity_iqn.2015-03.com.iofabric:ovirt-master-00|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } '

I saved the output in a file "lvs_pvs_output.txt" which I attached .

Created attachment 1250215 [details]
lvs pvs -vvvv --config output

(In reply to Avihai from comment #12)
> (In reply to Fred Rolland from comment #11)
> > Hi,
> > 
> > Can you provide the pvs output ?
> > 
> > Thanks
> 
> Lucky I saved it last time around .
> 
> I used :
> lvm pvs -vvvv --config ' devices { preferred_names = ["^/dev/mapper/"]
> ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3
> filter = [
> '\''a|/dev/mapper/SioFABRICVicinity_iqn.2015-03.com.iofabric:ovirt-master-
> 00|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1 
> prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup { 
> retain_min = 50  retain_days = 0 } '
> 
> I saved the output in a file "lvs_pvs_output.txt" which I attached .

Why did you use this command ?
Do you have a device named : /dev/mapper/SioFABRICVicinity_iqn.2015-03.com.iofabric:ovirt-master-00 ?

How did you reproduce ?
How did you get a PV with a name with colon?

I could no see anything related in the log.

I ran this command as it was used in the original bug to provide as much info as possible .( at least that was the intent )

My device/tartget name is "iqn.2015-03.com.iofabric:ovirt-master-00"

In the lvs_pvs_output.txt see a line with my current target name , 
is this what we expect? 

#device/dev-cache.c:352         /dev/disk/by-path/ip-10.35.88.157:3260-iscsi-iqn.2015-03.com.iofabric:ovirt-master-00-lun-0: Aliased to /dev/sdad in device cache (65:208)

Just to clarify , these are the steps of what I did :

1) I used a targetcli server that we have (ip=10.35.88.157 )  

2) On this server I created a target name with colon ":" similar to the bug :
   "iqn.2015-03.com.iofabric:ovirt-master-00" and added a 11G lun to this target.

Is this not what ensures that PVS is with ":" ? 
If not please suggest a way to do it .

3) From engine GUI I logged in to this server sucessfully & saw the expected lun.

4) I choose the lun tried to created a new storage domain with this lun & failed

Created attachment 1250511 [details]
Also adding print screen of engine GUI to make it more clear .

Created attachment 1250512 [details]
Another print screen with the GUI error

(In reply to Avihai from comment #15)
> 2) On this server I created a target name with colon ":" similar to the bug :
>    "iqn.2015-03.com.iofabric:ovirt-master-00" and added a 11G lun to this
> target.
> 
> Is this not what ensures that PVS is with ":" ? 

No

> If not please suggest a way to do it .

We don't know how to reproduce a device name including a ":".

The fix was verified by the reporter. I don't think there is anything to verify
here except checking for regressions.

If you have trouble with target name including ":" it may be another bug. Please
check that you login to this target from the shell using iscsiadm. If it works in
iscsiadm but not in vdsm, please open another bug for it.

@Nir , is there a way to simulate this special lun-id by hard coding it in the VDSM code somehow ? 

@Raz , Currently there is no way to verify this bug , 
how do you want to proceed here ?

According to Nir's comment - comment #18 closing this bug