multipathd claimed the sdb, sdc, and sdd devices and prevented them from being used directly:

# multipath -ll
Hitachi_HUA722010CLA330_JPW9M0N20D247E dm-1 ATA     ,Hitachi HUA72201
size=932G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:0:0:0 sdb 8:16 active ready running
Hitachi_HUA722010CLA330_JPW9J0N20BMZHC dm-2 ATA     ,Hitachi HUA72201
size=932G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 3:0:0:0 sdd 8:48 active ready running
Hitachi_HUA722010CLA330_JPW9M0N20D268E dm-0 ATA     ,Hitachi HUA72201
size=932G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 2:0:0:0 sdc 8:32 active ready running

Either these disks needs to be blacklisted from multipath or the systemctl ceph-disk need to use the multipath device.

To be completely safe with users preferences in case they are using dm-multipath for some system or OSD disks, we probably want find_multipaths = yes. I guess users seem to prefer this and for RHEL we override the upstream default and set that to yes. The ceph iscsi tools were setting it back to no because you cannot set it at the per device level like other settings.

In the ceph iscsi config modules we will want to add code (I made a patch for this) to run /sbin/multipath device_name for the specific rbd images, so it will not matter what the user has set for find_multipaths.


Assuming we cannot make any code changes, then here are the manual instructions that we could add to the ceph iscsi ansible doc to work around the bug here and they will work ok for when we can release a code fix later:


1. After ansible-playbook ceph-iscsi-gw.yml is run log into each node running a iSCSI target and run

multipath -ll

if there are disks that the user did not intend to be used by dm-multipath, for example disks being used by OSDs, run

multipath -w device_name
multipath -f device_name

example:

multipath -w mpatha
multipath -f mpatha

2. Open /etc/multiapth.conf on each node running a iSCSI target and in the defaults section remove the global skip_partx and change the global user_friendly_names value to yes:

defaults {
        user_friendly_names yes
        find_multipaths no
}


2. By default, the ansible iscsi modules unblacklisted everything. Unless, you are using dm-multipath for specific devices you can blacklist everything again by adding

devnode ".*"

to the uncommenented out blacklist {} section at the bottom of the file so it looks like this:

blacklist {
        devnode ".*"
}

3. We do want dm-multipath for rbd devices, so add an exception for it by adding the following to the multipath.conf:

blacklist_exceptions {
        devnode "^rbd[0-9]"
}

4.  For rbd devices add the following to multipath.conf:

devices {
        device {
                vendor  "Ceph"
                product "RBD"
                skip_kpartx yes
                user_friendly_names no
        }
}

5. Reload the new settings:

systemctl reload multipathd



Hemanth, if you want I can run those commands on your system for you.

(In reply to Mike Christie from comment #9)
> To be completely safe with users preferences in case they are using
> dm-multipath for some system or OSD disks, we probably want find_multipaths
> = yes. I guess users seem to prefer this and for RHEL we override the
> upstream default and set that to yes. The ceph iscsi tools were setting it
> back to no because you cannot set it at the per device level like other
> settings.
> 
> In the ceph iscsi config modules we will want to add code (I made a patch
> for this) to run /sbin/multipath device_name for the specific rbd images, so
> it will not matter what the user has set for find_multipaths.
> 
> 
> Assuming we cannot make any code changes, then here are the manual
> instructions that we could add to the ceph iscsi ansible doc to work around
> the bug here and they will work ok for when we can release a code fix later:
> 
> 
> 1. After ansible-playbook ceph-iscsi-gw.yml is run log into each node
> running a iSCSI target and run
> 
> multipath -ll
> 
> if there are disks that the user did not intend to be used by dm-multipath,
> for example disks being used by OSDs, run
> 
> multipath -w device_name
> multipath -f device_name
> 
> example:
> 
> multipath -w mpatha
> multipath -f mpatha
> 
> 2. Open /etc/multiapth.conf on each node running a iSCSI target and in the
> defaults section remove the global skip_partx and change the global
> user_friendly_names value to yes:
> 
> defaults {
>         user_friendly_names yes
>         find_multipaths no
> }
> 
> 
> 2. By default, the ansible iscsi modules unblacklisted everything. Unless,
> you are using dm-multipath for specific devices you can blacklist everything
> again by adding
> 
> devnode ".*"
> 
> to the uncommenented out blacklist {} section at the bottom of the file so
> it looks like this:
> 
> blacklist {
>         devnode ".*"
> }
> 
> 3. We do want dm-multipath for rbd devices, so add an exception for it by
> adding the following to the multipath.conf:
> 
> blacklist_exceptions {
>         devnode "^rbd[0-9]"
> }
> 
> 4.  For rbd devices add the following to multipath.conf:
> 
> devices {
>         device {
>                 vendor  "Ceph"
>                 product "RBD"
>                 skip_kpartx yes
>                 user_friendly_names no
>         }
> }
> 
> 5. Reload the new settings:
> 
> systemctl reload multipathd
> 
> 
> 
> Hemanth, if you want I can run those commands on your system for you.

Mike, The machines are no more in that state to run the commands..

If this are the steps we are documenting as a workaround for the customers  then I am okay in running them. If not I will wait for the fix.

As the BZ is moved to 2.2, Can comment #9 be documented in Known Issues of 2.1??

(In reply to Hemanth Kumar from comment #12)
> As the BZ is moved to 2.2, Can comment #9 be documented in Known Issues of
> 2.1??

Yes. I am working on it.

For the most part the instructions are what you will do.

I am just trying to test and add more info about how to handle if the user is using dm-multipath for root or other disks on the OSD/gw machine.

Created attachment 1219550 [details]
Failover on N/W Failure

Hi Paul,

Failed the primary GW Node's N/W and the Failover happened within 15 secs..

Refer the attachment for the Performance monitor stats on Windows..


Will update the same after reboot..

(In reply to Hemanth Kumar from comment #14)
> Created attachment 1219550 [details]
> Failover on N/W Failure
> 
> Hi Paul,
> 
> Failed the primary GW Node's N/W and the Failover happened within 15 secs..
> 
> Refer the attachment for the Performance monitor stats on Windows..
> 
> 
> Will update the same after reboot..

Ignore Comment#14

Updated in the wrong BZ..

Looks good. Thanks Bara.

This issue only affects RHCS 2.1 since RHCS 2.2 will utilize a new approach for incorporating RBD-backed iSCSI.