Bug 1303580 - RFE -Full Fibre Channel multi-pathing support in Overcloud nodes [NEEDINFO]
RFE -Full Fibre Channel multi-pathing support in Overcloud nodes
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Hugh Brock
Shai Revivo
: FutureFeature, ZStream
Depends On:
  Show dependency treegraph
Reported: 2016-02-01 06:08 EST by Jeremy
Modified: 2018-06-20 22:07 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
pablo.iranzo: needinfo? (hbrock)
gkadam: needinfo? (hbrock)

Attachments (Terms of Use)

  None (edit)
Description Jeremy 2016-02-01 06:08:43 EST
Description of problem:
When deploying Red Hat OpenStack 7.x using Red Hat Director, I noticed support for full Fibre Channel multi-pathing support seems to be missing. Neither the "multipathd" daemon nor its configuration is available. Also, I can't seem to find any references in Red Hat OpenStack documentation or Red Hat Director templates on how to configure Fibre Channel multi-pathing in the Overcloud nodes.

This is a must for us, as we have several compute nodes using SAN-provided disk (instead of DAS) over Fibre Channel and, for reliability, we need multi-pathing enabled and configured.

Where are you experiencing the behavior?  What environment?

Full Fibre Channel multi-pathing support available in Overcloud nodes and configured via Red Hat Director.
Comment 3 Mike Burns 2016-04-07 17:07:13 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 5 Andres Toomsalu 2018-01-17 17:29:25 EST
Proper multipath support in overcloud-full image is still missing as of RHOSP 12. device-mapper-multipath package is included in the image - but /etc/multipath.conf is missing from image filesystem as from initramfs - so that multipath daemon fails to start and multipathed OS root device is not configured.

However there is a post-deployment workaround to "migrate" overcloud nodes to use multipathed OS device. 

Steps are (exec as root user):

1. Create /etc/multipath.conf on overcloud node

cat << EOF > /mnt/etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes

blacklist {

2. Restart multipath daemon on overcloud node

systemctl restart multipathd
systemctl status multipathd

3. Run dracut to include /etc/multipath.conf for initramfs

dracut -f -v

4. Reboot overcloud node -- to pick up multipathed OS device

Follow procedures from https://access.redhat.com/solutions/1977013

5. Verify that OS is using multipathed root device now

multipath -ll
df -h

Question for RH engineers:
Why /etc/multipath.conf cannot be included into overcloud-full image OR alternatively why can't there be overcloud-full image version with working multipathd -- if multipath cannot be enabled for all?
Comment 6 Andres Toomsalu 2018-01-18 05:23:49 EST
Here is the full recipe for customizing overcloud-full.qcow2 and overcloud-full.initrd for multipath support.

### Modify overcloud-full.initrd image

# the following needs root user for proper file rights
mkdir ~/tmp/overcloud-full-initrd
cd ~/tmp/overcloud-full-initrd
/usr/lib/dracut/skipcpio /home/stack/images/overcloud-full.initrd | zcat | cpio -ivd | pax -r
# add missing dir/file
mkdir etc/multipath
chmod 755 etc/multipath
cat << EOF > etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
blacklist {
chmod 664 etc/multipath.conf
# verify change
find -name "*multipath*"
# compile image
find . 2>/dev/null | cpio --quiet -c -o | gzip -8  > ../overcloud-full-mpath.initrd
# verify image contents
cd ..
lsinitrd overcloud-full-mpath.initrd | less
# replace original
cp overcloud-full-mpath.initrd /home/stack/images/overcloud-full.initrd
# load updated initrd to undercloud glance
su - stack
source stackrc
openstack overcloud image upload --update-existing --image-path /home/stack/images/
# NB! This might produce the following warning (which we will mitigate in qcow2 image properties update?):
Link overcloud image to it's initrd and kernel images is MISSING OR leads to OLD image. You can keep it or fix it manually.

### Modify overcloud-full.qcow2 partition image

# NB! Assuming su - stack & source stackrc here
# fix missing /etc/multipath.conf problem in order multipathd to start
# install tools for qcow2 modification
sudo yum -y install libguestfs-tools
# the following procedure needs root account
sudo su -
# mount and modify image filesystem contents
guestmount -a /home/stack/images/overcloud-full.qcow2 -i --rw /mnt
cat << EOF > /mnt/etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
blacklist {
ls -l /mnt/etc/multipath.conf
cat /mnt/etc/multipath.conf
# copy also updated ramdisk
cp /root/tmp/overcloud-full-mpath.initrd /mnt/boot/initramfs-3.10.0-693.11.1.el7.x86_64.img
umount /mnt
# update overcloud image in glance
openstack overcloud image upload --update-existing --image-path /home/stack/images/
# NB! Update ramdisk reference for overcloud-full image (otherwise still old image is being deployed)
openstack image list | awk '/overcloud-full.initrd / { print $2 }'
openstack image set --property ramdisk_id='<uuid_of_updated_overcloud-full.initrd>' overcloud-full
# baremetal nodes also need to pick up the image metadata update (for initramdisk update)
openstack baremetal node manage <baremetal_host>
openstack overcloud node configure --all-manageable

Note You need to log in before you can comment on or make changes to this bug.