Bug 1303580

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Proper multipath support in overcloud-full image is still missing as of RHOSP 12. device-mapper-multipath package is included in the image - but /etc/multipath.conf is missing from image filesystem as from initramfs - so that multipath daemon fails to start and multipathed OS root device is not configured.

However there is a post-deployment workaround to "migrate" overcloud nodes to use multipathed OS device. 

Steps are (exec as root user):

1. Create /etc/multipath.conf on overcloud node

cat << EOF > /mnt/etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }

blacklist {
}
EOF

2. Restart multipath daemon on overcloud node

systemctl restart multipathd
systemctl status multipathd

3. Run dracut to include /etc/multipath.conf for initramfs

dracut -f -v

4. Reboot overcloud node -- to pick up multipathed OS device

Follow procedures from https://access.redhat.com/solutions/1977013


5. Verify that OS is using multipathed root device now

lsblk
multipath -ll
df -h

Question for RH engineers:
Why /etc/multipath.conf cannot be included into overcloud-full image OR alternatively why can't there be overcloud-full image version with working multipathd -- if multipath cannot be enabled for all?

Here is the full recipe for customizing overcloud-full.qcow2 and overcloud-full.initrd for multipath support.

### Modify overcloud-full.initrd image

# the following needs root user for proper file rights
mkdir ~/tmp/overcloud-full-initrd
cd ~/tmp/overcloud-full-initrd
/usr/lib/dracut/skipcpio /home/stack/images/overcloud-full.initrd | zcat | cpio -ivd | pax -r
 
# add missing dir/file
mkdir etc/multipath
chmod 755 etc/multipath
cat << EOF > etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }
 
blacklist {
}
EOF
chmod 664 etc/multipath.conf
 
# verify change
find -name "*multipath*"
 
# compile image
find . 2>/dev/null | cpio --quiet -c -o | gzip -8  > ../overcloud-full-mpath.initrd
 
# verify image contents
cd ..
lsinitrd overcloud-full-mpath.initrd | less
 
# replace original
cp overcloud-full-mpath.initrd /home/stack/images/overcloud-full.initrd
 
# load updated initrd to undercloud glance
su - stack
source stackrc
openstack overcloud image upload --update-existing --image-path /home/stack/images/
# NB! This might produce the following warning (which we will mitigate in qcow2 image properties update?):
Link overcloud image to it's initrd and kernel images is MISSING OR leads to OLD image. You can keep it or fix it manually.

### Modify overcloud-full.qcow2 partition image

# NB! Assuming su - stack & source stackrc here
 
# fix missing /etc/multipath.conf problem in order multipathd to start
# install tools for qcow2 modification
sudo yum -y install libguestfs-tools
 
# the following procedure needs root account
sudo su -
# mount and modify image filesystem contents
export LIBGUESTFS_BACKEND=direct
guestmount -a /home/stack/images/overcloud-full.qcow2 -i --rw /mnt
cat << EOF > /mnt/etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }
 
blacklist {
}
EOF
ls -l /mnt/etc/multipath.conf
cat /mnt/etc/multipath.conf
 
# copy also updated ramdisk
cp /root/tmp/overcloud-full-mpath.initrd /mnt/boot/initramfs-3.10.0-693.11.1.el7.x86_64.img
 
umount /mnt
exit
 
# update overcloud image in glance
openstack overcloud image upload --update-existing --image-path /home/stack/images/
 
# NB! Update ramdisk reference for overcloud-full image (otherwise still old image is being deployed)
openstack image list | awk '/overcloud-full.initrd / { print $2 }'
openstack image set --property ramdisk_id='<uuid_of_updated_overcloud-full.initrd>' overcloud-full
 
# baremetal nodes also need to pick up the image metadata update (for initramdisk update)
openstack baremetal node manage <baremetal_host>
openstack overcloud node configure --all-manageable

This would be a document describing the generic process to enable it. Different storage backend have different configuration flags (or no config at all). We need to verify the process first then add a new article.

Verified that updated images were in the overcloud.  did suggested step in OSP13
$cat /etc/yum.repos.d/latest-installed 
13   -p 2018-08-03.3

[heat-admin@controller-0 ~]$ sudo -i
[root@controller-0 ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-08-10 15:23:04 UTC; 1h 29min ago
  Process: 609 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
  Process: 606 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
  Process: 601 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 612 (multipathd)
    Tasks: 6
   Memory: 2.8M
   CGroup: /system.slice/multipathd.service
           └─612 /sbin/multipathd

Aug 10 15:23:04 controller-0 multipathd[612]: path checkers start up
Aug 10 15:23:04 controller-0 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Aug 10 15:23:04 controller-0 systemd[1]: Started Device-Mapper Multipath Device Controller.
[root@controller-0 ~]# cat /etc/multipath.conf 
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }
 
blacklist {
}

As this is TestOnly and does not involve code changes marking as closed as it has been verified in Comment 13.

Ganesh - this bug is targeted and tested for OSP-13, please do not reopen this bug for a request to test for OSP-12.  

I have cloned this bug to a TestOnly bug targeted to OSP-12 - https://bugzilla.redhat.com/show_bug.cgi?id=1623050. Testing results will be posted to that bug.

Chen - please don't use this closed bug for questions on an unrelated procedure. You can update https://bugzilla.redhat.com/show_bug.cgi?id=1582974 which is the RFE for multipathd containers or open a separate RFE.