Description of problem: When deploying Red Hat OpenStack 7.x using Red Hat Director, I noticed support for full Fibre Channel multi-pathing support seems to be missing. Neither the "multipathd" daemon nor its configuration is available. Also, I can't seem to find any references in Red Hat OpenStack documentation or Red Hat Director templates on how to configure Fibre Channel multi-pathing in the Overcloud nodes. This is a must for us, as we have several compute nodes using SAN-provided disk (instead of DAS) over Fibre Channel and, for reliability, we need multi-pathing enabled and configured. Where are you experiencing the behavior? What environment? Full Fibre Channel multi-pathing support available in Overcloud nodes and configured via Red Hat Director.
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10.
Proper multipath support in overcloud-full image is still missing as of RHOSP 12. device-mapper-multipath package is included in the image - but /etc/multipath.conf is missing from image filesystem as from initramfs - so that multipath daemon fails to start and multipathed OS root device is not configured. However there is a post-deployment workaround to "migrate" overcloud nodes to use multipathed OS device. Steps are (exec as root user): 1. Create /etc/multipath.conf on overcloud node cat << EOF > /mnt/etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { } EOF 2. Restart multipath daemon on overcloud node systemctl restart multipathd systemctl status multipathd 3. Run dracut to include /etc/multipath.conf for initramfs dracut -f -v 4. Reboot overcloud node -- to pick up multipathed OS device Follow procedures from https://access.redhat.com/solutions/1977013 5. Verify that OS is using multipathed root device now lsblk multipath -ll df -h Question for RH engineers: Why /etc/multipath.conf cannot be included into overcloud-full image OR alternatively why can't there be overcloud-full image version with working multipathd -- if multipath cannot be enabled for all?
Here is the full recipe for customizing overcloud-full.qcow2 and overcloud-full.initrd for multipath support. ### Modify overcloud-full.initrd image # the following needs root user for proper file rights mkdir ~/tmp/overcloud-full-initrd cd ~/tmp/overcloud-full-initrd /usr/lib/dracut/skipcpio /home/stack/images/overcloud-full.initrd | zcat | cpio -ivd | pax -r # add missing dir/file mkdir etc/multipath chmod 755 etc/multipath cat << EOF > etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { } EOF chmod 664 etc/multipath.conf # verify change find -name "*multipath*" # compile image find . 2>/dev/null | cpio --quiet -c -o | gzip -8 > ../overcloud-full-mpath.initrd # verify image contents cd .. lsinitrd overcloud-full-mpath.initrd | less # replace original cp overcloud-full-mpath.initrd /home/stack/images/overcloud-full.initrd # load updated initrd to undercloud glance su - stack source stackrc openstack overcloud image upload --update-existing --image-path /home/stack/images/ # NB! This might produce the following warning (which we will mitigate in qcow2 image properties update?): Link overcloud image to it's initrd and kernel images is MISSING OR leads to OLD image. You can keep it or fix it manually. ### Modify overcloud-full.qcow2 partition image # NB! Assuming su - stack & source stackrc here # fix missing /etc/multipath.conf problem in order multipathd to start # install tools for qcow2 modification sudo yum -y install libguestfs-tools # the following procedure needs root account sudo su - # mount and modify image filesystem contents export LIBGUESTFS_BACKEND=direct guestmount -a /home/stack/images/overcloud-full.qcow2 -i --rw /mnt cat << EOF > /mnt/etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { } EOF ls -l /mnt/etc/multipath.conf cat /mnt/etc/multipath.conf # copy also updated ramdisk cp /root/tmp/overcloud-full-mpath.initrd /mnt/boot/initramfs-3.10.0-693.11.1.el7.x86_64.img umount /mnt exit # update overcloud image in glance openstack overcloud image upload --update-existing --image-path /home/stack/images/ # NB! Update ramdisk reference for overcloud-full image (otherwise still old image is being deployed) openstack image list | awk '/overcloud-full.initrd / { print $2 }' openstack image set --property ramdisk_id='<uuid_of_updated_overcloud-full.initrd>' overcloud-full # baremetal nodes also need to pick up the image metadata update (for initramdisk update) openstack baremetal node manage <baremetal_host> openstack overcloud node configure --all-manageable
This would be a document describing the generic process to enable it. Different storage backend have different configuration flags (or no config at all). We need to verify the process first then add a new article.
Verified that updated images were in the overcloud. did suggested step in OSP13 $cat /etc/yum.repos.d/latest-installed 13 -p 2018-08-03.3 [heat-admin@controller-0 ~]$ sudo -i [root@controller-0 ~]# systemctl status multipathd ● multipathd.service - Device-Mapper Multipath Device Controller Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-08-10 15:23:04 UTC; 1h 29min ago Process: 609 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS) Process: 606 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS) Process: 601 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS) Main PID: 612 (multipathd) Tasks: 6 Memory: 2.8M CGroup: /system.slice/multipathd.service └─612 /sbin/multipathd Aug 10 15:23:04 controller-0 multipathd[612]: path checkers start up Aug 10 15:23:04 controller-0 systemd[1]: Starting Device-Mapper Multipath Device Controller... Aug 10 15:23:04 controller-0 systemd[1]: Started Device-Mapper Multipath Device Controller. [root@controller-0 ~]# cat /etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { }
As this is TestOnly and does not involve code changes marking as closed as it has been verified in Comment 13.
Ganesh - this bug is targeted and tested for OSP-13, please do not reopen this bug for a request to test for OSP-12. I have cloned this bug to a TestOnly bug targeted to OSP-12 - https://bugzilla.redhat.com/show_bug.cgi?id=1623050. Testing results will be posted to that bug.
Chen - please don't use this closed bug for questions on an unrelated procedure. You can update https://bugzilla.redhat.com/show_bug.cgi?id=1582974 which is the RFE for multipathd containers or open a separate RFE.