Bug 1303580 - RFE -Full Fibre Channel multi-pathing support in Overcloud nodes
Summary: RFE -Full Fibre Channel multi-pathing support in Overcloud nodes
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Bob Fournier
QA Contact: bjacot
URL:
Whiteboard:
Depends On:
Blocks: 1623050
TreeView+ depends on / blocked
 
Reported: 2016-02-01 11:08 UTC by Jeremy
Modified: 2023-12-15 15:50 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
: 1623050 (view as bug list)
Environment:
Last Closed: 2018-08-28 12:19:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-12794 0 None None None 2022-02-21 19:13:33 UTC

Description Jeremy 2016-02-01 11:08:43 UTC
Description of problem:
When deploying Red Hat OpenStack 7.x using Red Hat Director, I noticed support for full Fibre Channel multi-pathing support seems to be missing. Neither the "multipathd" daemon nor its configuration is available. Also, I can't seem to find any references in Red Hat OpenStack documentation or Red Hat Director templates on how to configure Fibre Channel multi-pathing in the Overcloud nodes.

This is a must for us, as we have several compute nodes using SAN-provided disk (instead of DAS) over Fibre Channel and, for reliability, we need multi-pathing enabled and configured.

Where are you experiencing the behavior?  What environment?

Full Fibre Channel multi-pathing support available in Overcloud nodes and configured via Red Hat Director.

Comment 3 Mike Burns 2016-04-07 21:07:13 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 5 Andres Toomsalu 2018-01-17 22:29:25 UTC
Proper multipath support in overcloud-full image is still missing as of RHOSP 12. device-mapper-multipath package is included in the image - but /etc/multipath.conf is missing from image filesystem as from initramfs - so that multipath daemon fails to start and multipathed OS root device is not configured.

However there is a post-deployment workaround to "migrate" overcloud nodes to use multipathed OS device. 

Steps are (exec as root user):

1. Create /etc/multipath.conf on overcloud node

cat << EOF > /mnt/etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }

blacklist {
}
EOF

2. Restart multipath daemon on overcloud node

systemctl restart multipathd
systemctl status multipathd

3. Run dracut to include /etc/multipath.conf for initramfs

dracut -f -v

4. Reboot overcloud node -- to pick up multipathed OS device

Follow procedures from https://access.redhat.com/solutions/1977013


5. Verify that OS is using multipathed root device now

lsblk
multipath -ll
df -h

Question for RH engineers:
Why /etc/multipath.conf cannot be included into overcloud-full image OR alternatively why can't there be overcloud-full image version with working multipathd -- if multipath cannot be enabled for all?

Comment 6 Andres Toomsalu 2018-01-18 10:23:49 UTC
Here is the full recipe for customizing overcloud-full.qcow2 and overcloud-full.initrd for multipath support.

### Modify overcloud-full.initrd image

# the following needs root user for proper file rights
mkdir ~/tmp/overcloud-full-initrd
cd ~/tmp/overcloud-full-initrd
/usr/lib/dracut/skipcpio /home/stack/images/overcloud-full.initrd | zcat | cpio -ivd | pax -r
 
# add missing dir/file
mkdir etc/multipath
chmod 755 etc/multipath
cat << EOF > etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }
 
blacklist {
}
EOF
chmod 664 etc/multipath.conf
 
# verify change
find -name "*multipath*"
 
# compile image
find . 2>/dev/null | cpio --quiet -c -o | gzip -8  > ../overcloud-full-mpath.initrd
 
# verify image contents
cd ..
lsinitrd overcloud-full-mpath.initrd | less
 
# replace original
cp overcloud-full-mpath.initrd /home/stack/images/overcloud-full.initrd
 
# load updated initrd to undercloud glance
su - stack
source stackrc
openstack overcloud image upload --update-existing --image-path /home/stack/images/
# NB! This might produce the following warning (which we will mitigate in qcow2 image properties update?):
Link overcloud image to it's initrd and kernel images is MISSING OR leads to OLD image. You can keep it or fix it manually.

### Modify overcloud-full.qcow2 partition image

# NB! Assuming su - stack & source stackrc here
 
# fix missing /etc/multipath.conf problem in order multipathd to start
# install tools for qcow2 modification
sudo yum -y install libguestfs-tools
 
# the following procedure needs root account
sudo su -
# mount and modify image filesystem contents
export LIBGUESTFS_BACKEND=direct
guestmount -a /home/stack/images/overcloud-full.qcow2 -i --rw /mnt
cat << EOF > /mnt/etc/multipath.conf
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }
 
blacklist {
}
EOF
ls -l /mnt/etc/multipath.conf
cat /mnt/etc/multipath.conf
 
# copy also updated ramdisk
cp /root/tmp/overcloud-full-mpath.initrd /mnt/boot/initramfs-3.10.0-693.11.1.el7.x86_64.img
 
umount /mnt
exit
 
# update overcloud image in glance
openstack overcloud image upload --update-existing --image-path /home/stack/images/
 
# NB! Update ramdisk reference for overcloud-full image (otherwise still old image is being deployed)
openstack image list | awk '/overcloud-full.initrd / { print $2 }'
openstack image set --property ramdisk_id='<uuid_of_updated_overcloud-full.initrd>' overcloud-full
 
# baremetal nodes also need to pick up the image metadata update (for initramdisk update)
openstack baremetal node manage <baremetal_host>
openstack overcloud node configure --all-manageable

Comment 12 Ramon Acedo 2018-08-01 15:59:30 UTC
This would be a document describing the generic process to enable it. Different storage backend have different configuration flags (or no config at all). We need to verify the process first then add a new article.

Comment 13 bjacot 2018-08-10 16:53:49 UTC
Verified that updated images were in the overcloud.  did suggested step in OSP13
$cat /etc/yum.repos.d/latest-installed 
13   -p 2018-08-03.3

[heat-admin@controller-0 ~]$ sudo -i
[root@controller-0 ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-08-10 15:23:04 UTC; 1h 29min ago
  Process: 609 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
  Process: 606 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
  Process: 601 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 612 (multipathd)
    Tasks: 6
   Memory: 2.8M
   CGroup: /system.slice/multipathd.service
           └─612 /sbin/multipathd

Aug 10 15:23:04 controller-0 multipathd[612]: path checkers start up
Aug 10 15:23:04 controller-0 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Aug 10 15:23:04 controller-0 systemd[1]: Started Device-Mapper Multipath Device Controller.
[root@controller-0 ~]# cat /etc/multipath.conf 
defaults {
           polling_interval      10
           user_friendly_names   no
           find_multipaths       yes
          }
 
blacklist {
}

Comment 15 Bob Fournier 2018-08-24 20:26:44 UTC
As this is TestOnly and does not involve code changes marking as closed as it has been verified in Comment 13.

Comment 17 Bob Fournier 2018-08-28 12:19:30 UTC
Ganesh - this bug is targeted and tested for OSP-13, please do not reopen this bug for a request to test for OSP-12.  

I have cloned this bug to a TestOnly bug targeted to OSP-12 - https://bugzilla.redhat.com/show_bug.cgi?id=1623050. Testing results will be posted to that bug.

Comment 19 Bob Fournier 2018-12-17 15:14:46 UTC
Chen - please don't use this closed bug for questions on an unrelated procedure. You can update https://bugzilla.redhat.com/show_bug.cgi?id=1582974 which is the RFE for multipathd containers or open a separate RFE.


Note You need to log in before you can comment on or make changes to this bug.