Bug 1303580
Summary: | RFE -Full Fibre Channel multi-pathing support in Overcloud nodes | |||
---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Jeremy <jmelvin> | |
Component: | rhosp-director | Assignee: | Bob Fournier <bfournie> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | bjacot | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 7.0 (Kilo) | CC: | abishop, andres, bfournie, cchen, djuran, gkadam, hbrock, jcoufal, jschluet, mburns, ojanas, pablo.iranzo, racedoro, rhel-osp-director-maint, svigan, tvvcox | |
Target Milestone: | --- | Keywords: | FutureFeature, TestOnly, ZStream | |
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Enhancement | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1623050 (view as bug list) | Environment: | ||
Last Closed: | 2018-08-28 12:19:30 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1623050 |
Description
Jeremy
2016-02-01 11:08:43 UTC
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10. Proper multipath support in overcloud-full image is still missing as of RHOSP 12. device-mapper-multipath package is included in the image - but /etc/multipath.conf is missing from image filesystem as from initramfs - so that multipath daemon fails to start and multipathed OS root device is not configured. However there is a post-deployment workaround to "migrate" overcloud nodes to use multipathed OS device. Steps are (exec as root user): 1. Create /etc/multipath.conf on overcloud node cat << EOF > /mnt/etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { } EOF 2. Restart multipath daemon on overcloud node systemctl restart multipathd systemctl status multipathd 3. Run dracut to include /etc/multipath.conf for initramfs dracut -f -v 4. Reboot overcloud node -- to pick up multipathed OS device Follow procedures from https://access.redhat.com/solutions/1977013 5. Verify that OS is using multipathed root device now lsblk multipath -ll df -h Question for RH engineers: Why /etc/multipath.conf cannot be included into overcloud-full image OR alternatively why can't there be overcloud-full image version with working multipathd -- if multipath cannot be enabled for all? Here is the full recipe for customizing overcloud-full.qcow2 and overcloud-full.initrd for multipath support. ### Modify overcloud-full.initrd image # the following needs root user for proper file rights mkdir ~/tmp/overcloud-full-initrd cd ~/tmp/overcloud-full-initrd /usr/lib/dracut/skipcpio /home/stack/images/overcloud-full.initrd | zcat | cpio -ivd | pax -r # add missing dir/file mkdir etc/multipath chmod 755 etc/multipath cat << EOF > etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { } EOF chmod 664 etc/multipath.conf # verify change find -name "*multipath*" # compile image find . 2>/dev/null | cpio --quiet -c -o | gzip -8 > ../overcloud-full-mpath.initrd # verify image contents cd .. lsinitrd overcloud-full-mpath.initrd | less # replace original cp overcloud-full-mpath.initrd /home/stack/images/overcloud-full.initrd # load updated initrd to undercloud glance su - stack source stackrc openstack overcloud image upload --update-existing --image-path /home/stack/images/ # NB! This might produce the following warning (which we will mitigate in qcow2 image properties update?): Link overcloud image to it's initrd and kernel images is MISSING OR leads to OLD image. You can keep it or fix it manually. ### Modify overcloud-full.qcow2 partition image # NB! Assuming su - stack & source stackrc here # fix missing /etc/multipath.conf problem in order multipathd to start # install tools for qcow2 modification sudo yum -y install libguestfs-tools # the following procedure needs root account sudo su - # mount and modify image filesystem contents export LIBGUESTFS_BACKEND=direct guestmount -a /home/stack/images/overcloud-full.qcow2 -i --rw /mnt cat << EOF > /mnt/etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { } EOF ls -l /mnt/etc/multipath.conf cat /mnt/etc/multipath.conf # copy also updated ramdisk cp /root/tmp/overcloud-full-mpath.initrd /mnt/boot/initramfs-3.10.0-693.11.1.el7.x86_64.img umount /mnt exit # update overcloud image in glance openstack overcloud image upload --update-existing --image-path /home/stack/images/ # NB! Update ramdisk reference for overcloud-full image (otherwise still old image is being deployed) openstack image list | awk '/overcloud-full.initrd / { print $2 }' openstack image set --property ramdisk_id='<uuid_of_updated_overcloud-full.initrd>' overcloud-full # baremetal nodes also need to pick up the image metadata update (for initramdisk update) openstack baremetal node manage <baremetal_host> openstack overcloud node configure --all-manageable This would be a document describing the generic process to enable it. Different storage backend have different configuration flags (or no config at all). We need to verify the process first then add a new article. Verified that updated images were in the overcloud. did suggested step in OSP13 $cat /etc/yum.repos.d/latest-installed 13 -p 2018-08-03.3 [heat-admin@controller-0 ~]$ sudo -i [root@controller-0 ~]# systemctl status multipathd ● multipathd.service - Device-Mapper Multipath Device Controller Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2018-08-10 15:23:04 UTC; 1h 29min ago Process: 609 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS) Process: 606 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS) Process: 601 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS) Main PID: 612 (multipathd) Tasks: 6 Memory: 2.8M CGroup: /system.slice/multipathd.service └─612 /sbin/multipathd Aug 10 15:23:04 controller-0 multipathd[612]: path checkers start up Aug 10 15:23:04 controller-0 systemd[1]: Starting Device-Mapper Multipath Device Controller... Aug 10 15:23:04 controller-0 systemd[1]: Started Device-Mapper Multipath Device Controller. [root@controller-0 ~]# cat /etc/multipath.conf defaults { polling_interval 10 user_friendly_names no find_multipaths yes } blacklist { } As this is TestOnly and does not involve code changes marking as closed as it has been verified in Comment 13. Ganesh - this bug is targeted and tested for OSP-13, please do not reopen this bug for a request to test for OSP-12. I have cloned this bug to a TestOnly bug targeted to OSP-12 - https://bugzilla.redhat.com/show_bug.cgi?id=1623050. Testing results will be posted to that bug. Chen - please don't use this closed bug for questions on an unrelated procedure. You can update https://bugzilla.redhat.com/show_bug.cgi?id=1582974 which is the RFE for multipathd containers or open a separate RFE. |