Workaround:
Manually running "multipath -r" after reboot gets the modified configuration from "/etc/multipath.conf"

Offhand this seems like a RHEV-H persistency issue, so flagging with whiteboard=node.
Fabian - if you need our hand here, just say so.

Just want to clarify, changes in /etc/multipath.conf is getting persisted after reboot but multipathd does not load those changes until we manually run 'multipath -r' on RHEV-H.

Should be handled in BZ #1225182.

Test version:
Red Hat Enterprise Virtualization Hypervisor 7.1 (20150512.1.el7ev)
ovirt-node-3.2.2-3.el7.noarch
vdsm-4.16.13.1-1.el7ev.x86_64

Test machine:
hp-z800-02(multipath iSCSI)
iSCSI: QLogic Corp. ISP4032-based iSCSI TOE IPv6 HBA /qla4xxx

Test steps:
1. Install RHEV-H 7.1
2. register it to rhevm
3. Modify /etc/multipath.conf(pasted from http://pastebin.test.redhat.com/295198) and persist it.
4. Reboot the system. 

Test result:
/etc/multipath.conf can be persisted after reboot, but I don't know how the multipath paths can effect after modify multipath.conf.

In our test machine, before modify /etc/multipath.conf, all multipath lun can work fine. after modify it, all luns still can work fine. I can't find any difference.

Hi fabiand,

Could you guide us to how to test this bug?

Thanks!

This bug can be tested by:

1. Ensure that /etc/multipath.conf is persisted
2. Look for a unique configuration item in /etc/multipath.conf, and look for this configuration item in the output of "multipathd -k" followed by a "show config"

Sachin, the assumption is that persisting the multipath.conf and reloading it during boot will bring up the right paths.

And problem is that we can not reproduce the issue in our environment, that means that we van only verify this bug by making sure that we persist the file, and the contents are reloaded.

Fabian, from patch on BZ #1225182, reload multipathd during boot.
Currently we need reload multipathd manually to make effect.

Then to verify this bug we can do this
To reproduce it:
1. Installed RHEV-H 
2. Registered RHEVH to RHEVM.
3. Modified /etc/multipath.conf, such as
    user_friendly_names     no  # default
    user_friendly_names     yes # new values
4. Reboot rhevh
5. check the new value after reboot rhevh, there still not update.
# multipathd show config | grep user_friendly_names
	user_friendly_names "no"
		user_friendly_names no
6. service multipathd reload
7. then modified configuration works

After the patches on BZ #1225182 in ovirt-node 3.5 branch, then the modified configuration in multipath.conf will take effect after rhevh reboot, no need reload multipathd manually.

Please give an ack for above to verify this bug. Thanks.

> Please give an ack for above to verify this bug. Thanks.

already answer my question on comment 8. Thanks Fabian.

Sachin, Is our customer willing to verify this bug? Thanks.
Because we can not reproduce the bug report issue in currently enviroment, we only can verify this bug to check persist the file, the modified configuration reload after rhevh reboot.

(In reply to Ying Cui from comment #10)
> Fabian, from patch on BZ #1225182, reload multipathd during boot.
> Currently we need reload multipathd manually to make effect.
> 
> Then to verify this bug we can do this
> To reproduce it:
> 1. Installed RHEV-H 
> 2. Registered RHEVH to RHEVM.
> 3. Modified /etc/multipath.conf, such as
>     user_friendly_names     no  # default
>     user_friendly_names     yes # new values
> 4. Reboot rhevh
> 5. check the new value after reboot rhevh, there still not update.
> # multipathd show config | grep user_friendly_names
> 	user_friendly_names "no"
> 		user_friendly_names no
> 6. service multipathd reload
> 7. then modified configuration works
> 
> After the patches on BZ #1225182 in ovirt-node 3.5 branch, then the modified
> configuration in multipath.conf will take effect after rhevh reboot, no need
> reload multipathd manually.
> 
> Please give an ack for above to verify this bug. Thanks.

Hi ycui and fabiand,

Thanks you for your help, I understand now. if need I will verify persist part.

Hi Ying,  customer has provided the 1.  sosreport of rebooted Host BEFORE running "multipath -r" and 2.  sosreport of rebooted Host AFTER running "multipath -r"

The sosrpeort has all the details required to verify this bug.

Let me know what other details needed so I will collect it further.

(In reply to Sachin Raje from comment #14)
> Hi Ying,  customer has provided the 1.  sosreport of rebooted Host BEFORE
> running "multipath -r" and 2.  sosreport of rebooted Host AFTER running
> "multipath -r"
> 
> The sosrpeort has all the details required to verify this bug.
> 
> Let me know what other details needed so I will collect it further.

Sachin, here we probably got misunderstanding on comment 12, I mean whether customer can help to fully verify this bug when this bug is fixed by devel and ON_QA. Because in QE side, we can not reproduce the original issue which customer encountered, we only can do partial verify on checking persist file and multipath reload after rhevh restart.

Thanks
Ying

Hello Ying,

   Thanks for clarification. I'll check and confirm with customer about testing and verification of this bug in his rhev environment.

I'll let you know soon.

Regards,
Sachin

(In reply to Sachin Raje from comment #16)
> Hello Ying,
> 
>    Thanks for clarification. I'll check and confirm with customer about
> testing and verification of this bug in his rhev environment.
> 
> I'll let you know soon.

Thanks, and please note, the rhevh build is not built yet on brew for testing, we will paste the brew link soon ~2 days on this bug, then you can provide the iso to customer to verify it.

according to comment 8, comment 9, comment 10 and comment 16, I will give qa_ack+ here.

Hi there,

basically you are able to reproduce it by changing one "non-path-routing-related" settings in /etc/multipath.conf, e.g. failback, user_friendly_names. If you set something nonstandard in there and that configuration is in place after a reboot, then the issue is fixed.

But I will test it in my environment if you provide me with a RHEV-H RPM to test with.

regards, Gunther

Test version:
rhev-hypervisor7-7.2-20151112.1
ovirt-node-3.6.0-0.20.20151103git3d3779a.el7ev.noarch

Test steps:
1. Installed RHEV-H 
2. Registered RHEVH to RHEVM.
3. Modified /etc/multipath.conf, such as
    user_friendly_names     no  # default
    user_friendly_names     yes # new values
4. Reboot rhevh
5. check the new value after reboot rhevh

# multipathd show config | grep user_friendly_names
	user_friendly_names "yes"
		user_friendly_names no


Hi fabian.

 The persist part work well, can you provide a download link for Gunther Schlegel to full verify this bug?

Thanks!

Checn, thanks for the reminder.

Gunther, can you kindly try to reproduce this bug with the RHEV-H 3.6 beta 1 build?

Please note that upgrades of RHEV-H 3.6 beta 1 are not working correctly (will be fixed in beta2), so in case you try, please use a clean installation.

Fabian,

I am actually not even sure if I have access to the software you have mentioned, nevertheless, I will definitely not find time to test any of this in 2015.

regards, Gunther

Thanks for your reply Gunther.

Chen, I'd say when then go forward and verify this bug according to comment 24.

(In reply to Fabian Deutsch from comment #28)
> Chen, I'd say when then go forward and verify this bug according to comment
> 24.

Verify this bug according #c24, #c28

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html