Hide Forgot
Since code went into 5.1. from bz #248931, this clone should be used for the continuing issues.
Uploading the console output would be useful. Do you know what programs are trying to access the private paths. If it's only LVM, you should try filtering out the devices in /etc/lvm/lvm.conf. This will keep lvm from trying to scan them, which is what you want, since they are owned by the multipath device anyway. If programs are accessing the passive path directly (for example by reading from /dev/sdX, where /dev/sdX is a passive path) there is no way that device-mapper-multipath can stop them. LVM does this, but you can add filters to avoid it. If there are other programs that do it, they hopefully have a method for filtering devices as well. If the accesses are coming from multipath, it's either a configuration problem, or a bug in the code.
I will do that and upload the output tomorrow. As far as LVM, this happens even with unconfigured LUNs before creating any volumes on them. However, I see what you are getting at, since LVM will scan all block devices looking for LVMs. I tried this: filter = [ "a/sda/", "r/.*/" ] But the behavior is the same.
Hmm. If it's not LVM, do have any idea what programs ARE sending IO to the passive paths? To check if it's multipathd (which would point to a bad checker function.. either misconfigured or buggy) you can run # multipathd -k and then repeating run (you can scroll through the command history with the arrow keys) > show paths while watching # tail -f /var/log/messages This way, you should be able to see if the error messages from the paths coincide with the path checker running on them. Use ctrl-d to exit the interactive multipathd shell. Also, instead of removing the device-mapper-multipath module, you can just blacklist all your devices blacklist { devnode "*" } This will let you know if you can see these errors even without any multipathed devices running. Also, just to double-check that LVM scanning isn't causing these errors, you can run # lvscan -vv And check to see if it causes any errors, and also check to make sure that the passive paths aren't listed in the list of paths that it checks.
It looks like the LVM filter did in fact solve part of this problem, but not all of it. When I tested the filter above with 20 LUNs earlier, I timed the boot like usual, and when it started spewing the same errors and was into 10 minutes of boot time I wrote it off as the same behavior. However, on further inspection, it looks like the filter actually did solve a lot of issues. Previously, lvscan, pvs, pvcreate, and other commands would generate these errors, but now they don't. Boot time with 20 LUNs was 13 minutes instead of 17. So it seems there are still two other spots that the boot process hangs: 1. udev and 2. haldaemon. Udev gives errors during "Staring udev" which causes boot time relative to the number of attached LUN. But I think the worst offender is haldaemon. I turned off this service, and rebooted my system. Boot time with 20 LUNs was just under 6 minutes. According to an article I read online, "HAL is used for discovering storage, networking <snip>". So this seems to be causing the most amount of boot time. After I get a login screen, if I run "service haldaemon restart" it will print tons of errors and be unavailable until complete. Do I need to open a bug under haldaemon? What happens on other types of shared storage such as EMC or iSCSI and 20 LUNs? fdisk -l generates the same errors as before, but that is because it is trying to list the partition table on each device, including the passive ones. I turned on multipathd and modified the filter to: filter = [ "a/sda/", "a|mapper|", "r/.*/" ] And access to the dm device worked fine. I will attach a console log of the system booting with 20 LUNs. I added a $(date) statement at the top and bottom of rc.sysinit, which gives a good idea of elapsed time. Also, there are so many paths that the devices were not filtered out that started at /dev/sda, I need to find a better way to not filter out the local disk.
Also, everything above was done with all devices blacklisted in multipath.conf and multipathd off
Created attachment 233141 [details] serial console output of a Dell PE2950 booting with 20 LUNs attached to an MD3000 with redundant paths
*** Bug 307151 has been marked as a duplicate of this bug. ***
To filter the your sda[a-z] devices, you should just be able to use a filter line like: filter = [ "a/sda$/", "a|mapper|", "r/.*/" ] I couldn't find any straightforward way to filter the haldaemon, but that doesn't mean that there isn't one. You should probably open a bugzilla against hal. Either that or you can just change the component of this bug to hal, if you don't have any more multipath specific issues.
Are there any more multipath issues related to this bug? Otherwise I will close it.
Dell decided to go with LSI's MPP driver, so I have not been able to test this out lately. I will open a bug with haldaemon about this. This bug can be closed. Thank you.