Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: Failed to start Device-Mapper Multipath Device Controller with " mpathconf --enable --user_friendly_names y --find_multipaths y --with_multipathd y" Version-Release number of selected component (if applicable): Fedora-27-20171101.n.0/device-mapper-multipath-0.7.1-6.git847cc43.fc27.x86_64 How reproducible: always Steps to Reproduce: # man mpathconf --with_multipathd { y | n } If set to y, this runs service multipathd start to start the multipathd daemon on --enable, service multipathd stop to stop the multipathd daemon on --disable, and ser‐ vice multipathd reload to reconfigure multipathd on --user_frindly_names and --find_mul‐ tipaths. This option is set to n by default. 1.#service multipathd stop 2.#rpm -q device-mapper-multipath || yum install -y device-mapper-multipath 3.#mpathconf --enable --user_friendly_names y --find_multipaths y --with_multipathd y Job for multipathd.service failed because the control process exited with error code. See "systemctl status multipathd.service" and "journalctl -xe" for details. 4. # systemctl status multipathd.service ● multipathd.service - Device-Mapper Multipath Device Controller Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2017-11-02 05:15:29 EDT; 1min 26s ago Process: 11140 ExecStart=/sbin/multipathd -d -s (code=exited, status=1/FAILURE) Process: 11139 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS) Process: 11138 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=0/SUCCESS) Main PID: 11140 (code=exited, status=1/FAILURE) Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Starting Device-Mapper Multipath Device Controller... Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com multipathd[11140]: process is already running Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com multipathd[11140]: failed to create pidfile Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Main process exited, code=exited, status=1/FAILUR Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Failed to start Device-Mapper Multipath Device Controller. Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Unit entered failed state. Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Failed with result 'exit-code'. 5. # journalctl -xe -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit dnf-makecache.service has finished starting up. -- -- The start-up result is done. Nov 02 05:12:20 storageqe-16.rhts.eng.bos.redhat.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 Nov 02 05:12:20 storageqe-16.rhts.eng.bos.redhat.com audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 s Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Starting Device-Mapper Multipath Device -- Subject: Unit multipathd.service has begun start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit multipathd.service has begun starting up. Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com multipathd[11140]: process is already running Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com multipathd[11140]: failed to create pidfile Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Main process exited, Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Failed to start Device-Mapper Multipath -- Subject: Unit multipathd.service has failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit multipathd.service has failed. -- -- The result is failed. Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Unit entered failed Nov 02 05:15:29 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Failed with result ' Actual results: Failed to start Device-Mapper Multipath Device Controller with " mpathconf --enable --user_friendly_names y --find_multipaths y --with_multipathd y" Expected results: multipathd starts Additional info:
# rpm -qa | grep multipath device-mapper-multipath-0.7.1-6.git847cc43.fc27.x86_64 device-mapper-multipath-libs-0.7.1-6.git847cc43.fc27.x86_64
Failed to start Device-Mapper Multipath Device Controller with "service multipathd restart" # service multipathd restart Redirecting to /bin/systemctl restart multipathd.service Job for multipathd.service failed because the control process exited with error code. See "systemctl status multipathd.service" and "journalctl -xe" for details. # systemctl status multipathd.service ● multipathd.service - Device-Mapper Multipath Device Controller Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2017-11-02 07:23:31 EDT; 22s ago Process: 12813 ExecStart=/sbin/multipathd -d -s (code=exited, status=1/FAILURE) Process: 12812 ExecStartPre=/sbin/multipath -A (code=exited, status=1/FAILURE) Process: 12811 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code= Main PID: 12813 (code=exited, status=1/FAILURE) Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Starting Device-Mapper Multipath Device Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipath[12812]: Nov 02 07:23:31 | multipath.conf + Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipath[12812]: Nov 02 07:23:31 | multipath.conf + Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipath[12812]: Nov 02 07:23:31 | error parsing co Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipathd[12813]: process is already running Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipathd[12813]: failed to create pidfile Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Main process exited, Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Failed to start Device-Mapper Multipath Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Unit entered failed Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Failed with result ' # journalctl -xe Nov 02 07:23:00 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Failed with result ' Nov 02 07:23:05 storageqe-16.rhts.eng.bos.redhat.com multipathd[12784]: multipath.conf +8, parsing failed Nov 02 07:23:05 storageqe-16.rhts.eng.bos.redhat.com multipathd[12784]: multipath.conf +9, parsing failed Nov 02 07:23:05 storageqe-16.rhts.eng.bos.redhat.com multipathd[12784]: error parsing config file Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Starting Device-Mapper Multipath Device -- Subject: Unit multipathd.service has begun start-up -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit multipathd.service has begun starting up. Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipath[12812]: Nov 02 07:23:31 | multipath.conf + Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipath[12812]: Nov 02 07:23:31 | multipath.conf + Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipath[12812]: Nov 02 07:23:31 | error parsing co Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipathd[12813]: process is already running Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com multipathd[12813]: failed to create pidfile Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Main process exited, Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: Failed to start Device-Mapper Multipath -- Subject: Unit multipathd.service has failed -- Defined-By: systemd -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit multipathd.service has failed. -- -- The result is failed. Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Unit entered failed Nov 02 07:23:31 storageqe-16.rhts.eng.bos.redhat.com systemd[1]: multipathd.service: Failed with result '
I just did a fresh f27 install and then ran # dnf install device-mapper-multipath # mpathconf --enable --user_friendly_names y --find_multipaths y --with_multipathd y and the service started just fine. Are there some other steps in your reproducer?
Created attachment 1348500 [details] multipath.conf The test creates the following multipath.conf. It seems the service is not able to start with it. -- Unit multipathd.service has begun starting up. Nov 06 10:05:38 localhost.localdomain multipath[12372]: Nov 06 10:05:38 | multipath.conf +6, parsing failed: vendor * Nov 06 10:05:38 localhost.localdomain multipath[12372]: Nov 06 10:05:38 | multipath.conf +7, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipath[12372]: Nov 06 10:05:38 | multipath.conf +17, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipath[12372]: Nov 06 10:05:38 | multipath.conf +21, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipath[12372]: Nov 06 10:05:38 | multipath.conf +25, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipath[12372]: Nov 06 10:05:38 | error parsing config file Nov 06 10:05:38 localhost.localdomain multipathd[12373]: --------start up-------- Nov 06 10:05:38 localhost.localdomain multipathd[12373]: read /etc/multipath.conf Nov 06 10:05:38 localhost.localdomain multipathd[12373]: multipath.conf +6, parsing failed: vendor * Nov 06 10:05:38 localhost.localdomain multipathd[12373]: multipath.conf +7, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipathd[12373]: multipath.conf +17, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipathd[12373]: multipath.conf +21, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipathd[12373]: multipath.conf +25, parsing failed: product * Nov 06 10:05:38 localhost.localdomain multipathd[12373]: error parsing config file Nov 06 10:05:38 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=multipathd comm="systemd" exe="/usr/lib/systemd/systemd" hostna Nov 06 10:05:38 localhost.localdomain systemd[1]: multipathd.service: Main process exited, code=exited, status=1/FAILURE Nov 06 10:05:38 localhost.localdomain systemd[1]: multipathd.service: Failed with result 'exit-code'. Nov 06 10:05:38 localhost.localdomain systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
These are do to a change in the regex parser. "*" was never really a valid regular expression. ".*" is. However, I'm no the fence about whether or not multipathd should startup without a valid config. On one hand, it forces you to fix it. On the other hand, well... it's not running.
Oops. I meant, "I'm ON the fence..."
Thinking about this more, since these previously worked, they should probably not suddenly stop multipath from working at all. A better answer (in the short term at least) is to make multipath print a warning message that they are not valid regular expressions, and continue. But I don't think this configuration issue explains all the multipath startup problems here.
Actually, looking back again, perhaps all the issues are due to the change in the regular expression parsing. Can you try the device-mapper-multipath-0.7.1-8.git847cc43.fc27 package https://kojipkgs.fedoraproject.org/packages/device-mapper-multipath/0.7.1/8.git847cc43.fc27/ It will still accept a string of "*" for the multipath.conf parameters that are regexes, but it will convert it to ".*" and print a warning.
Thanks Ben, it works for me: # multipath -l Nov 08 07:40:53 | Invalid regular expression "*" in multipath.conf. Using ".*" Nov 08 07:40:53 | Invalid regular expression "*" in multipath.conf. Using ".*" mpathb (33333333000001770) dm-0 Linux,scsi_debug size=100M features='1 retain_attached_hw_handler' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active `- 2:0:0:0 sda 8:0 active undef unknown Lin Li, could you also test it? Also, could you update the test to use ".*"?
device-mapper-multipath-0.7.1-8.git847cc43.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ce45cbf41e
So it looks like the issue is that multipathd is already running, but systemd doesn't know about it. There is now a multipathd.socket systemd unit, which autostarts multipathd if someone attempts to communicate with it while it's not running. That could be what's starting multipathd. However, I can't get it to start multipathd without systemd correctly picking up that multipathd is now running. The other possibility is that one of your tests is manually starting multipathd. As of f27, multipathd can now be run either as a regular unix thread, or as a systemd thread (which basically means that the program runs as a regular foreground process, and systemd takes care of making it into a thread). However there is no way to make the multipathd unit file able to detect both ways of starting multipathd. As of f27, it starts multipathd as a systemd thread, and can't detect if multipathd was started outside of systemd. However, only one multipathd thread can be active at a time, so if you manually start mutipathd, you will see exactly this behaviour (systemd failing to start multipathd because the pidfile can't be written). Is it possible that one of your tests is manually running # multipathd If not then what I need is a list of tests that you ran to get it to this state, so I can try to reproduce it, and see how this non-systemd multipathd thread got started.
Hi Ben, st73 boots from SAN which is FCoE boot using multipath. I am not sure whether we can stop multipathd on SAN boot.
If systemd didn't start multipathd, then it can't stop it. You just stop it like any other daemon you start # kill <multipathd_pid> or # killall multipathd or, if you want to use a multipath specific way # multipathd shutdown But more correctly, you should start multipathd via systemd ("service multipathd start" or "systemctl start multipathd.service". they are equivalent) in your tests. As far as SAN boot goes, nothing should exist that can possibly start multipathd before you get the initramfs running. Once the initramfs is running, systemd should be in charge of starting multipathd. And once that happens, everything should work fine. Are you seeing problems on this system immediately after boot?
device-mapper-multipath-0.7.1-8.git847cc43.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.