Description of problem: When trying to install Fedora 29 Beta RC 1.1 on a firmware RAID device, Anaconda did not see it and I could not install to it. Using "lsblk", I realized that even the lsblk cannot see the disks as RAID0, but only as standalone disks (sdb and sdc). Version-Release number of selected component (if applicable): * Fedora 29 Beta RC 1.1 * kernel 4.18.5-300 How reproducible: Always Steps to Reproduce: 1. Boot to anaconda on the system 2. The disks will not be available to anaconda. 3. Switch to console, the disks will be seen as standalone disks by lsblk. Actual results: Firmware RAID is not visible and available for installation. Expected results: Firmware RAID is visible and available for installation. Additional info: * No problem on Fedora 28. * The firmware RAID used is some Intel made RAID used on the MSI Z77A-G45 GAMING motherboard
Proposed as a Blocker for 29-beta by Fedora user lruzicka using the blocker tracking app because: I am proposing this as a blocker because it violates the Beta Release criteria in: The installer must be able to detect and install to hardware or firmware RAID storage devices.
Created attachment 1482655 [details] lsblk output on Fedora 29
Created attachment 1482656 [details] lsblk output on Fedora 28
Created attachment 1482657 [details] mdstat output on Fedora 28 There is no mdstat output from Fedora 29, cause the ls /proc/mdstat command returned nothing.
Created attachment 1482658 [details] Anaconda logs from Fedora 29
It is possible to install on the RAID after manually starting it using "mdadm --assemble --scan" and rescanning disks in anaconda. Unfortunately system doesn't boot from the raid, but it might be a different problem.
if it's not being assembled automatically during boot of the installer image, perhaps it is also not being assembled automatically during boot of the installed system? What happens if you add 'rd.auto' to the installer boot params?
The identification of these things is done by libblkid which is part of util-linux.
I'm seeing this in storage.log: 12:11:43,298 ERR blivet: failed to determine name for the md array 427eb9a3-eb4f-6e9f-ff53-c0844347414d 12:11:43,444 ERR blivet: failed to determine name for the md array 427eb9a3-eb4f-6e9f-ff53-c0844347414d
That may well be our problem, because, here's the relevant code: =========== class MDFormatPopulator(FormatPopulator): priority = 100 _type_specifier = "mdmember" ... def run(self): ... if md_name: log.info("using name %s for md array containing member %s", md_name, self.device.name) else: log.error("failed to determine name for the md array %s", (md_uuid or "unknown")) return array_type = MDRaidArrayDevice try: if raid.get_raid_level(md_level) is raid.Container and \ getattr(self.device.format, "biosraid", False): array_type = MDContainerDevice except raid.RaidError as e: log.error("failed to create md array: %s", e) return try: md_array = array_type( md_name, level=md_level, member_devices=md_devices, uuid=md_uuid, metadata_version=md_metadata, exists=True ) except (ValueError, DeviceError) as e: log.error("failed to create md array: %s", e) return md_array.update_sysfs_path() md_array.parents.append(self.device) self._devicetree._add_device(md_array) if md_array.status: array_info = udev.get_device(md_array.sysfs_path) if not array_info: log.error("failed to get udev data for %s", md_array.name) return self._devicetree.handle_device(array_info, update_orig_fmt=True) ============= Note that when logging this error ("failed to determine name") the method immediately returns; it doesn't do all the stuff that should happen next, all the initialization of an MDRaidArrayDevice and adding it to things and 'handling' it and so on. i.e. it sure looks a lot like we're just failing part way through the intended initialization of the device, here.
adamw and I discussed on irc, guessing an mdadm format change, reassigning to storaged folks to determine if it should get fixed in blivet or libblockdev.
So, I think I know why that error message is in the logs, now, at least. We bail if there's no md_name. There will be no md_name if we don't get it from udev (which we won't, if the device isn't active already), and there's no md_path. md_path is md_info.device, if that exists. md_info.device is from libblockdev's crazy mdadm parsing function, bd_md_examine , and works like this: argv[2] = "--brief"; success = bd_utils_exec_and_capture_output (argv, NULL, &output, error); if (!success) { /* error is already populated */ bd_md_examine_data_free (ret); return FALSE; } /* try to find the "ARRAY /dev/md/something" pair in the output */ output_fields = g_strsplit_set (output, " \n", 0); for (i=0; !found_array_line && (i < g_strv_length (output_fields) - 1); i++) if (g_strcmp0 (output_fields[i], "ARRAY") == 0) { found_array_line = TRUE; if (g_str_has_prefix (output_fields[i+1], "/dev/md/")) { ret->device = g_strdup (output_fields[i+1]); } else { ret->device = NULL; } } if (!found_array_line) ret->device = NULL; g_strfreev (output_fields); translated, that basically means: Run mdadm --examine --brief (device). Look for the first line in the output that starts with the string 'ARRAY '. if the next bit of the line looks like '/dev/md/(something) ', then that is the 'device'. If it doesn't, then the device is nothing. The output we actually get for the two devices we do this on - /dev/sdb and /dev/sdc - looks like this (it's identical both times): ARRAY metadata=imsm UUID=427eb9a3:eb4f6e9f:ff53c084:4347414d ARRAY /dev/md/Volume0 container=427eb9a3:eb4f6e9f:ff53c084:4347414d member=0 UUID=fbf70ce1:cb344bf6:91f2c113:4c303040 note the *first* "ARRAY" line in the output starts "ARRAY metadata=imsm", so we are going to wind up with the 'device' as nothing. The code intentionally does *not* look at the second line and set the 'device' to '/dev/md/Volume0'. However...I'm no longer 100% convinced this is the problem, as none of this looks particularly new. The whole chunk in blivet hasn't changed significantly since 2015. libblockdev was changed to intentionally consider only the *first* ARRAY line in 2016 - that change should have been in many Fedora releases by now, so if it's the problem, this hasn't worked for several releases... Now I'm wondering if the bail-out from MDFormatPopulator.run() for *these particular devices* (sdb and sdc) isn't as big of a deal as I thought, and we should wind up successfully activating the RAID set another way later somehow, but don't...
Can confirm exactly the same bug on my desktop, with P8P67 Deluxe motherboard. Set up a couple of spare disks in a RAID-0 set. Booted F28 Final installer, it sees the set just fine. Booted F29 Beta 1.1 installer, it doesn't see it at all. No /dev/md* devices are present. Does see it if I manually assemble and rescan disks. All exactly as Lukas reported.
rd.auto, rd.md, rd.md=1 don't seem to help (tried 'em all).
Poking around a bit and looking at timestamps, I'm fairly sure anaconda expects something else to activate the RAID set before it even starts up. This is what happens on F28 - I have the kernel logging a 'capacity change' for md126 before anaconda even runs. On F29 that does not happen. I *think* the 'something' may be udev, and we may be in a udev bug here.
Hah, and the udev rules for mdraid device assembly are provided by mdadm, bringing us back to mdadm... I see this commit as a potential suspect: https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=3a77acd7170199adc690332ded37c41f067c720e Also these: https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=cd6cbb08c458cee07acb1d854e04532b29ec87bf https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/?id=dd180cb136d6b2193a58ea0de23b8a7942ca6f36 which *set* the variable that 3a77acd checks for. Will try and dig into that a bit more later, but I have to go out right now.
Ug, no, that's probably not it. Earlier in that file: # Don't process any events if anaconda is running as anaconda brings up # raid devices manually ENV{ANACONDA}=="?*", GOTO="md_inc_end" so, presuming that happens on F28 too...I dunno. I still have no idea how the hell the array actually gets brought up on F28, the logs aren't very verbose, it just suddenly...shows up...
There should be no storaged in Fedora (udisks2 and storaged merged and it's been decided to name the result "udisks2").
Created attachment 1482948 [details] debug journal from f28 (where md devices show up) Here's the contents of the journal when booting F28 Final on a system with firmware RAID. This is with kernel args 'debug systemd.log_level=debug udev.log_priority=debug rd.udev.log_priority=debug'. Note that almost as soon as "dev-sda.device: Changed dead -> plugged" happens, md126 shows up: Sep 13 07:33:41 localhost systemd[1]: dev-disk-by\x2dpath-pci\x2d0000:00:1f.2\x2data\x2d1.device: Changed dead -> plugged Sep 13 07:33:41 localhost systemd[1]: dev-disk-by\x2did-wwn\x2d0x50025388a0a3c17f.device: Changed dead -> plugged Sep 13 07:33:41 localhost systemd[1]: dev-disk-by\x2did-ata\x2dSamsung_SSD_850_PRO_128GB_S24ZNSAG414951M.device: Changed dead -> plugged Sep 13 07:33:41 localhost systemd[1]: dev-sda.device: Changed dead -> plugged Sep 13 07:33:41 localhost systemd[1]: sys-devices-pci0000:00-0000:00:1f.2-ata1-host0-target0:0:0-0:0:0:0-block-sda.device: Changed dead -> plugged Sep 13 07:33:41 localhost kernel: md126: detected capacity change from 0 to 256066453504 Sep 13 07:33:41 localhost systemd[1]: dev-md-Volume0_0.device: Changed dead -> plugged
Created attachment 1482949 [details] debug journal from f29 (where md devices do not show up) Same logs from f29. udev seems to log a lot more verbosely now, for some reason, but it's clear that things are different. 'md126' does not appear in the logs at all. dev-sda.device and dev-sdb.device still show up, but this does not associate with md126 appearing any more.
I don't think we're in udisks2 here, purely for timing reasons: udisks2.service starts after the appearance of md126 (on f28). The only thing I can think of that we're in is udev, even though the udev rules for activating raid devices are _supposed_ to be disabled for anaconda environments (see above).
(In reply to Adam Williamson from comment #17) > Ug, no, that's probably not it. Earlier in that file: > > # Don't process any events if anaconda is running as anaconda brings up > # raid devices manually > ENV{ANACONDA}=="?*", GOTO="md_inc_end" > > so, presuming that happens on F28 too...I dunno. I still have no idea how > the hell the array actually gets brought up on F28, the logs aren't very > verbose, it just suddenly...shows up... We stopped setting that udev variable 2 years ago and it was set by anaconda so it shouldn't affect starting the array during boot -- https://github.com/rhinstaller/anaconda/commit/73af7adeaa72ff2c3d2427784ac5e6cc36839e6a
(In reply to Adam Williamson from comment #16) > Hah, and the udev rules for mdraid device assembly are provided by mdadm, > bringing us back to mdadm... > > I see this commit as a potential suspect: > > https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/commit/ > ?id=3a77acd7170199adc690332ded37c41f067c720e > I looks promising but we completely skip this udev rule file on Fedora -- https://src.fedoraproject.org/rpms/mdadm/blob/master/f/mdadm-3.3-udev.patch -- and the same patch is also on Fedora 28 so it shouldn't cause our problem.
I really don't know what is causing this problem, but it definitely isn't a udisks problem. Based on the fact that the array was started during boot on F28 and isn't started in F29 I also think this is not installer (blivet or libblockdev) related. I'm switching this to mdadm -- it ships the udev rules that (I think) should start the array.
Further notes from the Enduring Mystery Of What The Hell Causes /dev/md126 To Appear: * booting with `rd.break=cleanup`, it's not there, so I don't think dracut is doing it. * I don't see "Autodetecting RAID arrays" in either log, so I don't think kernel autodetection is doing it. (Which makes sense, as kernel autodetection appears to be set to only run when the mdraid bits are built-in, not when they're modules). I'll keep digging.
OK, so I've made a bunch of progress here, and the culprit looks to be...multipath! Bet you didn't see that coming... I made an updates.img that replaces mdadm with a little wrapper that logs the call time and call args, then calls the real mdadm. It showed me that two early calls happen on f28 which do *not* happen on f29: Called at 2018-09-13 19:42:16.471604998+00:00 with -I /dev/sda Called at 2018-09-13 19:42:16.471605010+00:00 with -I /dev/sdb Armed with this info, I went looking for stuff that would do 'mdadm -I /dev/sda'. And lo, /usr/lib/udev.rules.d/65-md-incremental.rules has something that very much fits the bill: SUBSYSTEM=="block", ACTION=="add", ENV{ID_FS_TYPE}=="isw_raid_member", \ RUN+="/sbin/mdadm -I $env{DEVNAME}" if you do 'blkid /dev/sda' on one of these fwraid member disks on f28 or f29, it says: /dev/sda: TYPE="isw_raid_member" so that all adds up very nicely. So the next question was, why is this udev rule apparently kicking in on f28 but not on f29? The obvious suspect is a couple of checks at the top of the file that basically skip the whole thing if some conditions are met: ====== # Don't process any events if anaconda is running as anaconda brings up # raid devices manually ENV{ANACONDA}=="?*", GOTO="md_end" # Also don't process disks that are slated to be a multipath device ENV{DM_MULTIPATH_DEVICE_PATH}=="?*", GOTO="md_end" ====== So I started playing with those. And...it seems to be the second one! I put a copy of the file in my updates.img and played with it. If I comment out the 'ANACONDA' check, we still get no /dev/md126, and anaconda still can't see the array. If we comment out the *second* check, suddenly I get the '-I /dev/sda' and '-I /dev/sdb' calls logged in my mdadm call log, /dev/md126 shows up, and anaconda can see the array. So...it seems like the multipath udev rules are setting DM_MULTIPATH_DEVICE_PATH for these disks even though they aren't actually multipath devices at all, and that is causing the md-incremental rules not to run and the array not to get set up on boot. Now...well, now I'm going to go have some lunch. *Then* I will start poking about in device-mapper-multipath (which owns the multipath udev rules) to see what it's doing wrong.
cmurf points out that in f28 journal log with udev debugging, we get this: Sep 13 07:33:35 localhost systemd-udevd[860]: '/sbin/multipath -u sdb'(out) 'sdb is not a valid multipath device path' Sep 13 07:33:35 localhost systemd-udevd[860]: Process '/sbin/multipath -u sdb' failed with exit code 1. but in f29 log, we get this: Sep 13 07:53:55 localhost systemd-udevd[863]: '/sbin/multipath -u sda'(out) 'DM_MULTIPATH_DEVICE_PATH="0"' Sep 13 07:53:55 localhost systemd-udevd[863]: Process '/sbin/multipath -u sda' succeeded. The string 'DM_MULTIPATH_DEVICE_PATH' does not appear at all in the f28 logs, but in the f29 logs it shows up as being set to 0 multiple times. Basically it looks like in f28, for devices that aren't valid multipath devices, DM_MULTIPATH_DEVICE_PATH is not set *at all*; but in f29, it is *explicitly set to 0*. Now, remember, the 65-md-incremental.rules check looks like this: ENV{DM_MULTIPATH_DEVICE_PATH}=="?*", GOTO="md_end" Note that "?*". udev man page states: "*" Matches zero or more characters. "?" Matches any single character. So this matches *any 1 or more characters*. It's not a check for truthiness or anything like that. So, I think this check will actually *match* on DM_MULTIPATH_DEVICE_PATH being explicitly set to 0, as opposed to not being set at all. After all, 0 is a character. I think we should probably change the 65-md-incremental.rules check to either this: ENV{DM_MULTIPATH_DEVICE_PATH}=="[12]", GOTO="md_end" Or this: ENV{DM_MULTIPATH_DEVICE_PATH}=="1", GOTO="md_end" as it seems like from the multipath rule comments and stuff, the value '1' means "definitely a multipath device" and the value '2' means "maybe a multipath device".
Note two other rules files do this 'right': 68-del-part-nodes.rules:ENV{DM_MULTIPATH_DEVICE_PATH}=="1", GOTO="del_part_nodes" 69-dm-lvm-metad.rules:ENV{DM_MULTIPATH_DEVICE_PATH}=="1", GOTO="lvm_end" but one other does it 'wrong': 80-udisks2.rules:ENV{DM_MULTIPATH_DEVICE_PATH}=="?*", GOTO="udisks_probe_end"
mdadm-4.1-rc2.0.2.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-1350db8c28
To test my fix, try: inst.updates=https://www.happyassassin.net/updates/1628192.0.img That updates image contains the same change as the update. It would be good if lruzicka can try this on his affected system, and also if lnie can try it on the multipath test system, to ensure this doesn't somehow break multipath installs.
Excellent debugging and sorting of this. It sounds to me like the LVM people changed the multipath udev behavior without notifying anyone. Given that multiple other people checked the multipath settings, this probably should have been coordinated. There should be Requires: and Conflicts: tags in all of the involved packages.
(In reply to Doug Ledford from comment #31) > There should be Requires: and Conflicts: tags in all of the involved packages. To be more clear, I mean that part of the resolution of this issue should include getting the proper Requires: and Conflicts: tags in the relevant packages.
OK, so I can point to the relevant changes on the multipath-tools side, and this adds an interesting new wrinkle: https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=b729b8f3a21fe6867165b2533496ebdfbc493263 https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=7e867718d1d0133e8ed34835320451f93358f2f9 The first one basically changed how DM_MULTIPATH_DEVICE_PATH gets set. Previous to that commit, 'multipath -u (somedevice)' succeeded (exited 0) if the device was definitely a multipath one, and failed (exited 1) if it wasn't. On stdout it printed either: "(somedevice) is a valid multipath device path" or: "(somedevice) is not a valid multipath device path" The udev rule looked like this: ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \ PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -u %k", \ ENV{DM_MULTIPATH_DEVICE_PATH}="1", ENV{ID_FS_TYPE}="mpath_member", \ ENV{SYSTEMD_READY}="0" which basically means "run multipath -u (somedevice). If it succeeds, set DM_MULTIPATH_DEVICE_PATH to 1, ID_FS_TYPE to 'mpath_member' and SYSTEMD_READY to '0'. If the command does *not* succeed, do nothing at all." After that commit, things were changed around so that 'multipath -u (somedevice)' prints either: DM_MULTIPATH_DEVICE_PATH="0" or: DM_MULTIPATH_DEVICE_PATH="1" to stdout, and the udev rule uses IMPORT: IMPORT{program}="$env{MPATH_SBIN_PATH}/multipath -u %k" IMPORT basically means "run this command, and expect its output to be lines of "KEY=value" pairs; import those pairs as udev env vars". So, that was the change that made us go from DM_MULTIPATH_DEVICE_PATH not being set to it being set to 0 in the case where the device isn't a multipath one. A way to keep the old behaviour while still using the new approach would be to just have the command not print *anything* to stdout when the device isn't a valid one, instead of printing DM_MULTIPATH_DEVICE_PATH="0". Having it print DM_MULTIPATH_DEVICE_PATH="" might also work, not sure. The second change is interesting because it's the one that introduces the possible '2' value, meaning "this might be a multipath device". I did not not realize until now that this value was *new*, as part of this "smart mode" commit. So, it's *possible* all those things that only checked for the value '1' before should now think about whether they should also check for '2'. But I'm not 100% sure about that, because this 'smart' mechanism is a bit hard to understand and I'm not 100% sure if *other* rules will ever see the value '2', or if only the multipath rule itself ever actually sees that value and other rules always only see 0, 1 or (not set). I'm going to mail the relevant list (dm-devel@redhat) about this and see whether folks think we should update things that check the DM_MULTIPATH_DEVICE_PATH values, or change multipath so it behaves more like it used to, or what. For now the update I've submitted fixes at least the worst known case here.
Discussed at 2018-09-13 Fedora 29 Beta Go/No-Go meeting, acting as a blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2018-09-13/f29-beta-go_no_go-meeting.2018-09-13-17.00.html . It was generally agreed that this violated the release criteria as written, but there was substantial support - 7 votes to 1 - for adjusting the criteria to either move this requirement to Final, or drop it entirely. On the basis that one of these adjustments will be proposed and implemented, this was rejected as a Beta blocker. It is accepted as a freeze exception issue, as obviously we would like this to work if possible.
I think you could just change the rules in 65-md-incremental.rules and 80-udisks2.rules to switch their tests from ="?*" to ="1" which would work with both the old and new method of multipath settings. I think we have to ignore 2. The whole "maybe a multipath device" is utter crap. We can't ignore it because it *might* be a multipath device, because we might never find another path, multipath might never take it over, and then we would have already skipped it and things would then be off the rails. So, either multipath takes it or it doesn't, and if they don't, we have to treat it like a bare drive. So, just change the test to check for 1, otherwise proceed, and that should fix things up properly.
Yeah, that's what the update I sent out did. I wrote a patch for udisks2 but didn't send it yet while waiting for feedback on whether the multipath behaviour should change, but it sounds like Martin thinks the answer to that is "no", so I'll go ahead and change it. Martin also says it is indeed the case that the value '2' is effectively "internal" - no other rules file should ever see it. So we don't have to worry about that.
(In reply to Doug Ledford from comment #35) > I think you could just change the rules in 65-md-incremental.rules and > 80-udisks2.rules to switch their tests from ="?*" to ="1" which would work > with both the old and new method of multipath settings. I think we have to > ignore 2. The whole "maybe a multipath device" is utter crap. We can't > ignore it because it *might* be a multipath device, because we might never > find another path, multipath might never take it over, and then we would > have already skipped it and things would then be off the rails. So, either > multipath takes it or it doesn't, and if they don't, we have to treat it > like a bare drive. So, just change the test to check for 1, otherwise > proceed, and that should fix things up properly. In smart mode, when it sees a maybe (2) device, the multipath udev rules set a systemd timer (for 1 second if the maybe device doesn't have a multipath configuration for it, and for a user configurable timeout if it does). If multipath can't definitely claim the device before that timeout, it will switch back to unclaimed (0), and only claim that device in the future if a multipath device gets created on top of it. So, you unless systemd fails to trigger the uevent after the timeout, you will never get left with a path sitting in maybe limbo for more than a few seconds. Also, in fedora and RHEL8, multipath is not set up to use smart mode. So, unless a user manually changes that, you should never see a maybe device at all. And even if a user does set this, you will never see a maybe device outside of the multipath rules. They simply look like claimed (1) devices, so ENV{DM_MULTIPATH_DEVICE_PATH}=="1" is correct.
(In reply to Doug Ledford from comment #32) > (In reply to Doug Ledford from comment #31) > > There should be Requires: and Conflicts: tags in all of the involved packages. > > To be more clear, I mean that part of the resolution of this issue should > include getting the proper Requires: and Conflicts: tags in the relevant > packages. Sorry about all the confusion this caused. The first device-mapper-multipath packages that have the changed udev variables are: device-mapper-multipath-0.7.7-1.fc29 Once there are updated builds for the other packages, I will update the device-mapper-multipath.spec file.
I've already done an updated build of mdadm for Rawhide and F29. udisks PR is here: https://github.com/storaged-project/udisks/pull/576 I'll backport the patch to the package and do builds for Rawhide and F29 for that too. I'm assuming this change won't be backported to F28 and earlier so we don't need to update mdadm and udisks2 for those releases?
mdadm-4.1-rc2.0.2.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-1350db8c28
(In reply to Adam Williamson from comment #39) > I've already done an updated build of mdadm for Rawhide and F29. udisks PR > is here: > > https://github.com/storaged-project/udisks/pull/576 > > I'll backport the patch to the package and do builds for Rawhide and F29 for > that too. I'm assuming this change won't be backported to F28 and earlier so > we don't need to update mdadm and udisks2 for those releases? No, f28 should be fine.
mdadm-4.1-rc2.0.2.fc29 udisks2-2.8.0-2.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-1350db8c28
mdadm-4.1-rc2.0.2.fc29, udisks2-2.8.0-2.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
The device-mapper-multipath update to conflict with the older versions of these packages is device-mapper-multipath-0.7.7-5.gitef6d98b
Cool, thanks. Since the update has been pushed stable, and as this generally shouldn't be a problem on upgrade (as the bug doesn't happen if /etc/multipath.conf exists), I don't think we need to worry about getting that in the Beta, it can just follow the normal process and go out after.