Bug 2011776

From the journal, the first error I can see is:
```
Oct 05 06:01:40 localhost kernel: Kernel command line: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-4db93642d1d8298df8c1c9b655ba64df137a7572854a8dbc6adc8ce6431a9cef/vmlinuz-4.18.0-305.10.2.el8_4.x86_64 random.trust_cpu=on console=tty0 ignition.platform.id=metal ostree=/ostree/boot.1/rhcos/4db93642d1d8298df8c1c9b655ba64df137a7572854a8dbc6adc8ce6431a9cef/0 rd.multipath=default root=/dev/disk/by-label/dm-mpath-root single
[...]
Oct 05 06:01:45 localhost systemd[1]: Mounting /sysroot...
Oct 05 06:01:45 localhost systemd-journald[737]: Missed 15 kernel messages
Oct 05 06:01:45 localhost kernel: SGI XFS with ACLs, security attributes, quota, no debug enabled
Oct 05 06:01:45 localhost kernel: XFS (dm-5): Mounting V5 Filesystem
Oct 05 06:01:45 localhost systemd[1]: Started CoreOS Trigger Multipath.
Oct 05 06:01:45 localhost systemd-journald[737]: Missed 1 kernel messages
Oct 05 06:01:45 localhost kernel: XFS (dm-5): Ending clean mount
Oct 05 06:01:45 localhost systemd[1]: Mounted /sysroot.
Oct 05 06:01:45 localhost systemd[1]: Starting OSTree Prepare OS/...
Oct 05 06:01:45 localhost ostree-prepare-root[1669]: Resolved OSTree target to: /sysroot/ostree/deploy/rhcos/deploy/57356ce06823a842bbb83e2ec652f7003f373d421589a0d37fa842ba918b170d.0
Oct 05 06:01:45 localhost ostree-prepare-root[1669]: sysroot configured read-only: 1, currently writable: 0
Oct 05 06:01:45 localhost ostree-prepare-root[1669]: ostree-prepare-root: sysroot=readonly currently requires writable / in initramfs
Oct 05 06:01:45 localhost systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
Oct 05 06:01:45 localhost systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
Oct 05 06:01:45 localhost systemd[1]: Failed to start OSTree Prepare OS/.
Oct 05 06:01:45 localhost systemd[1]: ostree-prepare-root.service: Triggering OnFailure= dependencies.
```

However I fear this is just a symptom of something else failing to properly set up the mounts in the initramfs.

Just to be sure: on this same node, can we please boot *without* the 'single' parameter on kernel command-line, and get the full journal from that?

Also, I'm curious about these messages:
```
Oct 05 06:01:44 localhost multipathd[1201]: mpatha: load table [0 1048576000 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 1 1 8:112 1 service-time 0 1 1 8:144 1]
Oct 05 06:01:44 localhost multipathd[1201]: sdh [8:112]: path added to devmap mpatha
Oct 05 06:01:44 localhost multipathd[1201]: mpathb: load table [0 419430400 multipath 1 queue_if_no_path 1 alua 2 1 service-time 0 1 1 8:96 1 service-time 0 1 1 8:128 1]
Oct 05 06:01:44 localhost multipathd[1201]: sdi [8:128]: path added to devmap mpathb
```

How does the underlying multipath environment look like?
Are there multiple exported virtual devices involved? If so, by chance, are there pre-existing filesystems on those devices with conflicting labels (i.e. "root" or "boot")?

Ahh right OK, the issue I think is that the multipath documentation is missing an important bit.

The error here is:

> Oct 05 06:01:45 localhost ostree-prepare-root[1669]: ostree-prepare-root: sysroot=readonly currently requires writable / in initramfs

This is because we're missing `rw` on the kernel cmdline. Can you try the installation again with `--append-karg rw` added?

Opened a patch to fix the docs.

If that's the only issue, then let's transform this into a docs bug to track the doc fix.

Thanks! I have asked to test the paramter. I will let you know as soon as I have the results.

I confirm that "--append-karg rw" fixed the issue.

Ack thanks for the update. Let's move this to the documentation component.

Docs team: note I have already submitted a patch to fix this: https://github.com/openshift/openshift-docs/pull/37181
It'll need to be backported to 4.8.

Thanks for opening the PR, Jonathan. The PR lgtm. Once it passes QE review, I'll get this merged and backported to 4.8+.

PR has been SME and QE reviewed/acked. PR merged and cherrypicked to 4.8+. Moving to RELEASE_PENDING while waiting for it to be live at https://docs.openshift.com/container-platform/4.9/installing/installing_bare_metal/installing-bare-metal.html#rhcos-enabling-multipath_installing-bare-metal

Docs are live on 4.8: https://docs.openshift.com/container-platform/4.8/installing/installing_bare_metal/installing-bare-metal.html
4.9: https://docs.openshift.com/container-platform/4.9/installing/installing_bare_metal/installing-bare-metal.html
Will be live on 4.10 pending GA.

Closing BZ