Bug 2061952

+++ This bug was initially created as a clone of Bug #2040195 +++

As the Assisted Installer Team, when testing RHCOS 410.84.202201102308-0 we are observing the following behaviour of a systemd mount unit

* `"enabled":true` in the ignition
* `disabled` and `inactive` in the booted OS

The unit definition itself seems to be correct, as manually starting it via `systemctl start` does not throw any errors and makes the mount to appear correctly.

The part of ignition defining the unit

```
{"units":[{"contents":"\n[Unit]\nAfter=systemd-tmpfiles-setup.service\n[Mount]\nWhere=/etc/NetworkManager/system-connections-merged\nWhat=overlay\nType=overlay\nOptions=lowerdir=/etc/NetworkManager/system-connections,upperdir=/run/nm-system-connections,workdir=/run/nm-system-connections-work\n[Install]\nWantedBy=multi-user.target\n","enabled":true,"name":"etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"}]}
```

and respective systemd unit inside the OS

```
[root@extraworker-0 NetworkManager]# systemctl status etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount 
● etc-NetworkManager-system\x2dconnections\x2dmerged.mount - /etc/NetworkManager/system-connections-merged
   Loaded: loaded (/etc/systemd/system/etc-NetworkManager-system\x2dconnections\x2dmerged.mount; disabled; vendor preset: disabled)
   Active: inactive (dead)
    Where: /etc/NetworkManager/system-connections-merged
     What: overlay
```

From the journal it can be seen that the unit was parsed and allegedly enabled

```
[root@extraworker-0 NetworkManager]# journalctl  | grep merged | grep -i network
Jan 12 13:45:55 localhost ignition[1049]: INFO     : files: op(7e): [started]  processing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:55 localhost ignition[1049]: INFO     : files: op(7e): op(7f): [started]  writing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount" at "/sysroot/etc/systemd/system/etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:55 localhost ignition[1049]: INFO     : files: op(7e): op(7f): [finished] writing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount" at "/sysroot/etc/systemd/system/etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:55 localhost ignition[1049]: INFO     : files: op(7e): [finished] processing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:55 localhost ignition[1049]: INFO     : files: op(a3): [started]  setting preset to enabled for "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:55 localhost ignition[1049]: INFO     : files: op(a3): [finished] setting preset to enabled for "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:56 extraworker-0 ignition[1049]: INFO     : files: op(7e): [started]  processing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:56 extraworker-0 ignition[1049]: INFO     : files: op(7e): op(7f): [started]  writing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount" at "/sysroot/etc/systemd/system/etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:56 extraworker-0 ignition[1049]: INFO     : files: op(7e): op(7f): [finished] writing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount" at "/sysroot/etc/systemd/system/etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:56 extraworker-0 ignition[1049]: INFO     : files: op(7e): [finished] processing unit "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:56 extraworker-0 ignition[1049]: INFO     : files: op(a3): [started]  setting preset to enabled for "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
Jan 12 13:45:56 extraworker-0 ignition[1049]: INFO     : files: op(a3): [finished] setting preset to enabled for "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
```

--- Additional comment from Mat Kowalski on 2022-01-13 08:55:22 UTC ---

Ignition file - http://pastebin.test.redhat.com/1021061
Slack thread in #forum-coreos - https://coreos.slack.com/archives/C999USB0D/p1642002024148700

--- Additional comment from Benjamin Gilbert on 2022-01-13 20:22:42 UTC ---

Complete repro:

{
  "ignition": {
    "version": "3.3.0"
  },
  "storage": {
    "directories": [
      {
        "path": "/etc/NetworkManager/system-connections-merged"
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "contents": "[Mount]\nWhere=/etc/NetworkManager/system-connections-merged\nWhat=tmpfs\nType=tmpfs\n[Install]\nWantedBy=multi-user.target\n",
        "enabled": true,
        "name": "etc-NetworkManager-system\\x2dconnections\\x2dmerged.mount"
      }
    ]
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa ..."
        ]
      }
    ]
  }
}

--- Additional comment from Benjamin Gilbert on 2022-01-13 20:31:15 UTC ---

systemd in RHEL 8.4 does not correctly parse singly-escaped unit names in preset files, which are used by Ignition for unit enablement.  This is fixed in RHEL 8.5, and backport to 8.4 was requested in https://bugzilla.redhat.com/show_bug.cgi?id=1952686#c9.

As a workaround, you could rename the mountpoint (and thus the mount unit) to something that doesn't include dashes.

--- Additional comment from Mat Kowalski on 2022-01-14 09:27:43 UTC ---

Thanks a lot! Given that we have implemented a solution not depending on this mountpoint, this has no urgency for our team any more. Feel free to close it as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1952686 or any other way you handle it internally

--- Additional comment from Benjamin Gilbert on 2022-01-14 17:00:25 UTC ---

Great.  Will leave this open as a tracking bug for now.

--- Additional comment from Micah Abbott on 2022-01-20 14:26:48 UTC ---

Updating the DependsOn to point to the 8.4.z backport BZ#2042439

--- Additional comment from Micah Abbott on 2022-03-08 19:02:08 UTC ---

Need to target this as 4.11, so we can track the 8.4.z fix in RHCOS 4.10.z appropriately

I've updated the Depends On field to point to the 8.5 version of the BZ, to indicate that RHCOS 4.11 should have the fix already included when it moves to RHEL 8.6