Affecting OpenShift baremetal IPI.

Some properties (such as ipv4.dhcp-client-id) support a global default, meaning that when the value in the connection profile is not set, the global default is used instead. This doesn't mean that the value in the profile gets updated; if you query the connection profile it still displays an empty value.

Unfortunately at the moment there isn't an easy way to show the effective value of a property that uses a global default.

Your script could notice that the value of ipv4.dhcp-client-id is empty in the profile and then it could try to parse the global default from the output of "NetworkManager --print-config", but that works only for very simple cases, because the script would need to understand the various "match*" lines and their precedence. So, I don't consider this feasible in the general case.

Is there another mechanism for getting DHCP client IDs used for the current leases?

> Is there another mechanism for getting DHCP client IDs used for the current leases?

unless you hardcode a MAC address in the profile's `ipv4.dhcp-client-id`, the actually used client-id depends on the interface and the moment of activation. That's the case with `ipv4.dhcp-client-id="mac"`, with `ipv4.dhcp-client-id="stable"` or with leaving it unset to fallback to the global connection default from NetworkManager.conf.

The first solution is to just explicitly hardcode the client-id in the profile. That seems to be what you want anyway.

If you clone a profile that has the client-id set to something else (one of the special keywords like "mac", "stable" or unset to mean global default), you cannot know the used client-id, unless you activate the profile. Because it depends on the device (with `ipv4.dhcp-client-id="stable" connection.stable-id="${RANDOM}"` it is even randomized at the moment of activation).

What we should add, is to expose the used client-id/DUID together with the lease information. We already expose the lease information on D-Bus, nmcli and /run/NetworkManager/devices. It would be simple and good to have also the client-id/DUID there. Then you could check at a particular moment which client ID was used. It's not clear to me what you are going to do with that information though.




> To do this we need to copy nmcli settings from the bond0 device to the OVS br-ex device.
> ...
> Because of this issue we fail to copy the bond0 dhcp settings.

Hm? You copy the profile. And the profile does not specify the client-id.

Maybe that is your real problem? If you want a fixed client-id per-profile, why do you write it to NetworkManager.conf, instead of configuring it in the profile?

(In reply to Thomas Haller from comment #4)
> > To do this we need to copy nmcli settings from the bond0 device to the OVS br-ex device.
> > ...
> > Because of this issue we fail to copy the bond0 dhcp settings.
> 
> Hm? You copy the profile. And the profile does not specify the client-id.
> 
> Maybe that is your real problem? If you want a fixed client-id per-profile,
> why do you write it to NetworkManager.conf, instead of configuring it in the
> profile?

We generate profiles via template for all worker and control-plane nodes before installation so we don't know each node's MAC at profile creation time, we only know primary interface, eth0, etc.  We use systemd oneshot to run a script on boot to write /etc/NetworkManager/conf.d/99-client-id.conf


https://access.redhat.com/solutions/6747451

{% macro set_dhcp_ip(bond, interfaces) -%}
#!/usr/bin/bash
cat <<EOF > /etc/NetworkManager/conf.d/99-client-id.conf
[connection-{{ bond }}]
match-device=interface-name:={{ bond }},interface-name:=br-ex
ipv4.dhcp-client-id=01:$(cat /sys/class/net/{{ interfaces | first }}/address)
ipv6.dhcp-duid=00:03:00:01:$(cat /sys/class/net/{{ interfaces | first }}/address)
EOF
{%- endmacro %}

Maybe if there was a nm-settings macro that could set lookup an arbitrary interface hwaddr or perm_hwaddr.  We also need to set IAID.


FYI DHCPv6 DUID doesn't really work in our use case because of dnsmasq DHCPv6 reservation issues when IAID changes.

(In reply to Ross Brattain from comment #5)

> FYI DHCPv6 DUID doesn't really work in our use case because of dnsmasq
> DHCPv6 reservation issues when IAID changes.

In theory we might need something like:

ipv6.dhcp-iaid=$(awk -Wposix -F: '{ printf "%d\n",  "0x" $4 $5 $6}' /sys/class/net/{{ interfaces | first }}/address)

Untested.

What in the meantime happened, is that NetworkManager exposes the used client-id/DUID on D-Bus, `nmcli -f all device show eth0` and in `/run/NetworkManager/devices/$IFINDEX`. This information is now exposed along the lease information.

This is in
- rhel-8.8+, upstream 1.40.10+ (https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/82d5f0096115f15ee63eab951d04785e6f3c9970)
- rhel-9.2+, upstream 1.41.8+ (https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/831b8f8e7e7c2660570bf36e3987f68047d1a91b)

However, what is not exposed is the IAID. Unclear whether that is necessary. For now, that's not there.

However, whether that information is in any way useful to your scripts, is unclear. To me, it doesn't seem something you should use for that purpose. Don't let NetworkManager start activating with a wrong configuration, to then asked which actual configuration was used, and then modify the profile. Instead, modify the profile *first* with the desired configuration.


(In reply to Ross Brattain from comment #6)
> (In reply to Ross Brattain from comment #5)
> 
> > FYI DHCPv6 DUID doesn't really work in our use case because of dnsmasq
> > DHCPv6 reservation issues when IAID changes.
> 
> In theory we might need something like:
> 
> ipv6.dhcp-iaid=$(awk -Wposix -F: '{ printf "%d\n",  "0x" $4 $5 $6}'
> /sys/class/net/{{ interfaces | first }}/address)

"ipv6.dhcp-iaid" is again a default value in NetworkManager.conf. For this usage, I think it's a bad idea to write default values there. Set the per-profile value. But yes, like you set the client-id/DUID, you can set the iaid.

---

Why not pre-deploy all necessary connection profiles in /etc/NetworkManager/system-connections?
Ensure that those profiles have `dhcp-client-id=mac` set (something you can match against), do a

  sed "s/^dhcp-client-id=mac\$/dhcp-client-id=01:$(get_mac)/" /etc/NetworkManager/system-connections/{files,...}

In the setup script. Make sure this script is running before NetworkManager starts.


Is there a problem with doing that?


---

I don't think there is anything else to do about this rhbz. Or is there a suggestion?

I cloned this bug as bug 2169869, which is about exposing the DHCP information on the API. Including the ipv6.dhcp-iaid.

The remainder of this very bug is about how to improve the reporter's case. It's not clear to me what to do about this. 

The scripts from comment 0 write global defaults for
```
  [connection-bond0]
  match-device=interface-name:=bond0,interface-name:=br-ex
```
That seems not good to me. Who creates those profiles for bond0/br-ex? Whoever it is, they should explicitly set the desired client-id/duid, when creating those profiles. Likewise, https://access.redhat.com/solutions/6747451 suggesting to set a per-profile value to hard-coded global default seems not the best choice.

If you don't set the values explicitly in the profile, then the actually used value gets generated. You cannot know it by looking at the profile alone, because the generated value only happens when the device activates (and it might differ every time, for example with "connection.stable-id=${RANDOM} ipv4.dhcp-client-id=stable"). Sure, the new API from bug 2169869 will allow you to ask for the used value at any moment, but it seems not right to persist that value back to the profile.

I think you should ensure that the profiles you care about to have a fixed client-id to being with. Set the client-id explicitly via a first boot script, or when creating the profiles. If the profile doesn't choose a fixed value, then that is a user's choice, no need to "work around" that. If the user (or whoever created the profile) is not happy about leaving the client-id generated based on the device, then they must explicitly choose one.

Having said that, you are of course welcome to script whatever works best for you...

For OpenShift we use MachineConfigs that contain RHCOS ignition settings.  We don't set per-machine ignition configs because that doesn't scale, so we can't specify individual unique client-ids.  We have one config for all the machines in a given role.

I guess we could modify the systemd oneshot to `nmcli con mod 802-3-ethernet.cloned-mac-address` instead of the `match-device`.  We have a whole shell script `ovs-configure.sh` that clones the bond0 and adds it to the OVS br-ex bridge, so we can do extra stuff.

CC @jcaamano

We want to get rid of `configure-ovs.sh` so adding workarounds and dependencies to it is something we should avoid otherwise it will be harder to remove. It is currently unknown if its replacement is going to be as flexible in this aspect.

As long as there is a way to obtain the currently used value from nmcli with reasonable or no parsing at all, I think it should be good enough to resolve the problem. So I don't think the values need to be set on the profile themselves.

@thaller do you have an example of nmcli output showing one of this values? Can it be requested specifically via `nmcli -g <field>`?

I also forgot we can't modify the NM profiles provided by MachineConfig because it monitors the files on disk and sets the state degraded.  I guess this is why we clone.

After 
`nmcli c mod "bond0" 802-3-ethernet.cloned-mac-address 52:54:00:3b:f3:bf`

machineconfiguration.openshift.io/reason: content mismatch for file "/etc/NetworkManager/system-connections/bond0 test .nmconnection"
machineconfiguration.openshift.io/state: Degraded

> For OpenShift we use MachineConfigs that contain RHCOS ignition settings.  We don't set per-machine ignition configs because that doesn't scale, so we can't specify individual unique client-ids.  We have one config for all the machines in a given role.
>
> I guess we could modify the systemd oneshot to `nmcli con mod 802-3-ethernet.cloned-mac-address` instead of the `match-device`.

Right. I didn't suggest that per-machine configs get created. There is already a script that writes /etc/NetworkManager/conf.d/99-client-id.conf.
What I think is a bad idea, is to write global-defauls in NetworkManager.conf, for settings that belong to the profile. The script can instead modify the profiles.

> I also forgot we can't modify the NM profiles provided by MachineConfig because it monitors the files on disk and sets the state degraded.  I guess this is why we clone.

I see. But modifying global settings seems more invasive than modifying certain pre-deployed profiles. If the degraded check forces you to not modifying a profile at expense of setting global defaults, than that seems a problem.

> We have a whole shell script `ovs-configure.sh` that clones the bond0 and adds it to the OVS br-ex bridge, so we can do extra stuff.

IMO, the second problem is that the configure-ovs.sh tries to guess the client-id/DUID/IAID, if it's not hard coded in the profile. The solution for that seems to ensure that it's hard coded in the profile and only support that usecase (that is, don't work around a problem)

You are already in strict control of the profiles, so that the system is marked as degraded when they change. So you can completely control, that the profile hard-codes the client-id/DUID/IAID, and consequently the migrate-to-ovs-script does not need to bother.

> We want to get rid of `configure-ovs.sh` so adding workarounds and dependencies to it is something we should avoid otherwise it will be harder to remove. 

I am suggesting to remove the code from configure-ovs.sh which aims to fill in the client-id/DUID/IAID.


Having said that, if you are content with continue doing that, it is fine by me :)

> @thaller do you have an example of nmcli output showing one of this values? Can it be requested specifically via `nmcli -g <field>`?


$ nmcli -f DHCP4,DHCP6 device show eth1 | grep 'dhcp_client_identifier\|dhcp6_client_id\|iaid'
DHCP4.OPTION[2]:                        dhcp_client_identifier = 01:aa:0f:f1:ce:00:01
DHCP6.OPTION[1]:                        dhcp6_client_id = 00:04:ab:4a:bf:f0:fc:cb:4a:c3:20:05:e0:26:28:07:14:e7
DHCP6.OPTION[4]:                        iaid = 2a:83:5d:db

$ cat "/run/NetworkManager/devices/$(cat /sys/devices/virtual/net/eth1/ifindex)" | grep '^\(dhcp4.dhcp_client_identifier\|dhcp6.dhcp6_client_id\|dhcp6.iaid\)='
dhcp4.dhcp_client_identifier=01:aa:0f:f1:ce:00:01
dhcp6.dhcp6_client_id=00:04:ab:4a:bf:f0:fc:cb:4a:c3:20:05:e0:26:28:07:14:e7
dhcp6.iaid=2a:83:5d:db

Above already works for client-id/DUID with current rhel-9.2+ and and rhel-8.8+.
Exposing the IAID is currently still missing (bug 2169869).

Would you be satisfied with that extention to NetworkManager?

@thaller that would work specifically for dhcp_client_identifier, dhcp6_client_id or dhcp6.iaid.

But as a general rule, would it work for any parameter that could be configured through NetworkManager.conf instead of the profile itself? Trying to come up with a general approach for this circumstance.

(In reply to Jaime Caamaño Ruiz from comment #13)
> @thaller that would work specifically for dhcp_client_identifier,
> dhcp6_client_id or dhcp6.iaid.
> 
> But as a general rule, would it work for any parameter that could be
> configured through NetworkManager.conf instead of the profile itself? Trying
> to come up with a general approach for this circumstance.

If you mean the NetworkManager.conf settings under the `[connection*]` section, then those are always default values, which are used as fallbacks to the respective per-profile setting. See "CONNECTION SECTION" in `man NetworkManager.conf`.

It means, all those keys can first and foremost be configured per-profile (via the similarly named per-profile property). Only if the per-profile property is set to a special value, which indicates to be the default, the fallback to the global default value in NetworkManager.conf is performed. And finally, if the global default value is unspecified in NetworkManager.conf, another, ultimate default value is chosen.

That ultimate default value is usually hard-coded (but maybe depending on the NetworkManager version). In some cases the ultimate default may depend on other things. For example, the ultimate default value for `ipv6.ip6-privacy` setting depends on `/proc/sys/net/ipv6/conf/default/use_tempaddr`.

All this means, if the profile does not explicitly set a value, then it can be tricky to know which setting will be used at runtime. Especially, for a program, which may not have the full logic to understand all the pieces. It gets further complicated because the `[connection*]` setting allows to set the global default per-device (see `[connection*].match-device=` key).

But if we look for example at `ipv4.dhcp-client-id` etc, then the problem was not only caused by default values. Even if you explicitly set the value in `ipv4.dhcp-client-id`, it might not be obvious which actual value is used at activation time. That is because the property can be set to special values `ipv4.dhcp-client-id=mac|perm-mac|ipv6-duid|duid|stable". While explicitly set per-profile, these values generate non-obvious client-ids at runtime, which would pose a similar problem as the original report had (even if it wasn't about default values in NetworkManager.conf).

The "solution" for client-id/duid/iaid might be that we now expose the currently used client-id/duid/iaid on a device(!) in the API and the user/tool can fetch them. However, as I said, I am doubting whether using that information from the script is really suitable to solve the original report. I would suggest to require setting a client-id as hex-string per-profile, if the tool expects to clone a profile and wants to ensure the same client-id is used. Trying to determine the dynamically generated client-id at runtime and bake it into a connection profile seems wrong to do. Require it to explicitly configured, and don't care otherwise.

---

TL;DR: so no, there is no general solution. If you have a property in the connection profile, then you need to understand what that property means. The profile may have the property explicitly set, but still some magic keywords (`ipv4.dhcp-client-id=mac`) that still won't help you. Or the property is unset and a default is chosen (which may depend on NetworkManager.conf or other settings, like sysctl). There is no general solution how to handle that or understand what any of this means. You need to understand the semantic and meaning of each property that you care about.

> You need to understand the semantic and meaning of each property that you care about.

But I don't think you need to worry about it beforehand.

The general solution is that you don't need to care and no special handling is required.

If you encounter a problem, you become aware of the specifics, and you can learn how to handle it specifically.

@thaller 

I think I might not have explained myself correctly when I said "the currently used value". Probably a better expression would have been "the configuration value in effect"

Let's say a system has a NetworkManager.conf file that contains

[connection]
ipv6.dhcp-iaid=mac

(or a variation that uses match-device). And let's say I have a device A activated with profile A' that does not specify ipv6.dhcp-iaid for which the fallback value specified in NetworkManager.conf applies.

We just want a (easy) way to tell the the ipv6.dhcp-iaid configuration value used to activate device A was "mac" (of course if it were a non special value, then we would like to get that non special value). 

To clarify, we don't want the actual value used for ipv6.dhcp-iaid, we want the configured value (which might be an special value or an actual value), if any, but regardless of whether it came from the profile or as a "fallback" in NetworkManager.conf. If no configuration value was specified for ipv6.dhcp-iaid either in the profile or as a fallback, then we would be happy to get a global default or an empty value if the default is internal or depends on other stuff.

dhcp-iaid would be an example. We would like this for any profile configuration parameter that allows specifying a fallback in NetworkManager.conf.

I don't think the discussion needs to be if it would be better to specify the configuration in the profile always. We agree there and already try to do this whenever we can. But using fallback values from NetworkManager.conf has proven useful and we would like to tap that flexibility and what we ask for here would allow us to do it better.

Anyway, I am particularly OK with this being closed as it is as we might be able to use it and might be enough for now. Otherwise we will consider opening a different RFE to continue the discussion, but of course don't refrain to add comments here to clarify.

And in any case, thanks for your time Thomas.

Right, the "solution" in bug 2169869 only allows you to see the actual (numerical) value for client-id/duid/iaid. Not the "meta" configuration that lead to those values.

Global connection defaults are IMO great, I use them myself heavily. But this is one of the major downsides that they have, and which is why I keep saying: maybe don't deal with global connection defaults.

---

On D-Bus, you can ask NetworkManager which is the content of the currently used configuration. For example, with

  nmcli connection up $PROFILE
  nmcli connection modify $PROFILE ...

the change to the profile did not take effect immediately (on the currently activated device). The device still remembers the previous setting that was used at activation time. There is the  GetAppliedConnection() D-Bus method, which gives the content of the profile when it originally activated. Note that properties that were set to the default, are there still the default in GetAppliedConnection(). Maybe that could be extended, where GetAppliedConnection() can (optionally) complete the profile's default values with the values from NetworkManager.conf. That might be a useful improvement.


But even then, it's not clear what you are gonna do with that information. If then you get:

  ipv4.dhcp-iaid=stable

What is the effect? As a human user, that information might be beneficial to confirm whether global-connection-defaults work as you intended. But what is a program/script gonna do with that information?



While that might be a nice improvement to global-connection-defaults, your use-case is not clear. If you would, please report an RFE. Thereby describe exactly what you want to do with that information. Since comment 0 talks about https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml#L313 , how would that be used there?