Bug 2144898

Workaround, disable network config via cloud-init:
/etc/cloud/cloud.cfg.d/99-app.cfg
network:
  config: disabled

I was informed my test case may not be complete, ofcause you need to have cloud-init networking config enabled, for me this is only happening on Azure, because that is where I use cloud-init.

This happens for example with 
"offer": "RHEL",
"publisher": "RedHat",
"sku": "8-lvm-gen2",
"version": "8.7.2022112213"


doesn't happen with 
"version": "8.6.2022102701"

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=00:22:48:5c:ff:b9
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
$ nmcli con modify 'System eth0' +ipv4.dns "8.8.8.8 8.8.1.1" +ipv4.dns-search example.com
$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=00:22:48:5C:FF:B9
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PROXY_METHOD=none
BROWSER_ONLY=no
DNS1=8.8.8.8
DNS2=8.8.1.1
DOMAIN=example.com
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03
$ systemctl reboot
[ ... After System is back from reboot ...]

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=00:22:48:5c:ff:b9
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

I can reproduce the issue on Azure rhel-8.7 and latest rhel-8.8(cloud-init-22.1-5.el8 and cloud-init-22.1-6.el8)

No such issue in rhel-8.6.z
No such issue on OpenStack

Test Steps:

1. Deploy VM on Azure with rhel-8.7
2. Check the ifcfg-eth0 and resolv.conf file:
# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=60:45:bd:d6:ac:88
METRIC=100
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

# cat /etc/resolv.conf 
# Generated by NetworkManager
search v21m1stkujze1kkrzxi2apwmcb.bx.internal.cloudapp.net
nameserver 168.63.129.16

3. Modify the file /etc/sysconfig/network-scripts/ifcfg-eth0 with below nmcli command and check the ifcfg-eth0:
# nmcli con modify 'System eth0' +ipv4.dns "8.8.8.8 8.8.1.1" +ipv4.dns-search example.com

# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=00:22:48:21:D6:2D
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PROXY_METHOD=none
BROWSER_ONLY=no
DNS1=8.8.8.8
DNS2=8.8.1.1
DOMAIN=example.com
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

4. Reboot the VM:
# systemctl reboot

5. Check the files ifcfg-eth0 and resolv.conf


Actual results:
After step 5, the modifications in step 3 were not persisted.

# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=60:45:bd:d6:ac:88
METRIC=100
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

# cat /etc/resolv.conf 
# Generated by NetworkManager
search v21m1stkujze1kkrzxi2apwmcb.bx.internal.cloudapp.net
nameserver 168.63.129.16


Expected results:
After step 5, the modifications in step 3 should be persisted.

# cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=00:22:48:21:D6:2D
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PROXY_METHOD=none
BROWSER_ONLY=no
DNS1=8.8.8.8
DNS2=8.8.1.1
DOMAIN=example.com
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME="System eth0"
UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03

# cat /etc/resolv.conf 
# Generated by NetworkManager
search example.com
nameserver 8.8.8.8
nameserver 8.8.1.1
nameserver 168.63.129.16


Will research more for the root cause, thanks!

After checking the cloud-init.log, I think this issue maybe related with the actions after reboot:

1. For the working rhel-8.6, after reboot, cloud-init does not config network anymore, so user's modifications can be persisted:
----------------------------------------
   1194 2022-12-06 02:27:18,123 - __init__.py[DEBUG]: Datasource DataSourceAzure [seed=/dev/sr0] not updated for events: System boot
   1195 2022-12-06 02:27:18,123 - stages.py[DEBUG]: No network config applied. Neither a new instance nor datasource network update on 'System boot' event
   1196 2022-12-06 02:27:18,123 - stages.py[DEBUG]: applying net config names for {'version': 2, 'ethernets': {'eth0': {'dhcp4': True, 'dhcp4-overrides': {'ro        ute-metric': 100}, 'dhcp6': False, 'match': {'macaddress': '00:22:48:21:d6:2d', 'driver': 'hv_netvsc'}, 'set-name': 'eth0'}}}

----------------------------------------


2. For the NOT-working rhel-8.7, after reboot, cloud-init reset network per metadata, so user's modifications are overwritten by the default metadata:
----------------------------------------
   1335 2022-12-06 02:11:07,829 - stages.py[DEBUG]: Allowed events: {<EventScope.NETWORK: 'network'>: {<EventType.BOOT_NEW_INSTANCE: 'boot-new-instance'>, <Ev        entType.BOOT: 'boot'>}}
   1336 2022-12-06 02:11:07,829 - stages.py[DEBUG]: Event Allowed: scope=network EventType=boot
   1337 2022-12-06 02:11:07,829 - __init__.py[DEBUG]: Update datasource metadata and network config due to events: boot
   1338 2022-12-06 02:11:07,829 - handlers.py[DEBUG]: start: azure-ds/_get_data: _get_data
   1339 2022-12-06 02:11:07,829 - handlers.py[DEBUG]: start: azure-ds/check-platform-viability: found azure asset tag
   1340 2022-12-06 02:11:07,829 - dmi.py[DEBUG]: querying dmi data /sys/class/dmi/id/chassis_asset_tag
----------------------------------------

And there are some error logs likes below:
----------------------------------------
   1363 2022-12-06 02:11:09,041 - util.py[DEBUG]: Failed to mount device: '/dev/sr0' with type: 'auto' using mount command: 'mount -o ro -t auto /dev/sr0 /run        /cloud-init/tmp/tmpfsr62ryn', which caused exception: Unexpected error while running command.
   1364 Command: ['mount', '-o', 'ro', '-t', 'auto', '/dev/sr0', '/run/cloud-init/tmp/tmpfsr62ryn']
   1365 Exit code: 32
   1366 Reason: -
   1367 Stdout:
   1368 Stderr: mount: /run/cloud-init/tmp/tmpfsr62ryn: no medium found on /dev/sr0.
   1369 2022-12-06 02:11:09,041 - azure.py[DEBUG]: /dev/sr0 was not mountable
----------------------------------------


Will attach the /var/log/cloud-init.log, please check the details if needed, thanks!

Setting BZ to high so we can take a look at it on this release.
I'll investigate this problem.

Thanks!

It seems a change on purpose in 21.4
Set Azure to apply networking config every BOOT (#1023)
https://github.com/canonical/cloud-init/commit/dc22786980a05129c5971e68ae37b1a9f76f882d

@yuxin
Do you know why Azure needs to apply networking every BOOT?

I'll change the topic to reflect the findings of @huzhao

Hi Klaas,

First of all, thank you for raising this issue.

May I ask you some questions about your usage scenario? Why do you have to add DNS and what are these DNS used for?
I asked this because the change you found is on purpose from cloud-init upstream, 
so we want to figure out if it meets the needs of users or it is a regression problem.

thanks,
Amy

(In reply to xiachen from comment #11)
> Hi Klaas,
> 
> First of all, thank you for raising this issue.
> 
> May I ask you some questions about your usage scenario? Why do you have to
> add DNS and what are these DNS used for?

We add additional searchdomains to certain servers, mostly because they communicate with legacy urls in our company without using fqdn. I also see this as a problem with our SAP systems that use this kind of setup, but they are not on 8.7 (nor will they ever be), but they'll migrate to 8.8 in future.

> I asked this because the change you found is on purpose from cloud-init
> upstream, 
> so we want to figure out if it meets the needs of users or it is a
> regression problem.

So this may be a new feature, not a regression :) But yeah, my general statements for this would be these:
1) I need a way to add additional searchdomains for a couple of servers, searchdomains that are not provided through dhcp; I do not care how this is achieved, but there should be a general solution that is tested by Red Hat QA for future releases.
2) I think overwriting a file that is being used as configuration file by another active tool (network manager) is not really good practice. But maybe this should be a 2nd ticket for network manager, it should not allow modifications when cloud-init is active or something like that.

> 
> thanks,
> Amy

Klass -> I, an Azure customer, had the same exact issue that I recently raised w/ both RH and Azure support.

Simply add your Search Domains in /etc/dhcp/dhclient.conf and put whatever cloud-init changes you made back to original.

Example:
[root@e2-redisucn3-04 ~]# cat /etc/dhcp/dhclient.conf
timeout 300;
append domain-search "searchdomain1.com";
append domain-search "searchdomain2.com";


This will persist across reboots too.

---John

Hi John,
thanks for your workaround, but this case is more about providing a way hat is Red Hat tested. I do not want to be in the same situation on the next cloud-init update because it decides to also overwrite dhclient.conf :) So I want Red Hat to create an approved solution that they actively test in QA.

Greetings
Klaas

Thanks John provide the workaround.

I tested it on Azure, it works.

My steps,
1.Modify /etc/dhcp/dhclient.conf
timeout 300;
append domain-search "searchdomain1.com";
append domain-search "searchdomain2.com";
append domain-name-servers 8.8.8.8;

2. Reboot the VM and then check the dns information, the modifications are not overwritten.
 cat /etc/resolv.conf 
# Generated by NetworkManager
search searchdomain1.com searchdomain2.com
nameserver 168.63.129.16
nameserver 8.8.8.8


(In reply to John Yacubovich from comment #15)
> Klass -> I, an Azure customer, had the same exact issue that I recently
> raised w/ both RH and Azure support.
> 
> Simply add your Search Domains in /etc/dhcp/dhclient.conf and put whatever
> cloud-init changes you made back to original.
> 
> Example:
> [root@e2-redisucn3-04 ~]# cat /etc/dhcp/dhclient.conf
> timeout 300;
> append domain-search "searchdomain1.com";
> append domain-search "searchdomain2.com";
> 
> 
> This will persist across reboots too.
> 
> ---John

Hi Klaas,

I quite understand you. I tested the workaround, it works, see comment#21.
You can try it and let me know if it works for you.

Besides, I raised my questions to cloud-init upstream and asked why they made that changes.
https://github.com/canonical/cloud-init/pull/1023
and they said it is designed for Azure, upon every boot, cloud-init needs to fetch network config again to make sure it has the most up-to-date network config for the VM because the user might add/remove network interface when the VM is stopped.
If something is broken, it'd be more helpful to file a bug that describes what behavior was working along with the first version you see broken. Include the results from cloud-init collect-logs --include-userdata whenever possible.

So if the workaround doesn't work for you, we can file a bug with your requirements to cloud-init upstream.


(In reply to Klaas Demter from comment #18)
> Hi John,
> thanks for your workaround, but this case is more about providing a way hat
> is Red Hat tested. I do not want to be in the same situation on the next
> cloud-init update because it decides to also overwrite dhclient.conf :) So I
> want Red Hat to create an approved solution that they actively test in QA.
> 
> Greetings
> Klaas

Hi @xiachen ,
so my main concern is/was that a test for this solution included into Red Hat QA for new releases on Azure. I don't want to re-open this in 6 months for the next minor release again. I want Red Hat to find such a problem within their QA :)

Apart from that, I am happy with the workaround, it is easily handled in my environment.

Greetings
Klaas

(In reply to Klaas Demter from comment #23)
> Hi @xiachen ,
> so my main concern is/was that a test for this solution included into Red
> Hat QA for new releases on Azure. I don't want to re-open this in 6 months
> for the next minor release again. I want Red Hat to find such a problem
> within their QA :)
> 
> Apart from that, I am happy with the workaround, it is easily handled in my
> environment.
> 
> Greetings
> Klaas

Hi Klaas,

I am really sorry that we(QA) did not detect this change and notify customers timely.

To be honest, we did not cover this scenario test before, we were mainly focus on the tests of instance first boot config as this is cloud-init's base/main function. And we are trying to extend our test coverage to cover customers' normal and specific scenarios, thank you so much for raising the issue and scenario, we already discussed this in our meetings, will involve this scenario tests in future. And will try to check/test the main changes/patches from upstream community rebase, notify users timely for the big changes.

As this change comes from upstream community patch[1], we need to discuss more with patch owner to check if can enhance the patch to meet our requirements(persist customized config after reboot). xiachen and I both left comments to discuss it, and I just reported a bug[2] to track the process. Will update here and let you know if any update.
[1] https://github.com/canonical/cloud-init/pull/1023
[2] https://bugs.launchpad.net/cloud-init/+bug/2003048


For your requirement about persist the DNS customized config, I think you can use the workaround mentioned in comment 15, comment 21 and comment 23, glad to know it works for you. Thanks John Yacubovich providing the workaround, and QA xiachen already tested it on Azure.


Regarding the QA recommended workaround or solution, I think we need some more time to verify and Document it, as need to wait for upstream community patch enhancement solution, then can decide how to resolve or workaround it in official document.

Does this make sense for you?

Thanks!
Huijuan Zhao

Hi Germano,

We would like to create a KCS for this issue to let customers know the change/enhancement, could you please help to review if the content is suitable?

----------------------------------------
cloud-init updates the network configuration on every boot on Microsoft Azure

This change comes from a new patch https://github.com/canonical/cloud-init/pull/1023 since rhel-8.7/9.1, it is designed for Azure, upon every boot, cloud-init needs to fetch network config again to make sure it has the most up-to-date network config for the VM because the user might add/remove network interface when the VM is stopped.

As a consequence, if you manually configured settings on interfaces, such as an additional search domain, cloud-init maybe overrides them when you reboot the VM. There are two workarounds to persist manually configured settings on interfaces.

1. Create the /etc/cloud/cloud.cfg.d/99-disable-network.cfg file with the following settings.
----
network:
  config: disabled
----
But this setting maybe cause other issues, e.g. can not set new network configs automatic via cloud-init, so any networking changes from the cloud would likely leave the instance in an unreachable state

2. Do not manually modify /etc/sysconfig/network-scripts/ifcfg-eth* which will be overwritten by cloud-init after reboot. 
Write modifications to other config files which are not handled by cloud-init. e.g. For the DNS specific requirement, can write modifications to /etc/dhcp/dhclient.conf manually, then it will be persist after reboot.
----
# cat /etc/dhcp/dhclient.conf
append domain-search "searchdomain1.com";
append domain-search "searchdomain2.com";
append domain-name-servers 8.8.8.8;
----
----------------------------------------

Thanks,
Huijuan

(In reply to Huijuan Zhao from comment #31)
> Hi Germano,
> 
> We would like to create a KCS for this issue to let customers know the
> change/enhancement, could you please help to review if the content is
> suitable?

Hi Huijuan, thanks for raising this. Of course, sounds like a KCS is really
important and necessary here. Maybe a release notes update too.

> 
> ----------------------------------------
> cloud-init updates the network configuration on every boot on Microsoft Azure
> 
> This change comes from a new patch
> https://github.com/canonical/cloud-init/pull/1023 since rhel-8.7/9.1, it is
> designed for Azure, upon every boot, cloud-init needs to fetch network
> config again to make sure it has the most up-to-date network config for the
> VM because the user might add/remove network interface when the VM is
> stopped.

I think this should have been an opt-in behaviour in cloud.cfg. Changing this
so late in the RHEL8 lifecycle and having it by default just breaks customers
use cases.

> 
> As a consequence, if you manually configured settings on interfaces, such as
> an additional search domain, cloud-init maybe overrides them when you reboot
> the VM. There are two workarounds to persist manually configured settings on
> interfaces.
> 
> 1. Create the /etc/cloud/cloud.cfg.d/99-disable-network.cfg file with the
> following settings.
> ----
> network:
>   config: disabled
> ----
> But this setting maybe cause other issues, e.g. can not set new network
> configs automatic via cloud-init, so any networking changes from the cloud
> would likely leave the instance in an unreachable state
> 
> 2. Do not manually modify /etc/sysconfig/network-scripts/ifcfg-eth* which
> will be overwritten by cloud-init after reboot. 
> Write modifications to other config files which are not handled by
> cloud-init. e.g. For the DNS specific requirement, can write modifications
> to /etc/dhcp/dhclient.conf manually, then it will be persist after reboot.
> ----
> # cat /etc/dhcp/dhclient.conf
> append domain-search "searchdomain1.com";
> append domain-search "searchdomain2.com";
> append domain-name-servers 8.8.8.8;
> ----

This workaround is a very corner case of:
a) User using DHCP, otherwise that is not even called
b) Needs to set specific DNS configuration

While we can start with this I think need something more generic for the KCS, 
able to apply more kinds of settings and ideally using NetworkManager.

Maybe a way to run extra 'nmcli con mod' commands at the end of every network config?

> ----------------------------------------
> 
> Thanks,
> Huijuan

I'll start the KCS and add here for review.

Thanks!

Done, could you please take a look?

If we have better or more generic workarounds for item 2 we can update.

(In reply to Germano Veit Michel from comment #33)
> Done, could you please take a look?
> 
> If we have better or more generic workarounds for item 2 we can update.

Germano, thank you so much for the quick response. The KCS looks great, I appreciate it.

And thanks for your suggestions, agree with you. Marc just helped adding this issue to release notes, and will ask him to link this KCS in release notes once done.

Regarding the workaround option 2, there is another config method which can persist the DNS, could you please help to review if it is more generic?
--------------------------------------------
Manually add DNS to /etc/resolv.conf, and disable NetworkManager update DNS in /etc/NetworkManager/conf.d/dns_disable.conf. Then the manually added DNS can be persist after reboot.

e.g.

# cat /etc/resolv.conf
search searchdomain1.com
search searchdomain2.com
nameserver 8.8.8.8

# cat /etc/NetworkManager/conf.d/dns_disable.conf 
[main]
dns=none
--------------------------------------------

(In reply to Marc Muehlfeld from comment #30)
> If you plan to address the workaround(s) in the KCS you want to write, we
> can link this KCS from the release note for further details and the
> workaround.
> Please let me know when your KCS is ready.

Hi Marc,

Thank you for the suggestion.
Germano helped wrote the KCS https://access.redhat.com/solutions/6995988, could you please help to review if need to link it to release notes rhel-8.7/9.1?

Thanks!
Huijuan

(In reply to Huijuan Zhao from comment #34)
> (In reply to Germano Veit Michel from comment #33)
> > Done, could you please take a look?
> > 
> > If we have better or more generic workarounds for item 2 we can update.
> 
> Germano, thank you so much for the quick response. The KCS looks great, I
> appreciate it.
> 
> And thanks for your suggestions, agree with you. Marc just helped adding
> this issue to release notes, and will ask him to link this KCS in release
> notes once done.
> 
> Regarding the workaround option 2, there is another config method which can
> persist the DNS, could you please help to review if it is more generic?
> --------------------------------------------
> Manually add DNS to /etc/resolv.conf, and disable NetworkManager update DNS
> in /etc/NetworkManager/conf.d/dns_disable.conf. Then the manually added DNS
> can be persist after reboot.
> 
> e.g.
> 
> # cat /etc/resolv.conf
> search searchdomain1.com
> search searchdomain2.com
> nameserver 8.8.8.8
> 
> # cat /etc/NetworkManager/conf.d/dns_disable.conf 
> [main]
> dns=none
> --------------------------------------------

Germano, sorry for a small update after read the KCS again.

For the option 1, the path /etc/cloud/cloud.cfg.d/ will be created by cloud-init when install/run cloud-init, so is it necessary to remove below command from the KCS? Thanks!
-----------
# mkdir -p /etc/cloud/cloud.cfg.d/
-----------

(In reply to Huijuan Zhao from comment #36)
> Germano, sorry for a small update after read the KCS again.
> 
> For the option 1, the path /etc/cloud/cloud.cfg.d/ will be created by
> cloud-init when install/run cloud-init, so is it necessary to remove below
> command from the KCS? Thanks!
> -----------
> # mkdir -p /etc/cloud/cloud.cfg.d/
> -----------

Probably not necessary to remove, it won't hurt. But if we can guarantee
the directory will always there I can remove it to make it simpler.

So no matter what happens or what is configured, this directory will always
be there?

(In reply to Germano Veit Michel from comment #37)
> (In reply to Huijuan Zhao from comment #36)
> > Germano, sorry for a small update after read the KCS again.
> > 
> > For the option 1, the path /etc/cloud/cloud.cfg.d/ will be created by
> > cloud-init when install/run cloud-init, so is it necessary to remove below
> > command from the KCS? Thanks!
> > -----------
> > # mkdir -p /etc/cloud/cloud.cfg.d/
> > -----------
> 
> Probably not necessary to remove, it won't hurt. But if we can guarantee
> the directory will always there I can remove it to make it simpler.
> 
> So no matter what happens or what is configured, this directory will always
> be there?

Yes, this directory will be there once cloud-init is installed. But I agree with you that it won't hurt, so it is up to you whether keep the command :)

The release note is now available on the Customer Portal:

RHEL 8.7: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/8.7_release_notes/index#BZ-2144898
RHEL 9.1: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/9.1_release_notes/index#BZ-2144898

Hi Klaas,

This is an enhancement from https://github.com/canonical/cloud-init/pull/1023 per the discussions above, we already recorded the change and workaround to RHEL release notes(see comment 44) and KCS https://access.redhat.com/solutions/6995988, could we close this BZ as WONTFIX?

Thanks,
Huijuan

(In reply to Huijuan Zhao from comment #45)
> Hi Klaas,
> 
> This is an enhancement from
> https://github.com/canonical/cloud-init/pull/1023 per the discussions above,
> we already recorded the change and workaround to RHEL release notes(see
> comment 44) and KCS https://access.redhat.com/solutions/6995988, could we
> close this BZ as WONTFIX?
> 
> Thanks,
> Huijuan

Yes, you can. I still think we should follow up with NetworkManager, I don't think it's good that both cloud-init and network manager can't live side by side properly.

(In reply to Klaas Demter from comment #47)
> Yes, you can. I still think we should follow up with NetworkManager, I don't
> think it's good that both cloud-init and network manager can't live side by
> side properly.

Thank you Klaas. I see, currently cloud-init mainly re-write /etc/sysconfig/network-scripts/ifcfg-eth* after reboot, there is reminder message in ifcfg-eth* like "# Created by cloud-init on instance boot automatically, do not edit.", so we can modify network config manually via other files except ifcfg-eth* to persist the customized configurations.

Closing the BZ, please re-open it or report new BZ if still have issue. Thanks!