Bug 1820540

This change was introduced by the commit:

commit 6954486d1318a3d1aa5bbcfcb119bcfcb73618bf
Author: Eduardo Otubo <otubo>
Date:   Fri Sep 6 12:12:11 2019 +0200

    Fix for network configuration not persisting after reboot
    
    RH-Author: Eduardo Otubo <otubo>
    Message-id: <20190906121211.23172-1-otubo>
    Patchwork-id: 90300
    O-Subject: [RHEL-7.8/RHEL-8.1.0 cloud-init PATCH] Fix for network configuration not persisting after reboot
    Bugzilla: 1593010
    RH-Acked-by: Mohammed Gamal <mgamal>
    RH-Acked-by: Miroslav Rezanina <mrezanin>
    
    The reasons the configuration does not persist after reboot includes
    different aspects and they're all fixed on this patch:
    
      1) The rpm package doesn't include the systemd-generator and
    ds-identify. The systemd-generator is called early in the boot process
    that calls ds-identify to check if there's any Data Source available in
    the current boot. In the current use case, the Data Source is removed
    from the VM on the second boot, this means cloud-init should disable
    itself in order to keep the configuration it did in the first boot.
    
      2) Even after adding those scripts, cloud-init was still being
    executed and the configurations were being lost. The reason for this is
    that the cloud-init systemd units had a wrong dependency
    
         WantedBy: multi-user.target
    
         Which would start them every time no matter the return of
    ds-identify. The fix is to replace the dependency by the systemd unit to
    cloud-init.target, which is the main cloud-init target enabled - or in
    this case, disabled by ds-identify. The file cloud-init.target was also
    missing on rpm package.
    
    After adding both scripts, the main cloud-init systemd target and
    adjusting the systemd dependencies the configuration persists after
    reboots and shutdowns.
    
    Signed-off-by: Eduardo Otubo <otubo>
    Signed-off-by: Miroslav Rezanina <mrezanin>


In the use case you're reporting, if you're doind a RHEL *upgrade* I assume this is not the first boot of your instance. If this is correct, cloud-init *should not* start as designed, it should start only on the first boot or if a DataSource is available. If your DataSource is available or this is indeed the first boot, then I think we got a corner case to work on. Please provide the logs, the complete scenario and the cloud-init version it's installed.

Thanks!

Dear Team,

I'm facing the same problem. It was working just fine on RHEL7.7 with WantedBy=multi-user.target.
I think it's related to this canonical bug:
https://bugs.launchpad.net/cloud-init/+bug/1771382
And to the previously mentioned non-persistent data problem:
https://access.redhat.com/errata/product/69/ver=/rhel---7/x86_64/RHBA-2020:1155

@Eduard Barrera
As a "not so pretty" workaround which works for me:
# mv /etc/systemd/system/cloud-init.target.wants/* /etc/systemd/system/multi-user.target.wants/

@Eduardo Otubo
Even if it works on the first run, there are modules in cloud-init what executes on every restart, like:
https://cloudinit.readthedocs.io/en/18.5/topics/modules.html#update-etc-hosts

Moreover, we should be able to re-trigger the first-run scenario via running cloud-init clean, or removing the /var/lib/cloud/data/status.json.
I know it's far from perfect, but since I'm building RedHat VM templates with Packer... the installation happens with a custom kickstart file, the customization on the first boot on a local hypervisor.
So the "real first boot" will be the second, after the installation.
From the kickstart I install and disable cloud-init's services / tried to disable it with touch /etc/cloud/cloud-init.disabled.
When the first boot happens, I install/modify the other necessary things and as a last step I enable cloud-init's services or remove the /etc/cloud/cloud-init.disabled or execute cloud-init clean.
At the end I have a full-blown ovf-format VM template.
Previously it was working fine.

For me cloud-init reports now:
# cat /run/cloud-init/cloud-init-generator.log 
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.x86_64 root=UUID=7b921c59-425e-45a3-ba67-6c38386d6e33 ro elevator=noop transparent_hugepage=never spectre_v2=retpoline biosdevname=0 net.ifnames=0 LANG=en_US.UTF-8
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=1
ds-identify _RET=notfound
cloud-init is enabled but no datasource found, disabling
already disabled: no change needed [no /run/systemd/generator.early/multi-user.target.wants/cloud-init.target]

When I switch to multiuser.target.wants as I mentioned previously, it works.

Best regards,
András Kovács

I gave it a try like including my cloud.cfg in the kickstart file (with heredoc format).
The config file is there, but "no datasource found"... like in my previous comment.
Moving to multi-user.target.wants it works.

Does it help if I open a customer support ticket with this bug in it?

Thank you!

Best regards,
András Kovács

Dear Team, 

Do I need to provide anything else?
I really hope I provided all the necessary info previously, but please, correct me if I'm wrong.

Thank you!

Best regards,
András Kovács

Test on OpenStack platform, could not reproduce the issue.

My test steps,
1. launch a instance with image RHEL-7.7-Server-x86_64-production
2. login and register the system
# subscription-manager register
# subscription-manager attach --auto
3. check 
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)
# rpm -qa|grep cloud-init
cloud-init-18.5-3.el7.x86_64

# cd /etc/systemd/system
# ls -l
lrwxrwxrwx. 1 root root  ...  default.target -> /lib/systemd/system/multi-user.target

4. upgrade to rhel 7.8
#yum upgrade
5. check 
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 (Maipo)
# rpm -qa|grep cloud-init
cloud-init-18.5-6.el7.x86_64

#cd /etc/systemd/system/
#ls -l
drwxr-xr-x. 2 root root 4096 Apr  7 03:53 multi-user.target.wants
drwxr-xr-x. 2 root root  119 Apr  7 03:53 cloud-init.target.wants
lrwxrwxrwx. 1 root root   37 Mar 10 06:40 default.target -> /lib/systemd/system/multi-user.target

6. check cloud-init 4 services(cloud-init-local, cloud-init, cloud-config, cloud-final), all are active.
7. clean and reboot
#cloud-init clean
#reboot
8. login and check cloud-init 4 services(cloud-init-local, cloud-init, cloud-config, cloud-final), all are active.
# cat /run/cloud-init/cloud-init-generator.log 
/usr/lib/systemd/system-generators/cloud-init-generator normal=/run/systemd/generator early=/run/systemd/generator.early late=/run/systemd/generator.late
kernel command line (/proc/cmdline): BOOT_IMAGE=/boot/vmlinuz-3.10.0-1127.el7.x86_64 root=UUID=3425ddda-aed9-4a29-886c-3bd6b776b539 ro console=tty0 console=ttyS0,115200n8 no_timer_check net.ifnames=0 crashkernel=auto LANG=en_US.UTF-8
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=0
ds-identify _RET=found
enabled via /run/systemd/generator.early/multi-user.target.wants/cloud-init.target -> /lib/systemd/system/cloud-init.target


Hi Andras,
Did I miss any key step to reproduce the issue?

(In reply to Andras Kovacs from comment #6)
> Dear Team, 
> 
> Do I need to provide anything else?
> I really hope I provided all the necessary info previously, but please,
> correct me if I'm wrong.
> 
> Thank you!
> 
> Best regards,
> András Kovács

Yes I think you provided all info I needed. I'll work on this issue.
Thanks for the report!

(In reply to Eduardo Otubo from comment #8)
> (In reply to Andras Kovacs from comment #6)
> > Dear Team, 
> > 
> > Do I need to provide anything else?
> > I really hope I provided all the necessary info previously, but please,
> > correct me if I'm wrong.
> > 
> > Thank you!
> > 
> > Best regards,
> > András Kovács
> 
> Yes I think you provided all info I needed. I'll work on this issue.
> Thanks for the report!

Hi Eduardo,

Do you have any suggestion about how to reproduce this issue? Did I miss anything in my steps Comment #7 ?

(In reply to xiachen from comment #10)
> (In reply to Eduardo Otubo from comment #8)
> > (In reply to Andras Kovacs from comment #6)
> > > Dear Team, 
> > > 
> > > Do I need to provide anything else?
> > > I really hope I provided all the necessary info previously, but please,
> > > correct me if I'm wrong.
> > > 
> > > Thank you!
> > > 
> > > Best regards,
> > > András Kovács
> > 
> > Yes I think you provided all info I needed. I'll work on this issue.
> > Thanks for the report!
> 
> Hi Eduardo,
> 
> Do you have any suggestion about how to reproduce this issue? Did I miss
> anything in my steps Comment #7 ?

I don't have more info on how to reproduce the issue.

@Eduard Barrera,  Andras Kovacs : in which environment did you found this issue?

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days