Bug 1846393

Summary: RHEL8.2 instances fail to contact metadata at boot
Product: Red Hat Enterprise Linux 8 Reporter: Eduardo Olivares <eolivare>
Component: cloud-initAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED DUPLICATE QA Contact: xiachen
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.2CC: amuller, chrisw, jgreguske, michele, myadla, ralonsoh, scohen, xiachen
Target Milestone: rc   
Target Release: 8.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-16 07:52:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eduardo Olivares 2020-06-11 13:36:52 UTC
Description of problem:
In an environment with OSP16.1 installed, VM instances are created, connected to an external/provider network with external dhcp enabled (no tenant networks are used).

When a RHEL8.1 image is used, boot sequence is correct and metadata information is always obtained successfully (hostname, authorized keys, ...).
Versions:
Red Hat Enterprise Linux 8.1 (Ootpa)
Kernel 4.18.0-147.3.1.el8_1.x86_64 on an x86_64

Boot logs:
[  OK  ] Reached target Network (Pre).
         Starting Network Manager...
[  OK  ] Started Network Manager.
         Starting Initial cloud-init job (metadata service crawler)...
         Starting Network Manager Wait Online...
[  OK  ] Reached target Network.
         Starting Dynamic System Tuning Daemon...
         Starting Hostname Service...
[  OK  ] Started Dynamic System Tuning Daemon.
[  OK  ] Started Hostname Service.
         Starting Network Manager Script Dispatcher Service...
[   21.537350] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  OK  ] Started Network Manager Script Dispatcher Service.
[  OK  ] Started Network Manager Wait Online.
[   25.024121] cloud-init[773]: Cloud-init v. 18.5 running 'init' at Thu, 11 Jun 2020 09:15:28 +0000. Up 21.64 seconds.
[   25.026453] cloud-init[773]: ci-info: ++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
[   25.028843] cloud-init[773]: ci-info: +--------+------+------------------------------+-----------------+--------+-------------------+
[   25.031312] cloud-init[773]: ci-info: | Device |  Up  |           Address            |       Mask      | Scope  |     Hw-Address    |
[   25.033665] cloud-init[773]: ci-info: +--------+------+------------------------------+-----------------+--------+-------------------+
[   25.036049] cloud-init[773]: ci-info: |  eth0  | True |         10.46.21.232         | 255.255.255.192 | global | fa:16:3e:b4:73:19 |
[   25.038432] cloud-init[773]: ci-info: |  eth0  | True | fe80::f816:3eff:feb4:7319/64 |        .        |  link  | fa:16:3e:b4:73:19 |
[   25.040730] cloud-init[773]: ci-info: |   lo   | True |          127.0.0.1           |    255.0.0.0    |  host  |         .         |
[   25.043110] cloud-init[773]: ci-info: |   lo   | True |           ::1/128            |        .        |  host  |         .         |
[   25.045515] cloud-init[773]: ci-info: +--------+------+------------------------------+-----------------+--------+-------------------+
[   25.047885] cloud-init[773]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++++
[   25.050071] cloud-init[773]: ci-info: +-------+-----------------+--------------+-----------------+-----------+-------+
[   25.052214] cloud-init[773]: ci-info: | Route |   Destination   |   Gateway    |     Genmask     | Interface | Flags |
[   25.054368] cloud-init[773]: ci-info: +-------+-----------------+--------------+-----------------+-----------+-------+
[   25.056531] cloud-init[773]: ci-info: |   0   |     0.0.0.0     | 10.46.21.254 |     0.0.0.0     |    eth0   |   UG  |
[   25.058689] cloud-init[773]: ci-info: |   1   |   10.46.21.192  |   0.0.0.0    | 255.255.255.192 |    eth0   |   U   |
[   25.060869] cloud-init[773]: ci-info: |   2   | 169.254.169.254 | 10.46.21.215 | 255.255.255.255 |    eth0   |  UGH  |
[   25.063051] cloud-init[773]: ci-info: +-------+-----------------+--------------+-----------------+-----------+-------+
[   25.065218] cloud-init[773]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[   25.067021] cloud-init[773]: ci-info: +-------+-------------+---------+-----------+-------+
[   25.068808] cloud-init[773]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[   25.070594] cloud-init[773]: ci-info: +-------+-------------+---------+-----------+-------+
[   25.072328] cloud-init[773]: ci-info: |   1   |  fe80::/64  |    ::   |    eth0   |   U   |
[   25.074092] cloud-init[773]: ci-info: |   3   |   ff00::/8  |    ::   |    eth0   |   U   |
[   25.075814] cloud-init[773]: ci-info: +-------+-------------+---------+-----------+-------+
[  OK  ] Started Initial cloud-init job (metadata service crawler).
         Starting Permit User Sessions...
         Starting OpenSSH server daemon...
[  OK  ] Reached target Cloud-config availability.
[  OK  ] Reached target Network is Online.





When a RHEL8.2 image is used, metadata information sometimes is not obtained during boot. 
Versions:
Red Hat Enterprise Linux 8.2 (Ootpa)
Kernel 4.18.0-193.el8.x86_64 on an x86_64


Boot logs:
[  OK  ] Reached target Network (Pre).                                                                                                                                                                                              [422/1896]
         Starting Network Manager...
[  OK  ] Started Network Manager.
[  OK  ] Reached target Network.
         Starting Dynamic System Tuning Daemon...
         Starting Network Manager Wait Online...
         Starting Initial cloud-init job (metadata service crawler)...
         Starting Hostname Service...
[  OK  ] Started Dynamic System Tuning Daemon.
[  OK  ] Started Hostname Service.
         Starting Network Manager Script Dispatcher Service...
[   21.506159] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  OK  ] Started Network Manager Script Dispatcher Service.
[   21.580346] cloud-init[793]: Cloud-init v. 18.5 running 'init' at Thu, 11 Jun 2020 08:59:52 +0000. Up 21.48 seconds.
[   21.582467] cloud-init[793]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++
[   21.584623] cloud-init[793]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[   21.586834] cloud-init[793]: ci-info: | Device |  Up  |           Address            |    Mask   | Scope |     Hw-Address    |
[   21.589047] cloud-init[793]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[   21.591196] cloud-init[793]: ci-info: |  eth0  | True | fe80::f816:3eff:fe71:88a4/64 |     .     |  link | fa:16:3e:71:88:a4 |
[   21.593360] cloud-init[793]: ci-info: |   lo   | True |          127.0.0.1           | 255.0.0.0 |  host |         .         |
[   21.595528] cloud-init[793]: ci-info: |   lo   | True |           ::1/128            |     .     |  host |         .         |
[   21.597676] cloud-init[793]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[   21.599835] cloud-init[793]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[   21.601527] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[   21.603237] cloud-init[793]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[   21.604935] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[   21.606648] cloud-init[793]: ci-info: |   1   |  fe80::/64  |    ::   |    eth0   |   U   |
[   21.608373] cloud-init[793]: ci-info: |   3   |   ff00::/8  |    ::   |    eth0   |   U   |
[   21.610094] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[   21.611782] cloud-init[793]: 2020-06-11 08:59:52,886 - util.py[WARNING]: No active metadata service found
[  OK  ] Started Network Manager Wait Online.
[  OK  ] Started Initial cloud-init job (metadata service crawler).
         Starting OpenSSH server daemon...
[  OK  ] Reached target Cloud-config availability.
         Starting Permit User Sessions...
[  OK  ] Reached target Network is Online.





Main difference I found in the previous logs is that IPv4 address has not been obtained on RHEL8.2.

I compared the content of /usr/lib/systemd/system/cloud-init.service in 8.1 versus 8.2 instances and they are identical


Version-Release number of selected component (if applicable):
http://download.devel.redhat.com/rhel-8/rel-eng/RHEL-8/latest-RHEL-8.2.0/compose/BaseOS/x86_64/images/rhel-guest-image-8.2-290.x86_64.qcow2


How reproducible:
approx 50%


Steps to Reproduce:
1. openstack server create --image rhel82 --flavor rhel_flavor_1ram_1vpu_10disk --network public --key-name mykey --security-group sec_group vm-normal-prov-5
Network "public" should be an external/provider network with external dhcp enabled.


Actual results:
After boot, faulty VM cannot be accessed via SSH (because it received not public keys) and its hostname is wrong (it can be checked via virsh console)


Expected results:
Correct hostname and correctly obtained authorized public keys

Comment 1 Eduardo Olivares 2020-06-11 13:38:18 UTC
OSP16.1 version: RHOS-16.1-RHEL-8-20200609.n.0

Comment 9 Eduardo Olivares 2020-06-16 06:48:37 UTC
Hi Xiaoyi,

I just completed a test with 40 VM instances, created successfully. I used the RHEL8.2 version you recommended: http://download.eng.tlv.redhat.com/rhel-8/rel-eng/updates/RHEL-8/RHEL-8.2.0-updates-20200422.1/compose/BaseOS/x86_64/images/rhel-guest-image-8.2-326.x86_64.qcow2

[root@vm-direct ~]# rpm -qa | grep cloud-init
cloud-init-18.5-12.el8_2.1.noarch
[root@vm-direct ~]# uname -a
Linux vm-direct 4.18.0-193.el8.x86_64 #1 SMP Fri Mar 27 14:35:58 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@vm-direct ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)


In my opinion, this bug can be closed as duplicated of bug 1826262.

Thanks for your support!

Comment 10 xiachen 2020-06-16 07:52:53 UTC
Thanks Eduardo.
closed this bug as duplicated of bug 1826262.

If anyone still have this issue on cloud-init-18.5-12.el8_2.1.noarch, we can reopen bug 1826262.

*** This bug has been marked as a duplicate of bug 1826262 ***

Comment 11 Jakub Libosvar 2022-08-17 18:08:17 UTC
*** Bug 2103149 has been marked as a duplicate of this bug. ***