Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1846393

Summary: RHEL8.2 instances fail to contact metadata at boot
Product: Red Hat Enterprise Linux 8 Reporter: Eduardo Olivares <eolivare>
Component: cloud-initAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED DUPLICATE QA Contact: xiachen
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.2CC: amuller, chrisw, jgreguske, michele, myadla, ralonsoh, scohen, xiachen
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-16 07:52:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eduardo Olivares 2020-06-11 13:36:52 UTC
Description of problem:
In an environment with OSP16.1 installed, VM instances are created, connected to an external/provider network with external dhcp enabled (no tenant networks are used).

When a RHEL8.1 image is used, boot sequence is correct and metadata information is always obtained successfully (hostname, authorized keys, ...).
Versions:
Red Hat Enterprise Linux 8.1 (Ootpa)
Kernel 4.18.0-147.3.1.el8_1.x86_64 on an x86_64

Boot logs:
[  OK  ] Reached target Network (Pre).
         Starting Network Manager...
[  OK  ] Started Network Manager.
         Starting Initial cloud-init job (metadata service crawler)...
         Starting Network Manager Wait Online...
[  OK  ] Reached target Network.
         Starting Dynamic System Tuning Daemon...
         Starting Hostname Service...
[  OK  ] Started Dynamic System Tuning Daemon.
[  OK  ] Started Hostname Service.
         Starting Network Manager Script Dispatcher Service...
[   21.537350] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  OK  ] Started Network Manager Script Dispatcher Service.
[  OK  ] Started Network Manager Wait Online.
[   25.024121] cloud-init[773]: Cloud-init v. 18.5 running 'init' at Thu, 11 Jun 2020 09:15:28 +0000. Up 21.64 seconds.
[   25.026453] cloud-init[773]: ci-info: ++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
[   25.028843] cloud-init[773]: ci-info: +--------+------+------------------------------+-----------------+--------+-------------------+
[   25.031312] cloud-init[773]: ci-info: | Device |  Up  |           Address            |       Mask      | Scope  |     Hw-Address    |
[   25.033665] cloud-init[773]: ci-info: +--------+------+------------------------------+-----------------+--------+-------------------+
[   25.036049] cloud-init[773]: ci-info: |  eth0  | True |         10.46.21.232         | 255.255.255.192 | global | fa:16:3e:b4:73:19 |
[   25.038432] cloud-init[773]: ci-info: |  eth0  | True | fe80::f816:3eff:feb4:7319/64 |        .        |  link  | fa:16:3e:b4:73:19 |
[   25.040730] cloud-init[773]: ci-info: |   lo   | True |          127.0.0.1           |    255.0.0.0    |  host  |         .         |
[   25.043110] cloud-init[773]: ci-info: |   lo   | True |           ::1/128            |        .        |  host  |         .         |
[   25.045515] cloud-init[773]: ci-info: +--------+------+------------------------------+-----------------+--------+-------------------+
[   25.047885] cloud-init[773]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++++
[   25.050071] cloud-init[773]: ci-info: +-------+-----------------+--------------+-----------------+-----------+-------+
[   25.052214] cloud-init[773]: ci-info: | Route |   Destination   |   Gateway    |     Genmask     | Interface | Flags |
[   25.054368] cloud-init[773]: ci-info: +-------+-----------------+--------------+-----------------+-----------+-------+
[   25.056531] cloud-init[773]: ci-info: |   0   |     0.0.0.0     | 10.46.21.254 |     0.0.0.0     |    eth0   |   UG  |
[   25.058689] cloud-init[773]: ci-info: |   1   |   10.46.21.192  |   0.0.0.0    | 255.255.255.192 |    eth0   |   U   |
[   25.060869] cloud-init[773]: ci-info: |   2   | 169.254.169.254 | 10.46.21.215 | 255.255.255.255 |    eth0   |  UGH  |
[   25.063051] cloud-init[773]: ci-info: +-------+-----------------+--------------+-----------------+-----------+-------+
[   25.065218] cloud-init[773]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[   25.067021] cloud-init[773]: ci-info: +-------+-------------+---------+-----------+-------+
[   25.068808] cloud-init[773]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[   25.070594] cloud-init[773]: ci-info: +-------+-------------+---------+-----------+-------+
[   25.072328] cloud-init[773]: ci-info: |   1   |  fe80::/64  |    ::   |    eth0   |   U   |
[   25.074092] cloud-init[773]: ci-info: |   3   |   ff00::/8  |    ::   |    eth0   |   U   |
[   25.075814] cloud-init[773]: ci-info: +-------+-------------+---------+-----------+-------+
[  OK  ] Started Initial cloud-init job (metadata service crawler).
         Starting Permit User Sessions...
         Starting OpenSSH server daemon...
[  OK  ] Reached target Cloud-config availability.
[  OK  ] Reached target Network is Online.





When a RHEL8.2 image is used, metadata information sometimes is not obtained during boot. 
Versions:
Red Hat Enterprise Linux 8.2 (Ootpa)
Kernel 4.18.0-193.el8.x86_64 on an x86_64


Boot logs:
[  OK  ] Reached target Network (Pre).                                                                                                                                                                                              [422/1896]
         Starting Network Manager...
[  OK  ] Started Network Manager.
[  OK  ] Reached target Network.
         Starting Dynamic System Tuning Daemon...
         Starting Network Manager Wait Online...
         Starting Initial cloud-init job (metadata service crawler)...
         Starting Hostname Service...
[  OK  ] Started Dynamic System Tuning Daemon.
[  OK  ] Started Hostname Service.
         Starting Network Manager Script Dispatcher Service...
[   21.506159] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  OK  ] Started Network Manager Script Dispatcher Service.
[   21.580346] cloud-init[793]: Cloud-init v. 18.5 running 'init' at Thu, 11 Jun 2020 08:59:52 +0000. Up 21.48 seconds.
[   21.582467] cloud-init[793]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++
[   21.584623] cloud-init[793]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[   21.586834] cloud-init[793]: ci-info: | Device |  Up  |           Address            |    Mask   | Scope |     Hw-Address    |
[   21.589047] cloud-init[793]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[   21.591196] cloud-init[793]: ci-info: |  eth0  | True | fe80::f816:3eff:fe71:88a4/64 |     .     |  link | fa:16:3e:71:88:a4 |
[   21.593360] cloud-init[793]: ci-info: |   lo   | True |          127.0.0.1           | 255.0.0.0 |  host |         .         |
[   21.595528] cloud-init[793]: ci-info: |   lo   | True |           ::1/128            |     .     |  host |         .         |
[   21.597676] cloud-init[793]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+
[   21.599835] cloud-init[793]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[   21.601527] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[   21.603237] cloud-init[793]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[   21.604935] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[   21.606648] cloud-init[793]: ci-info: |   1   |  fe80::/64  |    ::   |    eth0   |   U   |
[   21.608373] cloud-init[793]: ci-info: |   3   |   ff00::/8  |    ::   |    eth0   |   U   |
[   21.610094] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[   21.611782] cloud-init[793]: 2020-06-11 08:59:52,886 - util.py[WARNING]: No active metadata service found
[  OK  ] Started Network Manager Wait Online.
[  OK  ] Started Initial cloud-init job (metadata service crawler).
         Starting OpenSSH server daemon...
[  OK  ] Reached target Cloud-config availability.
         Starting Permit User Sessions...
[  OK  ] Reached target Network is Online.





Main difference I found in the previous logs is that IPv4 address has not been obtained on RHEL8.2.

I compared the content of /usr/lib/systemd/system/cloud-init.service in 8.1 versus 8.2 instances and they are identical


Version-Release number of selected component (if applicable):
http://download.devel.redhat.com/rhel-8/rel-eng/RHEL-8/latest-RHEL-8.2.0/compose/BaseOS/x86_64/images/rhel-guest-image-8.2-290.x86_64.qcow2


How reproducible:
approx 50%


Steps to Reproduce:
1. openstack server create --image rhel82 --flavor rhel_flavor_1ram_1vpu_10disk --network public --key-name mykey --security-group sec_group vm-normal-prov-5
Network "public" should be an external/provider network with external dhcp enabled.


Actual results:
After boot, faulty VM cannot be accessed via SSH (because it received not public keys) and its hostname is wrong (it can be checked via virsh console)


Expected results:
Correct hostname and correctly obtained authorized public keys

Comment 1 Eduardo Olivares 2020-06-11 13:38:18 UTC
OSP16.1 version: RHOS-16.1-RHEL-8-20200609.n.0

Comment 9 Eduardo Olivares 2020-06-16 06:48:37 UTC
Hi Xiaoyi,

I just completed a test with 40 VM instances, created successfully. I used the RHEL8.2 version you recommended: http://download.eng.tlv.redhat.com/rhel-8/rel-eng/updates/RHEL-8/RHEL-8.2.0-updates-20200422.1/compose/BaseOS/x86_64/images/rhel-guest-image-8.2-326.x86_64.qcow2

[root@vm-direct ~]# rpm -qa | grep cloud-init
cloud-init-18.5-12.el8_2.1.noarch
[root@vm-direct ~]# uname -a
Linux vm-direct 4.18.0-193.el8.x86_64 #1 SMP Fri Mar 27 14:35:58 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@vm-direct ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)


In my opinion, this bug can be closed as duplicated of bug 1826262.

Thanks for your support!

Comment 10 xiachen 2020-06-16 07:52:53 UTC
Thanks Eduardo.
closed this bug as duplicated of bug 1826262.

If anyone still have this issue on cloud-init-18.5-12.el8_2.1.noarch, we can reopen bug 1826262.

*** This bug has been marked as a duplicate of bug 1826262 ***

Comment 11 Jakub Libosvar 2022-08-17 18:08:17 UTC
*** Bug 2103149 has been marked as a duplicate of this bug. ***