Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1843031

Summary: Intermittent failure to start cloud-init due to failure to detect macs
Product: Red Hat Enterprise Linux 8 Reporter: David Hill <dhill>
Component: cloud-initAssignee: Eduardo Otubo <eterrell>
Status: CLOSED CURRENTRELEASE QA Contact: xiachen
Severity: urgent Docs Contact:
Priority: low    
Version: 8.1CC: ailan, broose, coldford, eterrell, huzhao, jgreguske, lichen, ltamagno, marjones, mrezanin, mtessun, mvalsecc, oelswah, ribarry, tmicheli, xiachen, yacao
Target Milestone: rcKeywords: Reopened, TestOnly, ZStream
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: cloud-init-21.1-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1957135 1957136 1957137 (view as bug list) Environment:
Last Closed: 2021-06-22 13:42:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1958174    
Bug Blocks: 1957135, 1957136, 1957137    

Description David Hill 2020-06-02 15:19:12 UTC 
Description of problem:
Intermittent failure to start cloud-init due to failure to find nics based on macs:


May 30 00:39:09 overcloud-compute-0 cloud-init[913362]: ValueError: Unable to find a system nic for {'type': 'physical', 'mtu': 9000, 'subnets': [{'type': 'dhcp4'}], 'mac_address': '48:df:37:00:00:00'}

Version-Release number of selected component (if applicable):


How reproducible:
Intermitteng

Steps to Reproduce:
1. Reboot computes/controllers having bonds.
2.
3.

Actual results:
cloud-init fails

Expected results:
cloud-init should never fail

Additional info:
Comment 1 David Hill 2020-06-02 15:31:47 UTC 

*** This bug has been marked as a duplicate of bug 1768770 ***
Comment 2 David Hill 2020-06-02 15:32:50 UTC 
Customer have cloud-init-18.5-12 installed.
Comment 3 David Hill 2020-06-02 15:48:14 UTC 
We can see they used to have the previous issue [1]:
~~~
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]: Traceback (most recent call last):
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 652, in status_wrapper
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:    ret = functor(name, args)
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 362, in main_init
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:    init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 649, in apply_network_config
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:    netcfg, src = self._find_networking_config()
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 636, in _find_networking_config
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:    if self.datasource and hasattr(self.datasource, 'network_config'):
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:  File "/usr/lib/python3.6/site-packages/cloudinit/sources/DataSourceConfigDrive.py", line 155, in network_config
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:    self.network_json, known_macs=self.known_macs)
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:  File "/usr/lib/python3.6/site-packages/cloudinit/sources/helpers/openstack.py", line 655, in convert_net_json
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:    known_macs = net.get_interfaces_by_mac()
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:  File "/usr/lib/python3.6/site-packages/cloudinit/net/__init__.py", line 596, in get_interfaces_by_mac
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]:    (name, ret[mac], mac))
messages-20200531:May 25 22:39:36 overcloud-compute-0 cloud-init[40685]: RuntimeError: duplicate mac found! both 'ens2f0' and 'bond_api' have mac '48:df:37:00:00:00'
~~~

then updated cloud-init:
~~~
[dhill@supportshell log]$ cat dnf.rpm.log  | grep cloud
2020-05-25T22:58:38Z SUBDEBUG Upgrade: cloud-init-18.5-12.el8_2.1.noarch
2020-05-25T22:58:39Z SUBDEBUG Upgrade: cloud-init-18.5-12.el8_2.1.noarch
2020-05-25T22:58:40Z SUBDEBUG Upgraded: cloud-init-18.5-7.el8.noarch
2020-05-25T22:58:41Z SUBDEBUG Upgraded: cloud-init-18.5-7.el8.noarch
~~~

and now hit the new issue:
~~~
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]: failed run of stage init
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]: ------------------------------------------------------------
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]: Traceback (most recent call last):
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:    ret = functor(name, args)
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:  File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 362, in main_init
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:    init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 649, in apply_network_config
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:    netcfg, src = self._find_networking_config()
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:  File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 636, in _find_networking_config
messages-20200531:May 30 00:39:05 overcloud-compute-0 cloud-init[913362]:    if self.datasource and hasattr(self.datasource, 'network_config'):
messages-20200531:May 30 00:39:09 overcloud-compute-0 systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE
messages-20200531:May 30 00:39:09 overcloud-compute-0 cloud-init[913362]:  File "/usr/lib/python3.6/site-packages/cloudinit/sources/DataSourceConfigDrive.py", line 155, in network_config
messages-20200531:May 30 00:39:09 overcloud-compute-0 cloud-init[913362]:    self.network_json, known_macs=self.known_macs)
messages-20200531:May 30 00:39:09 overcloud-compute-0 cloud-init[913362]:  File "/usr/lib/python3.6/site-packages/cloudinit/sources/helpers/openstack.py", line 669, in convert_net_json
messages-20200531:May 30 00:39:09 overcloud-compute-0 cloud-init[913362]:    raise ValueError("Unable to find a system nic for %s" % d)
messages-20200531:May 30 00:39:09 overcloud-compute-0 cloud-init[913362]: ValueError: Unable to find a system nic for {'type': 'physical', 'mtu': 9000, 'subnets': [{'type': 'dhcp4'}], 'mac_address': '48:df:37:00:00:00'}
messages-20200531:May 30 00:39:09 overcloud-compute-0 cloud-init[913362]: ------------------------------------------------------------
~~~


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1768770
Comment 13 Michele Valsecchi 2021-03-08 08:16:51 UTC 
I'm reopening the BZ as I've seen this is still happening in the latest version of RHOSP(16.1.3), which uses "cloud-init-18.5-12.el8_2.3.noarch".
IIUC this looks like a regression to me.

Now, as RHOSP16.1 is tightly coupled to RHEL8.2 (i.e. we do not support any other RHEL minor version), we would need a fix on RHEL8.2.
Let me know if you prefer me to clone this BZ and have a second one to track the fix for 8.2.
Comment 20 Osama Elswah 2021-03-10 08:42:58 UTC 
Also hitting the same problem at my CU

Mar 09 09:52:41 d100siul0111 cloud-init[4862]: 2021-03-09 08:52:41,846 - util.py[WARNING]: failed stage init
Mar 09 09:52:41 d100siul0111 cloud-init[4862]: failed run of stage init
Mar 09 09:52:41 d100siul0111 cloud-init[4862]: ------------------------------------------------------------
Mar 09 09:52:41 d100siul0111 cloud-init[4862]: Traceback (most recent call last):
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:   File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 653, in status_wrapper
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:     ret = functor(name, args)
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:   File "/usr/lib/python3.6/site-packages/cloudinit/cmd/main.py", line 362, in main_init
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:     init.apply_network_config(bring_up=bool(mode != sources.DSMODE_LOCAL))
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:   File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 649, in apply_network_config
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:     netcfg, src = self._find_networking_config()
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:   File "/usr/lib/python3.6/site-packages/cloudinit/stages.py", line 636, in _find_networking_config
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:     if self.datasource and hasattr(self.datasource, 'network_config'):
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:   File "/usr/lib/python3.6/site-packages/cloudinit/sources/DataSourceConfigDrive.py", line 155, in network_config
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:     self.network_json, known_macs=self.known_macs)
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:   File "/usr/lib/python3.6/site-packages/cloudinit/sources/helpers/openstack.py", line 677, in convert_net_json
Mar 09 09:52:41 d100siul0111 cloud-init[4862]:     raise ValueError("Unable to find a system nic for %s" % d)
Mar 09 09:52:41 d100siul0111 cloud-init[4862]: ValueError: Unable to find a system nic for {'type': 'physical', 'mtu': 1500, 'subnets': [{'type': 'dhcp4'}], 'mac_address': '48:df:37:a9:b4:10'}
Mar 09 09:52:41 d100siul0111 cloud-init[4862]: ------------------------------------------------------------
Mar 09 09:52:41 d100siul0111 systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE
Mar 09 09:52:41 d100siul0111 systemd[1]: cloud-init.service: Failed with result 'exit-code'.
Mar 09 09:52:41 d100siul0111 systemd[1]: Failed to start Initial cloud-init job (metadata service crawler).

root@d100siul0111 ~ # rpm -qa | grep cloud-init
cloud-init-18.5-12.el8_2.3.noarch
root@d100siul0111 cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 (Ootpa)
Comment 56 Eduardo Otubo 2021-04-08 10:01:40 UTC 
The two identified commits seems to fix the problem:

  3c432b32 get_interfaces: don't exclude Open vSwitch bridge/bond members (#608)
  121bc04c net: exclude OVS internal interfaces in get_interfaces (#829)

Those fixes are written for cloud-init-21.1 code base and cannot be backported to the latest RHEL-8.2 / cloud-init-18.5-12.el8_2.6 package.
Comment 103 xiachen 2021-07-07 03:40:24 UTC 
Hi David,

Bug is fixed in cloud-init-21.1-2.el8, is it workable for your customer?

Would you please update the resolution section of this KCS?
https://access.redhat.com/solutions/5126581


thanks,
Amy