Bug 1363633

Summary: [cloud-init] - cloud-init should finish before NetworkManager starts or run ifdown ifup after cloud-init is finished
Product: Red Hat Enterprise Linux 7 Reporter: Michael Burman <mburman>
Component: cloud-initAssignee: Lars Kellogg-Stedman <lars>
Status: CLOSED ERRATA QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: danken, erich, huzhao, jherrman, mbracho, mburman, sasha, yacao
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: cloud-init-0.7.9-1.el7 Doc Type: Bug Fix
Doc Text:
Due to a race condition between the cloud-init application and the NetworkManager utility, systems with cloud-init set up in some cases booted with an incorrect network configuration. This update ensures that the cloud-init-local service runs before NetworkManager starts and that the cloud-init service runs afterwards. As a result, the described problem no longer occurs.
 Story Points: ---
Clone Of:
: 1413151 (view as bug list) Environment:
Last Closed: 2017-04-04 15:26:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1413151    
Attachments:
Description Flags
messages logs none

Description Michael Burman 2016-08-03 09:07:27 UTC 
Created attachment 1186987 [details]
messages logs

Description of problem:
[clout-init] - cloud-init should finish before NetworkManager starts or run ifdown ifup after cloud-init is finished.

This bug a new and clear version of - BZ 1288819 that was very long and confusing.

The bug is that cloud-init comes up after the NetworkManager and that can cause to some problems like:
cloud-init with static ip configuration can end up that the guest OS is up with ip from dhcp, cause NetworkManager finished first. 

The solution should be-
- cloud-init should finish before NetworkManager starts
- run ifdown and ifup after cloud-init finished or run restart network when cloud-init ends

cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
BOOTPROTOv6=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=yes
PERSISTENT_DHCLIENT=1
NETMASK=255.255.255.0
IPADDR=8.8.8.8
GATEWAY=8.8.8.254

Version-Release number of selected component (if applicable):
cloud-init-0.7.6-6.el7.x86_64

How reproducible:
100

Steps to Reproduce:
1. Create VM with rhel7.2 image and configure the cloud-init with static ip configuration.
It means that the guest should be up with static ip and ifcfg-eth0 with static config.

Actual results:
But from the messages and dmesg logs it is seems like the NetworkManager configures the eth0 nic first(eth0 get ip from dhcp) and only then cloud-init touch the nic. 

cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
BOOTPROTOv6=dhcp
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=yes
IPV6INIT=yes
PERSISTENT_DHCLIENT=1
NETMASK=255.255.255.0
IPADDR=8.8.8.8
GATEWAY=8.8.8.254

Expected results:
cloud-init should finish before NetworkManager starts or run ifdown ifup after cloud-init is finished.

Additional info:
To work around it we need to restart network or reboot guest. 

See also BZ 1288819
Comment 2 Lars Kellogg-Stedman 2016-12-05 15:40:58 UTC 
The systemd units in upstream cloud-init have gone through a number of changes recently to improve the ordering of things w/r/t networking.  These changes will be available in the next release of our cloud-init package, which I am working on right now.
Comment 3 Lars Kellogg-Stedman 2016-12-07 22:59:24 UTC 
How are you providing a static ip configuration?  If you are doing that via a userdata script, would you please attach that to this bug report?  If you are doing that via some other mechanism, let me know.   Thanks!
Comment 4 Michael Burman 2016-12-08 12:28:29 UTC 
Hello Lars

I'm using the rhv-m ui in order to configure the static ip in the clout-init.
And there is a custom-script in cloud-init via run once or edit vm.

I think that the custom-script should be similar to something like:

network-interfaces: |
  auto lo
  iface lo inet loopback

  iface eth0 inet static
    address xxx.xxx.xxx.xxx (static ip)
    netmask xxx.xxx.xxx.xxx
    gateway xxx.xxx.xxx.xxx (gateway ip, usually ip address of router)
Comment 9 Lars Kellogg-Stedman 2017-01-16 14:02:16 UTC 
Awesome, thank you for verifying the fix!
Comment 14 errata-xmlrpc 2017-04-04 15:26:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0871