Bug 1465730
Summary: | [Regression] cloud-init fails to configure network while EC2 creates instance from AMI | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Chen Shi <cheshi> | ||||||
Component: | cloud-init | Assignee: | Ryan McCabe <rmccabe> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Chen Shi <cheshi> | ||||||
Severity: | urgent | Docs Contact: | |||||||
Priority: | urgent | ||||||||
Version: | 7.4 | CC: | aherr, ailan, cavery, cfeist, cheshi, ctwelve, fdinitto, huzhao, james.cuzella, jbadiapa, jgreguske, lars, leiwang, linl, mmagr, mrunge, rmccabe, sacpatil, ssigwald, ushkalim, vkuznets, wshi, yuxisun | ||||||
Target Milestone: | rc | Keywords: | EC2, Regression, Triaged | ||||||
Target Release: | --- | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | cloud-init-0.7.9-13.el7 | Doc Type: | If docs needed, set a value | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2018-04-10 14:05:07 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 1451548, 1456511 | ||||||||
Attachments: |
|
Description
Chen Shi
2017-06-28 04:27:27 UTC
Some useful information from /var/log/messages: ``` Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info> [1498643617.9597] device (lo): link connected Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info> [1498643617.9621] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1) Jun 28 09:53:37 ip-172-31-0-87 systemd: Started Hostname Service. Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info> [1498643617.9680] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2) Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info> [1498643617.9723] device (eth0): state change: unmanaged -> unavailable (reason 'managed') [10 20 2] Jun 28 09:53:37 ip-172-31-0-87 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info> [1498643617.9836] device (eth0): link connected Jun 28 09:53:38 ip-172-31-0-87 NetworkManager[480]: <info> [1498643618.0186] device (eth0): state change: unavailable -> disconnected (reason 'none') [20 30 0] Jun 28 09:53:38 ip-172-31-0-87 NetworkManager[480]: <info> [1498643618.0219] manager: startup complete Jun 28 09:53:38 ip-172-31-0-87 systemd: Started Network Manager Wait Online. Jun 28 09:53:38 ip-172-31-0-87 systemd: Starting LSB: Bring up/down networking... Jun 28 09:53:38 ip-172-31-0-87 network: Bringing up loopback interface: [ OK ] Jun 28 09:53:38 ip-172-31-0-87 NetworkManager[480]: <info> [1498643618.5053] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" result="fail" reason="No suitable device found for this connection." Jun 28 09:53:38 ip-172-31-0-87 network: Bringing up interface eth0: Error: Connection activation failed: No suitable device found for this connection. Jun 28 09:53:38 ip-172-31-0-87 network: [FAILED] Jun 28 09:53:38 ip-172-31-0-87 systemd: network.service: control process exited, code=exited status=1 Jun 28 09:53:38 ip-172-31-0-87 systemd: Failed to start LSB: Bring up/down networking. Jun 28 09:53:38 ip-172-31-0-87 systemd: Unit network.service entered failed state. Jun 28 09:53:38 ip-172-31-0-87 systemd: network.service failed. ``` We added scripts to rc.local, and get the following information during system boot: $ ifconfig -a eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 06:9e:0f:f7:c4:65 txqueuelen 1000 (Ethernet) RX packets 2 bytes 140 (140.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ...... $ cat /etc/sysconfig/network-scripts/ifcfg-eth0 BOOTPROTO=dhcp DEVICE=eth0 HWADDR=06:91:96:e1:6e:e7 ONBOOT=yes TYPE=Ethernet USERCTL=no $ cat /etc/udev/rules.d/70-persistent-net.rules SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="06:91:96:e1:6e:e7", NAME="eth0" We believe that mismatched MAC address in configure file caused "eth0" boot up failure. (We also checked with "cloud-init-0.7.9-6.el7.x86_64", they are same MAC address) I believe that the problem is due to the way you are creating the new AMI in step 4:
> 4. Create an AMI from this instance, make sure the AMI contain cloud-init-0.7.9-8.el7.x86_64 package.
It doesn't appear that you are running "virt-sysprep" or otherwise attempting to de-configure the instance before generating the AMI. Minimally, you will need to remove the HWADDR line from /etc/sysconfig/network-scripts/ifcfg-eth0, and you will need to remove the corresponding udev rule in /etc/udev/rules.d/70-persistent-net.rules.
Running "virt-sysprep" should accomplish both tasks for you.
If you are still experiencing a problem after implementing these steps, please update this bug with details.
(In reply to Lars Kellogg-Stedman from comment #4) > I believe that the problem is due to the way you are creating the new AMI in > step 4: > > > 4. Create an AMI from this instance, make sure the AMI contain cloud-init-0.7.9-8.el7.x86_64 package. > > It doesn't appear that you are running "virt-sysprep" or otherwise > attempting to de-configure the instance before generating the AMI. > Minimally, you will need to remove the HWADDR line from > /etc/sysconfig/network-scripts/ifcfg-eth0, and you will need to remove the > corresponding udev rule in /etc/udev/rules.d/70-persistent-net.rules. > > Running "virt-sysprep" should accomplish both tasks for you. > > If you are still experiencing a problem after implementing these steps, > please update this bug with details. Hi Lars, Thank you for the inputs. I think this procedure should be done by AWS, and I have no idea about "virt-sysprep", is this used to deal with the AMI? We can't download and even not to download the AMI, deal with it, then upload it back to AWS. So what is the usage of "virt-sysprep" with AWS EC2? Thanks, Charles Hi, I tried to reproduce it in Azure and it seems that if I update cloud-init package, the service is not active during booting phase. I think it might be the root cause of this issue. My steps: 1. Update cloud-init from 0.7.9-3 to 0.7.9-9 through rpm -Uvh 2. Ensure the services are enabled: # systemctl is-enabled cloud-{init-local,init,config,final} enabled enabled enabled enabled 3. Reboot 4. Check /var/log/cloud-init.log Actual result: There's no new log in cloud-init.log Then check /var/log/messages, there's no cloud-init related messages Debug: Compare the files in 0.7.9-3 and 0.7.9-9, the /usr/lib/systemd/system/cloud-init.target /usr/lib/systemd/system-generators/cloud-init-generator 2 files are not in the 0.7.9-9. It seems that in old version(0.7.9-3) it put cloud-*.services into /etc/systemd/system/cloud-init.target.wants/ folder, but not in multi-user.target.wants/. Then use che cloud-init.target to run cloud-init related services after multi-user.target. The cloud-init.target needs cloud-init-generator to enable itself. # ll /etc/systemd/system/cloud-init.target.wants/ total 0 lrwxrwxrwx. 1 root root 44 Jul 20 09:24 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service lrwxrwxrwx. 1 root root 43 Jul 20 09:24 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service lrwxrwxrwx. 1 root root 48 Jul 20 09:24 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service lrwxrwxrwx. 1 root root 42 Jul 20 09:24 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service [root@wala73cloud0793 ~]# ll /usr/lib/systemd/system/cloud-init.target -rw-r--r--. 1 root root 255 Dec 23 2016 /usr/lib/systemd/system/cloud-init.target [root@wala73cloud0793 ~]# ll /usr/lib/systemd/system-generators/cloud-init-generator -rwxr-xr-x. 1 root root 3972 Dec 23 2016 /usr/lib/systemd/system-generators/cloud-init-generator # cat /usr/lib/systemd/system/cloud-init.target # cloud-init target is enabled by cloud-init-generator # To disable it you can either: # a.) boot with kernel cmdline of 'cloudinit=disabled' # b.) touch a file /etc/cloud/cloud-init.disabled [Unit] Description=Cloud-init target After=multi-user.target In v0.7.9-9, if fresh install, cloud-*.service are in multi-user.target.wants/, so the cloud-init.target and cloud-init-generator are not needed anymore. # ll /etc/systemd/system/multi-user.target.wants/|grep cloud lrwxrwxrwx. 1 root root 44 Jul 20 11:26 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service lrwxrwxrwx. 1 root root 43 Jul 20 11:26 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service lrwxrwxrwx. 1 root root 48 Jul 20 11:26 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service lrwxrwxrwx. 1 root root 42 Jul 20 11:26 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service But if upgrade from v0.7.9-3 to 0.7.9-9, the cloud-init related services are not re-enabled, so the .service soft links are still in /etc/systemd/system/cloud-init.target.wants/. But the cloud-init.target and cloud-init-generator files are removed. So systemd cannot run cloud-init related services. [root@wala73cloud0793bak ~]# ls /usr/lib/systemd/system/cloud-init.target ls: cannot access /usr/lib/systemd/system/cloud-init.target: No such file or directory [root@wala73cloud0793bak ~]# ls /usr/lib/systemd/system-generators/cloud-init-generator ls: cannot access /usr/lib/systemd/system-generators/cloud-init-generator: No such file or directory [root@wala73cloud0793bak ~]# ll /etc/systemd/system/multi-user.target.wants/|grep cloud [root@wala73cloud0793bak ~]# ll /etc/systemd/system/cloud-init.target.wants/ total 0 lrwxrwxrwx. 1 root root 44 Jul 20 09:24 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service lrwxrwxrwx. 1 root root 43 Jul 20 09:24 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service lrwxrwxrwx. 1 root root 48 Jul 20 09:24 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service lrwxrwxrwx. 1 root root 42 Jul 20 09:24 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service As verified on AWS with RHEL7.4 RC1, we can see "cloud-init services are not re-enabled" is the root cause for issue "Bring up networking failed". And for RHEL-7.4_HVM_GA-20170724-x86_64-1-Hourly2-GP2 (ami-3901e15f), the .service soft links are already in multi-user.target.wants . So upgrading cloud-init based on RHEL7.4 GA will have no issue. RHEL7.4 GA: [ec2-user@ip-172-31-9-0 system]$ uname -a Linux ip-172-31-9-0.ap-northeast-1.compute.internal 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux [ec2-user@ip-172-31-9-0 system]$ rpm -qa | grep cloud-init cloud-init-0.7.9-9.el7.x86_64 [ec2-user@ip-172-31-9-0 system]$ find . | grep cloud ./multi-user.target.wants/cloud-config.service ./multi-user.target.wants/cloud-final.service ./multi-user.target.wants/cloud-init.service ./multi-user.target.wants/cloud-init-local.service [ec2-user@ip-172-31-9-0 system]$ As a conclusion, customer will encounter this issue when: 1. Upgrading cloud-init based on RHEL-7.4_HVM_Beta-20170518-x86_64-1-Hourly2-GP2 - ami (or earlier version); AND, 2. Make an AMI for further use. Workaround: execute the following commands before making the AMI: # systemctl disable cloud-{init-local,init,config,final}.service # systemctl enable cloud-{init-local,init,config,final}.service make sure the .service soft links are located in multi-user.target.wants . Created attachment 1389947 [details]
nvidia-installer_on_3.10.0-705.el7.x86_64.log
Created attachment 1389948 [details]
nvidia-installer_on_3.10.0-706.el7.x86_64.log
Comment on attachment 1389947 [details]
nvidia-installer_on_3.10.0-705.el7.x86_64.log
the attachment is not for this bug
Comment on attachment 1389948 [details]
nvidia-installer_on_3.10.0-706.el7.x86_64.log
the attachment is not for this bug
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0806 The package providing "virt-sysprep" seems unnecessarily bulky given that it has so many dependencies. Does RedHat really expect all cloud users to install 116 extra packages just to fix networking for AMI based instances!? This is a breaking change to cloud-init which causes networking to fail during boot. This will affect any instance using AMIs containing the hardcoded MAC address of the virtual machine used to generate the instance. It's a pretty big deal to break networking for any company depending on an image creation process for a cloud instance. Many cloud providers such as AWS do not provide an easy way to debug such issues other than the boot console output. For those searching for "virt-sysprep", it's provided by the "libguestfs-tools" package. This package depends on a whopping 116 other extra packages! Seems like a waste of space on a new AMI, just to fix networking, something that was previously working fine. $ yum whatprovides virt-sysprep Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.advancedhosters.com * epel: epel.mirror.constant.com * extras: mirrors.advancedhosters.com * updates: mirrors.advancedhosters.com epel/x86_64/filelists_db | 11 MB 00:00:02 1:libguestfs-tools-c-1.40.2-5.el7.x86_64 : System administration tools for virtual machines Repo : base Matched from: Filename : /usr/bin/virt-sysprep 1:libguestfs-tools-c-1.40.2-5.el7_7.1.x86_64 : System administration tools for virtual machines Repo : updates Matched from: Filename : /usr/bin/virt-sysprep $ sudo yum install libguestfs-tools Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile epel/x86_64/metalink | 16 kB 00:00:00 * base: repos-va.psychz.net * epel: iad.mirror.rackspace.com * extras: repos-va.psychz.net * updates: repos-va.psychz.net base | 3.6 kB 00:00:00 epel | 5.3 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): epel/x86_64/updateinfo | 1.0 MB 00:00:00 (2/2): epel/x86_64/primary_db | 6.9 MB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package libguestfs-tools.noarch 1:1.40.2-5.el7_7.1 will be installed --> Processing Dependency: libguestfs-tools-c = 1:1.40.2-5.el7_7.1 for package: 1:libguestfs-tools-1.40.2-5.el7_7.1.noarch --> Processing Dependency: libguestfs = 1:1.40.2-5.el7_7.1 for package: 1:libguestfs-tools-1.40.2-5.el7_7.1.noarch [...SNIP...] Dependencies Resolved ================================================================================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================================================================================ Installing: libguestfs-tools noarch 1:1.40.2-5.el7_7.1 updates 28 k Installing for dependencies: alsa-lib x86_64 1.1.8-1.el7 base 425 k attr x86_64 2.4.46-13.el7 base 66 k augeas-libs x86_64 1.4.0-9.el7 base 356 k boost-iostreams x86_64 1.53.0-27.el7 base 61 k boost-random x86_64 1.53.0-27.el7 base 39 k boost-system x86_64 1.53.0-27.el7 base 40 k boost-thread x86_64 1.53.0-27.el7 base 57 k bridge-utils x86_64 1.5-9.el7 base 32 k celt051 x86_64 0.5.1.3-8.el7 base 53 k cryptsetup x86_64 2.0.3-5.el7 base 154 k cyrus-sasl-gssapi x86_64 2.1.26-23.el7 base 41 k device-mapper-event x86_64 7:1.02.158-2.el7 base 189 k device-mapper-event-libs x86_64 7:1.02.158-2.el7 base 189 k device-mapper-persistent-data x86_64 0.8.5-1.el7 base 423 k dnsmasq x86_64 2.76-9.el7 base 278 k dosfstools x86_64 3.0.20-10.el7 base 101 k ebtables x86_64 2.0.10-16.el7 base 123 k flac-libs x86_64 1.3.0-5.el7_1 base 169 k fuse x86_64 2.9.2-11.el7 base 86 k fuse-libs x86_64 2.9.2-11.el7 base 93 k gdisk x86_64 0.8.10-3.el7 base 190 k genisoimage x86_64 1.1.11-25.el7 base 299 k glusterfs x86_64 3.12.2-47.2.el7 base 512 k glusterfs-api x86_64 3.12.2-47.2.el7 base 77 k glusterfs-cli x86_64 3.12.2-47.2.el7 base 179 k glusterfs-client-xlators x86_64 3.12.2-47.2.el7 base 883 k glusterfs-libs x86_64 3.12.2-47.2.el7 base 387 k gnutls x86_64 3.3.29-9.el7_6 base 680 k gperftools-libs x86_64 2.6.1-1.el7 base 272 k gsm x86_64 1.0.13-11.el7 base 30 k hexedit x86_64 1.2.13-5.el7 base 39 k hivex x86_64 1.3.10-6.9.el7 base 101 k ipxe-roms-qemu noarch 20180825-2.git133f4c.el7 base 791 k iscsi-initiator-utils x86_64 6.2.0.874-11.el7 base 422 k iscsi-initiator-utils-iscsiuio x86_64 6.2.0.874-11.el7 base 92 k libICE x86_64 1.0.9-9.el7 base 66 k libSM x86_64 1.2.2-2.el7 base 39 k libX11 x86_64 1.6.7-2.el7 base 607 k libX11-common noarch 1.6.7-2.el7 base 164 k libXau x86_64 1.0.8-2.1.el7 base 29 k libXext x86_64 1.3.3-3.el7 base 39 k libXi x86_64 1.7.9-1.el7 base 40 k libXtst x86_64 1.2.3-1.el7 base 20 k libaio x86_64 0.3.109-13.el7 base 24 k libasyncns x86_64 0.8-7.el7 base 26 k libconfig x86_64 1.4.9-5.el7 base 59 k libguestfs x86_64 1:1.40.2-5.el7_7.1 updates 2.4 M libguestfs-tools-c x86_64 1:1.40.2-5.el7_7.1 updates 4.9 M libibverbs x86_64 22.1-3.el7 base 267 k libiscsi x86_64 1.9.0-7.el7 base 60 k libjpeg-turbo x86_64 1.2.90-8.el7 base 135 k libogg x86_64 2:1.3.0-7.el7 base 24 k libpciaccess x86_64 0.14-1.el7 base 26 k librados2 x86_64 1:10.2.5-4.el7 base 1.8 M librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M librdmacm x86_64 22.1-3.el7 base 63 k libreport-filesystem x86_64 2.1.11-43.el7.centos base 40 k libsndfile x86_64 1.0.25-10.el7 base 149 k libusal x86_64 1.1.11-25.el7 base 136 k libusbx x86_64 1.0.21-1.el7 base 61 k libvirt-daemon x86_64 4.5.0-23.el7_7.1 updates 841 k libvirt-daemon-driver-interface x86_64 4.5.0-23.el7_7.1 updates 236 k libvirt-daemon-driver-network x86_64 4.5.0-23.el7_7.1 updates 410 k libvirt-daemon-driver-nodedev x86_64 4.5.0-23.el7_7.1 updates 236 k libvirt-daemon-driver-nwfilter x86_64 4.5.0-23.el7_7.1 updates 260 k libvirt-daemon-driver-qemu x86_64 4.5.0-23.el7_7.1 updates 745 k libvirt-daemon-driver-secret x86_64 4.5.0-23.el7_7.1 updates 226 k libvirt-daemon-driver-storage x86_64 4.5.0-23.el7_7.1 updates 197 k libvirt-daemon-driver-storage-core x86_64 4.5.0-23.el7_7.1 updates 435 k libvirt-daemon-driver-storage-disk x86_64 4.5.0-23.el7_7.1 updates 227 k libvirt-daemon-driver-storage-gluster x86_64 4.5.0-23.el7_7.1 updates 235 k libvirt-daemon-driver-storage-iscsi x86_64 4.5.0-23.el7_7.1 updates 225 k libvirt-daemon-driver-storage-logical x86_64 4.5.0-23.el7_7.1 updates 229 k libvirt-daemon-driver-storage-mpath x86_64 4.5.0-23.el7_7.1 updates 224 k libvirt-daemon-driver-storage-rbd x86_64 4.5.0-23.el7_7.1 updates 230 k libvirt-daemon-driver-storage-scsi x86_64 4.5.0-23.el7_7.1 updates 225 k libvirt-daemon-kvm x86_64 4.5.0-23.el7_7.1 updates 197 k libvirt-libs x86_64 4.5.0-23.el7_7.1 updates 4.2 M libvorbis x86_64 1:1.3.3-8.el7.1 base 204 k libxcb x86_64 1.13-1.el7 base 214 k libxslt x86_64 1.1.28-5.el7 base 242 k lsscsi x86_64 0.27-6.el7 base 47 k lvm2 x86_64 7:2.02.185-2.el7 base 1.3 M lvm2-libs x86_64 7:2.02.185-2.el7 base 1.1 M lzop x86_64 1.03-10.el7 base 54 k mdadm x86_64 4.1-1.el7 base 435 k mtools x86_64 4.0.18-5.el7 base 203 k netcf-libs x86_64 0.2.8-4.el7 base 70 k nettle x86_64 2.7.1-8.el7 base 327 k numad x86_64 0.5-18.20150602git.el7 base 35 k opus x86_64 1.0.2-6.el7 base 630 k osinfo-db noarch 20190319-2.el7 base 191 k pciutils x86_64 3.5.1-3.el7 base 93 k perl-Sys-Guestfs x86_64 1:1.40.2-5.el7_7.1 updates 306 k perl-Sys-Virt x86_64 4.5.0-2.el7 base 296 k perl-hivex x86_64 1.3.10-6.9.el7 base 41 k perl-libintl x86_64 1.20-12.el7 base 875 k pixman x86_64 0.34.0-1.el7 base 248 k psmisc x86_64 22.20-16.el7 base 141 k pulseaudio-libs x86_64 10.0-5.el7 base 651 k qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k radvd x86_64 2.17-3.el7 base 94 k rdma-core x86_64 22.1-3.el7 base 50 k scrub x86_64 2.5.2-7.el7 base 41 k seabios-bin noarch 1.11.0-2.el7 base 112 k seavgabios-bin noarch 1.11.0-2.el7 base 38 k sgabios-bin noarch 1:0.20110622svn-4.el7 base 7.1 k spice-server x86_64 0.14.0-7.el7 base 404 k squashfs-tools x86_64 4.3-0.21.gitaae0aff4.el7 base 101 k supermin5 x86_64 5.1.19-1.el7 base 637 k syslinux x86_64 4.05-15.el7 base 990 k syslinux-extlinux x86_64 4.05-15.el7 base 364 k trousers x86_64 0.3.14-2.el7 base 289 k usbredir x86_64 0.7.1-3.el7 base 47 k Transaction Summary ================================================================================================================================================================================================================ Install 1 Package (+116 Dependent packages) Total download size: 44 M Installed size: 143 M Is this ok [y/d/N]: N Ok, so it wasn't clear at first that "virt-sysprep" tool is intended to be used on a disk image file, not a running system. Given the following: $ /usr/bin/virt-sysprep --dry-run --enable udev-persistent-net --verbose virt-sysprep: error: you must give either -a or -d options. Read virt-sysprep(1) man page for further information. If reporting bugs, run virt-sysprep with debugging enabled and include the complete output: virt-sysprep -v -x [...] Therefore only these options are valid: -a, --add <file> Add disk image file -d, --domain <domain> Set libvirt guest name So, given that this bug affects AWS EC2 and presumably any cloud instance using CentOS / RHEL 7 + cloud-init, we are unable to use "virt-sysprep" here. The other workaround seems to be: sudo yum install cloud-init sudo systemctl disable cloud-{init-local,init,config,final}.service sudo systemctl enable cloud-{init-local,init,config,final}.service sudo -n sed -i '/HWADDR/d' /etc/sysconfig/network-scripts/ifcfg-eth* /etc/sysconfig/network-scripts/ifcfg-ens* sudo -n sed -i '/UUID/d' /etc/sysconfig/network-scripts/ifcfg-eth* /etc/sysconfig/network-scripts/ifcfg-ens* Then removing the MAC address also from udev rules file: /etc/udev/rules.d/70-persistent-net.rules For example: # BEFORE SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="01:12:23:34:45:56", KERNEL=="ens*", NAME="ens3" # AFTER SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{dev_id}=="0x0", KERNEL=="ens*", ATTR{type}=="1", NAME="ens3" |