Bug 1465730

Summary: [Regression] cloud-init fails to configure network while EC2 creates instance from AMI
Product: Red Hat Enterprise Linux 7 Reporter: Chen Shi <cheshi>
Component: cloud-initAssignee: Ryan McCabe <rmccabe>
Status: CLOSED ERRATA QA Contact: Chen Shi <cheshi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.4CC: aherr, ailan, cavery, cfeist, cheshi, ctwelve, fdinitto, huzhao, james.cuzella, jbadiapa, jgreguske, lars, leiwang, linl, mmagr, mrunge, rmccabe, sacpatil, ssigwald, ushkalim, vkuznets, wshi, yuxisun
Target Milestone: rcKeywords: EC2, Regression, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: cloud-init-0.7.9-13.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 14:05:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1451548, 1456511    
Attachments:
Description Flags
nvidia-installer_on_3.10.0-705.el7.x86_64.log
none
nvidia-installer_on_3.10.0-706.el7.x86_64.log none

Description Chen Shi 2017-06-28 04:27:27 UTC
Description of problem:
The network can't be reached after creating the instance. And this instance was created from an AMI which includes cloud-init-0.7.9-8.el7.x86_64 package.


Version-Release number of selected component (if applicable):
cloud-init-0.7.9-4.el7.x86_64-> PASS
cloud-init-0.7.9-6.el7.x86_64-> PASS
cloud-init-0.7.9-8.el7.x86_64-> FAIL
cloud-init-0.7.9-9.el7.x86_64-> FAIL

How reproducible:
100%

Steps to Reproduce:
1. Login to the AWS, in Tokyo region.
2. Create an instance from RHEL-7.4_HVM_Beta-20170518-x86_64-1-Hourly2-GP2 (ami-31261f56)
3. Upgrade cloud-init-0.7.9-8.el7.x86_64 package.
4. Create an AMI from this instance, make sure the AMI contain cloud-init-0.7.9-8.el7.x86_64 package.
5. Create an instance from this AMI.
6. This instance can be booted up but the network can't be reached.

Actual results:
This instance can be booted up but the network can't be reached.

Expected results:
This instance can be booted up and the network can be reached.

Additional info:
1. This problem can be reproduced regardless the AMI was created via web or cli.
2. This problem can be reproduced on other region.
3. Both Public DNS/IP and Private DNS/IP can't be reached.

Comment 3 Chen Shi 2017-06-28 11:30:02 UTC
Some useful information from /var/log/messages:

```
Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643617.9597] device (lo): link connected
Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643617.9621] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Jun 28 09:53:37 ip-172-31-0-87 systemd: Started Hostname Service.
Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643617.9680] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643617.9723] device (eth0): state change: unmanaged -> unavailable (reason 'managed') [10 20 2]
Jun 28 09:53:37 ip-172-31-0-87 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jun 28 09:53:37 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643617.9836] device (eth0): link connected
Jun 28 09:53:38 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643618.0186] device (eth0): state change: unavailable -> disconnected (reason 'none') [20 30 0]
Jun 28 09:53:38 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643618.0219] manager: startup complete
Jun 28 09:53:38 ip-172-31-0-87 systemd: Started Network Manager Wait Online.
Jun 28 09:53:38 ip-172-31-0-87 systemd: Starting LSB: Bring up/down networking...
Jun 28 09:53:38 ip-172-31-0-87 network: Bringing up loopback interface:  [  OK  ]
Jun 28 09:53:38 ip-172-31-0-87 NetworkManager[480]: <info>  [1498643618.5053] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" result="fail" reason="No suitable device found for this connection."
Jun 28 09:53:38 ip-172-31-0-87 network: Bringing up interface eth0:  Error: Connection activation failed: No suitable device found for this connection.
Jun 28 09:53:38 ip-172-31-0-87 network: [FAILED]
Jun 28 09:53:38 ip-172-31-0-87 systemd: network.service: control process exited, code=exited status=1
Jun 28 09:53:38 ip-172-31-0-87 systemd: Failed to start LSB: Bring up/down networking.
Jun 28 09:53:38 ip-172-31-0-87 systemd: Unit network.service entered failed state.
Jun 28 09:53:38 ip-172-31-0-87 systemd: network.service failed.
```

We added scripts to rc.local, and get the following information during system boot:

$ ifconfig -a

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 06:9e:0f:f7:c4:65  txqueuelen 1000  (Ethernet)
        RX packets 2  bytes 140 (140.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
......

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0

BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=06:91:96:e1:6e:e7
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

$ cat /etc/udev/rules.d/70-persistent-net.rules

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="06:91:96:e1:6e:e7", NAME="eth0"


We believe that mismatched MAC address in configure file caused "eth0" boot up failure. (We also checked with "cloud-init-0.7.9-6.el7.x86_64", they are same MAC address)

Comment 4 Lars Kellogg-Stedman 2017-06-28 15:49:21 UTC
I believe that the problem is due to the way you are creating the new AMI in step 4:

> 4. Create an AMI from this instance, make sure the AMI contain cloud-init-0.7.9-8.el7.x86_64 package.

It doesn't appear that you are running "virt-sysprep" or otherwise attempting to de-configure the instance before generating the AMI.  Minimally, you will need to remove the HWADDR line from /etc/sysconfig/network-scripts/ifcfg-eth0, and you will need to remove the corresponding udev rule in /etc/udev/rules.d/70-persistent-net.rules.

Running "virt-sysprep" should accomplish both tasks for you.

If you are still experiencing a problem after implementing these steps, please update this bug with details.

Comment 6 Chen Shi 2017-06-29 09:16:28 UTC
(In reply to Lars Kellogg-Stedman from comment #4)
> I believe that the problem is due to the way you are creating the new AMI in
> step 4:
> 
> > 4. Create an AMI from this instance, make sure the AMI contain cloud-init-0.7.9-8.el7.x86_64 package.
> 
> It doesn't appear that you are running "virt-sysprep" or otherwise
> attempting to de-configure the instance before generating the AMI. 
> Minimally, you will need to remove the HWADDR line from
> /etc/sysconfig/network-scripts/ifcfg-eth0, and you will need to remove the
> corresponding udev rule in /etc/udev/rules.d/70-persistent-net.rules.
> 
> Running "virt-sysprep" should accomplish both tasks for you.
> 
> If you are still experiencing a problem after implementing these steps,
> please update this bug with details.

Hi Lars,

  Thank you for the inputs. I think this procedure should be done by AWS, and I have no idea about "virt-sysprep", is this used to deal with the AMI? We can't download and even not to download the AMI, deal with it, then upload it back to AWS. So what is the usage of "virt-sysprep" with AWS EC2?

Thanks,
Charles

Comment 9 Yuxin Sun 2017-07-20 09:49:53 UTC
Hi,

I tried to reproduce it in Azure and it seems that if I update cloud-init package, the service is not active during booting phase. I think it might be the root cause of this issue.

My steps:
1. Update cloud-init from 0.7.9-3 to 0.7.9-9 through rpm -Uvh
2. Ensure the services are enabled:
# systemctl is-enabled cloud-{init-local,init,config,final}
enabled
enabled
enabled
enabled
3. Reboot
4. Check /var/log/cloud-init.log

Actual result:
There's no new log in cloud-init.log
Then check /var/log/messages, there's no cloud-init related messages

Debug:
Compare the files in 0.7.9-3 and 0.7.9-9, the 
/usr/lib/systemd/system/cloud-init.target
/usr/lib/systemd/system-generators/cloud-init-generator 
2 files are not in the 0.7.9-9. 

It seems that in old version(0.7.9-3) it put cloud-*.services into /etc/systemd/system/cloud-init.target.wants/ folder, but not in multi-user.target.wants/. Then use che cloud-init.target to run cloud-init related services after multi-user.target. The cloud-init.target needs cloud-init-generator to enable itself.

# ll /etc/systemd/system/cloud-init.target.wants/
total 0
lrwxrwxrwx. 1 root root 44 Jul 20 09:24 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service
lrwxrwxrwx. 1 root root 43 Jul 20 09:24 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service
lrwxrwxrwx. 1 root root 48 Jul 20 09:24 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service
lrwxrwxrwx. 1 root root 42 Jul 20 09:24 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service

[root@wala73cloud0793 ~]# ll /usr/lib/systemd/system/cloud-init.target
-rw-r--r--. 1 root root 255 Dec 23  2016 /usr/lib/systemd/system/cloud-init.target
[root@wala73cloud0793 ~]# ll /usr/lib/systemd/system-generators/cloud-init-generator 
-rwxr-xr-x. 1 root root 3972 Dec 23  2016 /usr/lib/systemd/system-generators/cloud-init-generator

# cat /usr/lib/systemd/system/cloud-init.target
# cloud-init target is enabled by cloud-init-generator
# To disable it you can either:
#  a.) boot with kernel cmdline of 'cloudinit=disabled'
#  b.) touch a file /etc/cloud/cloud-init.disabled
[Unit]
Description=Cloud-init target
After=multi-user.target

In v0.7.9-9, if fresh install, cloud-*.service are in multi-user.target.wants/, so the cloud-init.target and cloud-init-generator are not needed anymore. 

# ll /etc/systemd/system/multi-user.target.wants/|grep cloud
lrwxrwxrwx. 1 root root 44 Jul 20 11:26 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service
lrwxrwxrwx. 1 root root 43 Jul 20 11:26 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service
lrwxrwxrwx. 1 root root 48 Jul 20 11:26 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service
lrwxrwxrwx. 1 root root 42 Jul 20 11:26 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service


But if upgrade from v0.7.9-3 to 0.7.9-9, the cloud-init related services are not re-enabled, so the .service soft links are still in /etc/systemd/system/cloud-init.target.wants/. But the cloud-init.target and cloud-init-generator files are removed. So systemd cannot run cloud-init related services.

[root@wala73cloud0793bak ~]# ls /usr/lib/systemd/system/cloud-init.target 
ls: cannot access /usr/lib/systemd/system/cloud-init.target: No such file or directory
[root@wala73cloud0793bak ~]# ls /usr/lib/systemd/system-generators/cloud-init-generator
ls: cannot access /usr/lib/systemd/system-generators/cloud-init-generator: No such file or directory
[root@wala73cloud0793bak ~]# ll /etc/systemd/system/multi-user.target.wants/|grep cloud
[root@wala73cloud0793bak ~]# ll /etc/systemd/system/cloud-init.target.wants/
total 0
lrwxrwxrwx. 1 root root 44 Jul 20 09:24 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service
lrwxrwxrwx. 1 root root 43 Jul 20 09:24 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service
lrwxrwxrwx. 1 root root 48 Jul 20 09:24 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service
lrwxrwxrwx. 1 root root 42 Jul 20 09:24 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service

Comment 10 Chen Shi 2017-08-04 05:30:49 UTC
As verified on AWS with RHEL7.4 RC1, we can see "cloud-init services are not re-enabled" is the root cause for issue "Bring up networking failed".

And for RHEL-7.4_HVM_GA-20170724-x86_64-1-Hourly2-GP2 (ami-3901e15f), the .service soft links are already in multi-user.target.wants . So upgrading cloud-init based on RHEL7.4 GA will have no issue.

RHEL7.4 GA:
[ec2-user@ip-172-31-9-0 system]$ uname -a
Linux ip-172-31-9-0.ap-northeast-1.compute.internal 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[ec2-user@ip-172-31-9-0 system]$ rpm -qa | grep cloud-init
cloud-init-0.7.9-9.el7.x86_64
[ec2-user@ip-172-31-9-0 system]$ find . | grep cloud
./multi-user.target.wants/cloud-config.service
./multi-user.target.wants/cloud-final.service
./multi-user.target.wants/cloud-init.service
./multi-user.target.wants/cloud-init-local.service
[ec2-user@ip-172-31-9-0 system]$ 


As a conclusion, customer will encounter this issue when:
1. Upgrading cloud-init based on RHEL-7.4_HVM_Beta-20170518-x86_64-1-Hourly2-GP2 - ami (or earlier version); AND,
2. Make an AMI for further use.

Workaround: execute the following commands before making the AMI:
# systemctl disable cloud-{init-local,init,config,final}.service
# systemctl enable cloud-{init-local,init,config,final}.service
make sure the .service soft links are located in multi-user.target.wants .

Comment 15 Chen Shi 2018-02-02 06:54:54 UTC
Created attachment 1389947 [details]
nvidia-installer_on_3.10.0-705.el7.x86_64.log

Comment 16 Chen Shi 2018-02-02 06:55:42 UTC
Created attachment 1389948 [details]
nvidia-installer_on_3.10.0-706.el7.x86_64.log

Comment 17 Chen Shi 2018-02-02 06:57:55 UTC
Comment on attachment 1389947 [details]
nvidia-installer_on_3.10.0-705.el7.x86_64.log

the attachment is not for this bug

Comment 18 Chen Shi 2018-02-02 06:58:15 UTC
Comment on attachment 1389948 [details]
nvidia-installer_on_3.10.0-706.el7.x86_64.log

the attachment is not for this bug

Comment 23 errata-xmlrpc 2018-04-10 14:05:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0806

Comment 24 James 2019-10-16 16:16:57 UTC
The package providing "virt-sysprep" seems unnecessarily bulky given that it has so many dependencies.  Does RedHat really expect all cloud users to install 116 extra packages just to fix networking for AMI based instances!?

This is a breaking change to cloud-init which causes networking to fail during boot.  This will affect any instance using AMIs containing the hardcoded MAC address of the virtual machine used to generate the instance.  It's a pretty big deal to break networking for any company depending on an image creation process for a cloud instance.  Many cloud providers such as AWS do not provide an easy way to debug such issues other than the boot console output.

For those searching for "virt-sysprep", it's provided by the "libguestfs-tools" package.  This package depends on a whopping 116 other extra packages!  Seems like a waste of space on a new AMI, just to fix networking, something that was previously working fine.



$ yum whatprovides virt-sysprep
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.advancedhosters.com
 * epel: epel.mirror.constant.com
 * extras: mirrors.advancedhosters.com
 * updates: mirrors.advancedhosters.com
epel/x86_64/filelists_db                                                                                                                                                                 |  11 MB  00:00:02
1:libguestfs-tools-c-1.40.2-5.el7.x86_64 : System administration tools for virtual machines
Repo        : base
Matched from:
Filename    : /usr/bin/virt-sysprep



1:libguestfs-tools-c-1.40.2-5.el7_7.1.x86_64 : System administration tools for virtual machines
Repo        : updates
Matched from:
Filename    : /usr/bin/virt-sysprep


$ sudo yum install libguestfs-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink                                                                                                                                                                     |  16 kB  00:00:00
 * base: repos-va.psychz.net
 * epel: iad.mirror.rackspace.com
 * extras: repos-va.psychz.net
 * updates: repos-va.psychz.net
base                                                                                                                                                                                     | 3.6 kB  00:00:00
epel                                                                                                                                                                                     | 5.3 kB  00:00:00
extras                                                                                                                                                                                   | 2.9 kB  00:00:00
updates                                                                                                                                                                                  | 2.9 kB  00:00:00
(1/2): epel/x86_64/updateinfo                                                                                                                                                            | 1.0 MB  00:00:00
(2/2): epel/x86_64/primary_db                                                                                                                                                            | 6.9 MB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package libguestfs-tools.noarch 1:1.40.2-5.el7_7.1 will be installed
--> Processing Dependency: libguestfs-tools-c = 1:1.40.2-5.el7_7.1 for package: 1:libguestfs-tools-1.40.2-5.el7_7.1.noarch
--> Processing Dependency: libguestfs = 1:1.40.2-5.el7_7.1 for package: 1:libguestfs-tools-1.40.2-5.el7_7.1.noarch

[...SNIP...]

Dependencies Resolved

================================================================================================================================================================================================================
 Package                                                             Arch                                 Version                                                   Repository                             Size
================================================================================================================================================================================================================
Installing:
 libguestfs-tools                                                    noarch                               1:1.40.2-5.el7_7.1                                        updates                                28 k
Installing for dependencies:
 alsa-lib                                                            x86_64                               1.1.8-1.el7                                               base                                  425 k
 attr                                                                x86_64                               2.4.46-13.el7                                             base                                   66 k
 augeas-libs                                                         x86_64                               1.4.0-9.el7                                               base                                  356 k
 boost-iostreams                                                     x86_64                               1.53.0-27.el7                                             base                                   61 k
 boost-random                                                        x86_64                               1.53.0-27.el7                                             base                                   39 k
 boost-system                                                        x86_64                               1.53.0-27.el7                                             base                                   40 k
 boost-thread                                                        x86_64                               1.53.0-27.el7                                             base                                   57 k
 bridge-utils                                                        x86_64                               1.5-9.el7                                                 base                                   32 k
 celt051                                                             x86_64                               0.5.1.3-8.el7                                             base                                   53 k
 cryptsetup                                                          x86_64                               2.0.3-5.el7                                               base                                  154 k
 cyrus-sasl-gssapi                                                   x86_64                               2.1.26-23.el7                                             base                                   41 k
 device-mapper-event                                                 x86_64                               7:1.02.158-2.el7                                          base                                  189 k
 device-mapper-event-libs                                            x86_64                               7:1.02.158-2.el7                                          base                                  189 k
 device-mapper-persistent-data                                       x86_64                               0.8.5-1.el7                                               base                                  423 k
 dnsmasq                                                             x86_64                               2.76-9.el7                                                base                                  278 k
 dosfstools                                                          x86_64                               3.0.20-10.el7                                             base                                  101 k
 ebtables                                                            x86_64                               2.0.10-16.el7                                             base                                  123 k
 flac-libs                                                           x86_64                               1.3.0-5.el7_1                                             base                                  169 k
 fuse                                                                x86_64                               2.9.2-11.el7                                              base                                   86 k
 fuse-libs                                                           x86_64                               2.9.2-11.el7                                              base                                   93 k
 gdisk                                                               x86_64                               0.8.10-3.el7                                              base                                  190 k
 genisoimage                                                         x86_64                               1.1.11-25.el7                                             base                                  299 k
 glusterfs                                                           x86_64                               3.12.2-47.2.el7                                           base                                  512 k
 glusterfs-api                                                       x86_64                               3.12.2-47.2.el7                                           base                                   77 k
 glusterfs-cli                                                       x86_64                               3.12.2-47.2.el7                                           base                                  179 k
 glusterfs-client-xlators                                            x86_64                               3.12.2-47.2.el7                                           base                                  883 k
 glusterfs-libs                                                      x86_64                               3.12.2-47.2.el7                                           base                                  387 k
 gnutls                                                              x86_64                               3.3.29-9.el7_6                                            base                                  680 k
 gperftools-libs                                                     x86_64                               2.6.1-1.el7                                               base                                  272 k
 gsm                                                                 x86_64                               1.0.13-11.el7                                             base                                   30 k
 hexedit                                                             x86_64                               1.2.13-5.el7                                              base                                   39 k
 hivex                                                               x86_64                               1.3.10-6.9.el7                                            base                                  101 k
 ipxe-roms-qemu                                                      noarch                               20180825-2.git133f4c.el7                                  base                                  791 k
 iscsi-initiator-utils                                               x86_64                               6.2.0.874-11.el7                                          base                                  422 k
 iscsi-initiator-utils-iscsiuio                                      x86_64                               6.2.0.874-11.el7                                          base                                   92 k
 libICE                                                              x86_64                               1.0.9-9.el7                                               base                                   66 k
 libSM                                                               x86_64                               1.2.2-2.el7                                               base                                   39 k
 libX11                                                              x86_64                               1.6.7-2.el7                                               base                                  607 k
 libX11-common                                                       noarch                               1.6.7-2.el7                                               base                                  164 k
 libXau                                                              x86_64                               1.0.8-2.1.el7                                             base                                   29 k
 libXext                                                             x86_64                               1.3.3-3.el7                                               base                                   39 k
 libXi                                                               x86_64                               1.7.9-1.el7                                               base                                   40 k
 libXtst                                                             x86_64                               1.2.3-1.el7                                               base                                   20 k
 libaio                                                              x86_64                               0.3.109-13.el7                                            base                                   24 k
 libasyncns                                                          x86_64                               0.8-7.el7                                                 base                                   26 k
 libconfig                                                           x86_64                               1.4.9-5.el7                                               base                                   59 k
 libguestfs                                                          x86_64                               1:1.40.2-5.el7_7.1                                        updates                               2.4 M
 libguestfs-tools-c                                                  x86_64                               1:1.40.2-5.el7_7.1                                        updates                               4.9 M
 libibverbs                                                          x86_64                               22.1-3.el7                                                base                                  267 k
 libiscsi                                                            x86_64                               1.9.0-7.el7                                               base                                   60 k
 libjpeg-turbo                                                       x86_64                               1.2.90-8.el7                                              base                                  135 k
 libogg                                                              x86_64                               2:1.3.0-7.el7                                             base                                   24 k
 libpciaccess                                                        x86_64                               0.14-1.el7                                                base                                   26 k
 librados2                                                           x86_64                               1:10.2.5-4.el7                                            base                                  1.8 M
 librbd1                                                             x86_64                               1:10.2.5-4.el7                                            base                                  2.4 M
 librdmacm                                                           x86_64                               22.1-3.el7                                                base                                   63 k
 libreport-filesystem                                                x86_64                               2.1.11-43.el7.centos                                      base                                   40 k
 libsndfile                                                          x86_64                               1.0.25-10.el7                                             base                                  149 k
 libusal                                                             x86_64                               1.1.11-25.el7                                             base                                  136 k
 libusbx                                                             x86_64                               1.0.21-1.el7                                              base                                   61 k
 libvirt-daemon                                                      x86_64                               4.5.0-23.el7_7.1                                          updates                               841 k
 libvirt-daemon-driver-interface                                     x86_64                               4.5.0-23.el7_7.1                                          updates                               236 k
 libvirt-daemon-driver-network                                       x86_64                               4.5.0-23.el7_7.1                                          updates                               410 k
 libvirt-daemon-driver-nodedev                                       x86_64                               4.5.0-23.el7_7.1                                          updates                               236 k
 libvirt-daemon-driver-nwfilter                                      x86_64                               4.5.0-23.el7_7.1                                          updates                               260 k
 libvirt-daemon-driver-qemu                                          x86_64                               4.5.0-23.el7_7.1                                          updates                               745 k
 libvirt-daemon-driver-secret                                        x86_64                               4.5.0-23.el7_7.1                                          updates                               226 k
 libvirt-daemon-driver-storage                                       x86_64                               4.5.0-23.el7_7.1                                          updates                               197 k
 libvirt-daemon-driver-storage-core                                  x86_64                               4.5.0-23.el7_7.1                                          updates                               435 k
 libvirt-daemon-driver-storage-disk                                  x86_64                               4.5.0-23.el7_7.1                                          updates                               227 k
 libvirt-daemon-driver-storage-gluster                               x86_64                               4.5.0-23.el7_7.1                                          updates                               235 k
 libvirt-daemon-driver-storage-iscsi                                 x86_64                               4.5.0-23.el7_7.1                                          updates                               225 k
 libvirt-daemon-driver-storage-logical                               x86_64                               4.5.0-23.el7_7.1                                          updates                               229 k
 libvirt-daemon-driver-storage-mpath                                 x86_64                               4.5.0-23.el7_7.1                                          updates                               224 k
 libvirt-daemon-driver-storage-rbd                                   x86_64                               4.5.0-23.el7_7.1                                          updates                               230 k
 libvirt-daemon-driver-storage-scsi                                  x86_64                               4.5.0-23.el7_7.1                                          updates                               225 k
 libvirt-daemon-kvm                                                  x86_64                               4.5.0-23.el7_7.1                                          updates                               197 k
 libvirt-libs                                                        x86_64                               4.5.0-23.el7_7.1                                          updates                               4.2 M
 libvorbis                                                           x86_64                               1:1.3.3-8.el7.1                                           base                                  204 k
 libxcb                                                              x86_64                               1.13-1.el7                                                base                                  214 k
 libxslt                                                             x86_64                               1.1.28-5.el7                                              base                                  242 k
 lsscsi                                                              x86_64                               0.27-6.el7                                                base                                   47 k
 lvm2                                                                x86_64                               7:2.02.185-2.el7                                          base                                  1.3 M
 lvm2-libs                                                           x86_64                               7:2.02.185-2.el7                                          base                                  1.1 M
 lzop                                                                x86_64                               1.03-10.el7                                               base                                   54 k
 mdadm                                                               x86_64                               4.1-1.el7                                                 base                                  435 k
 mtools                                                              x86_64                               4.0.18-5.el7                                              base                                  203 k
 netcf-libs                                                          x86_64                               0.2.8-4.el7                                               base                                   70 k
 nettle                                                              x86_64                               2.7.1-8.el7                                               base                                  327 k
 numad                                                               x86_64                               0.5-18.20150602git.el7                                    base                                   35 k
 opus                                                                x86_64                               1.0.2-6.el7                                               base                                  630 k
 osinfo-db                                                           noarch                               20190319-2.el7                                            base                                  191 k
 pciutils                                                            x86_64                               3.5.1-3.el7                                               base                                   93 k
 perl-Sys-Guestfs                                                    x86_64                               1:1.40.2-5.el7_7.1                                        updates                               306 k
 perl-Sys-Virt                                                       x86_64                               4.5.0-2.el7                                               base                                  296 k
 perl-hivex                                                          x86_64                               1.3.10-6.9.el7                                            base                                   41 k
 perl-libintl                                                        x86_64                               1.20-12.el7                                               base                                  875 k
 pixman                                                              x86_64                               0.34.0-1.el7                                              base                                  248 k
 psmisc                                                              x86_64                               22.20-16.el7                                              base                                  141 k
 pulseaudio-libs                                                     x86_64                               10.0-5.el7                                                base                                  651 k
 qemu-img                                                            x86_64                               10:1.5.3-167.el7_7.1                                      updates                               699 k
 qemu-kvm                                                            x86_64                               10:1.5.3-167.el7_7.1                                      updates                               1.9 M
 qemu-kvm-common                                                     x86_64                               10:1.5.3-167.el7_7.1                                      updates                               435 k
 radvd                                                               x86_64                               2.17-3.el7                                                base                                   94 k
 rdma-core                                                           x86_64                               22.1-3.el7                                                base                                   50 k
 scrub                                                               x86_64                               2.5.2-7.el7                                               base                                   41 k
 seabios-bin                                                         noarch                               1.11.0-2.el7                                              base                                  112 k
 seavgabios-bin                                                      noarch                               1.11.0-2.el7                                              base                                   38 k
 sgabios-bin                                                         noarch                               1:0.20110622svn-4.el7                                     base                                  7.1 k
 spice-server                                                        x86_64                               0.14.0-7.el7                                              base                                  404 k
 squashfs-tools                                                      x86_64                               4.3-0.21.gitaae0aff4.el7                                  base                                  101 k
 supermin5                                                           x86_64                               5.1.19-1.el7                                              base                                  637 k
 syslinux                                                            x86_64                               4.05-15.el7                                               base                                  990 k
 syslinux-extlinux                                                   x86_64                               4.05-15.el7                                               base                                  364 k
 trousers                                                            x86_64                               0.3.14-2.el7                                              base                                  289 k
 usbredir                                                            x86_64                               0.7.1-3.el7                                               base                                   47 k

Transaction Summary
================================================================================================================================================================================================================
Install  1 Package (+116 Dependent packages)

Total download size: 44 M
Installed size: 143 M
Is this ok [y/d/N]: N

Comment 25 James 2019-10-18 15:47:52 UTC
Ok, so it wasn't clear at first that "virt-sysprep" tool is intended to be used on a disk image file, not a running system.  Given the following:

    $ /usr/bin/virt-sysprep --dry-run --enable udev-persistent-net  --verbose
    virt-sysprep: error: you must give either -a or -d options.  Read
    virt-sysprep(1) man page for further information.

    If reporting bugs, run virt-sysprep with debugging enabled and include the
    complete output:

      virt-sysprep -v -x [...]

Therefore only these options are valid:

    -a, --add <file>                    Add disk image file
    -d, --domain <domain>               Set libvirt guest name

So, given that this bug affects AWS EC2 and presumably any cloud instance using CentOS / RHEL 7 + cloud-init, we are unable to use "virt-sysprep" here.

The other workaround seems to be:

    sudo yum install cloud-init
    sudo systemctl disable cloud-{init-local,init,config,final}.service
    sudo systemctl enable cloud-{init-local,init,config,final}.service
    sudo -n sed -i '/HWADDR/d' /etc/sysconfig/network-scripts/ifcfg-eth* /etc/sysconfig/network-scripts/ifcfg-ens*
    sudo -n sed -i '/UUID/d' /etc/sysconfig/network-scripts/ifcfg-eth* /etc/sysconfig/network-scripts/ifcfg-ens*

Then removing the MAC address also from udev rules file: /etc/udev/rules.d/70-persistent-net.rules

For example:

    # BEFORE
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="01:12:23:34:45:56", KERNEL=="ens*", NAME="ens3"
    # AFTER
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{dev_id}=="0x0", KERNEL=="ens*", ATTR{type}=="1", NAME="ens3"