Proposed as a Blocker for 28-beta by Fedora user kevin using the blocker tracking app because:

 "The cloud-init package must be functional for release blocking cloud images. "

+1 blocker, obviously hits the criterion right on the nose.

cloud-init has a disable_metadata switch one can use to block access to the EC2 metadata service, but it shouldn't be enabled by default and it *definitely* shouldn't be applied this early in the boot process.  Do you happen to have the rest of the VM's boot-time output handy?

Created attachment 1410821 [details]
boot logs from cloud instance

Here's the logs...

+1 blocker if this is indeed true for all installations. If it turns out that it's limited to old versions of OpenStack, I'll revise that.

Agreed with Stephen, there, I'm +1 assuming it's a general failure (has anyone tested EC2 yet?)

Has anyone tested with other/newer clouds, if its happening in every instance then

+1 Blocker

That's +3, setting accepted - if it's shown that this doesn't happen on other clouds, I will drop accepted status for a revote.

The route also gets added on EC2, although due to networking differences between openstack and EC2, it does not actually break the metadata gathering there.
Regardless, this means that the networking behaviour changed and it breaks on clouds that aren't setup for these routes.


Cloud image from Fedora-28-20180321.n.0:
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: | Route | Destination |  Gateway   |    Genmask    | Interface | Flags |
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: |   0   |   0.0.0.0   | 172.30.2.1 |    0.0.0.0    |    eth0   |   UG  |
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: |   1   | 169.254.0.0 |  0.0.0.0   |  255.255.0.0  |    eth0   |   U   |
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: |   2   |  172.30.2.0 |  0.0.0.0   | 255.255.255.0 |    eth0   |   U   |
Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: +-------+-------------+------------+---------------+-----------+-------+


Current live Fedora 27 cloud image:
Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: | Route | Destination |  Gateway   |    Genmask    | Interface | Flags |
Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: +-------+-------------+------------+---------------+-----------+-------+
Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: |   0   |   0.0.0.0   | 172.30.2.1 |    0.0.0.0    |    eth0   |   UG  |
Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: |   1   |  172.30.2.0 |  0.0.0.0   | 255.255.255.0 |    eth0   |   U   |
Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: +-------+-------------+------------+---------------+-----------+-------+

So, after some debug images and poking around I am pretty sure the bug is this: 

- cloud-init overwrites our /etc/sysconfig/network file with: 

# Created by cloud-init on instance boot automatically, do not edit.
#
NETWORKING=yes

- This means that 2 lines we add in the kickstart are gone: 

NOZEROCONF=yes
DEVTIMEOUT=10

- network starts, NOZEROCONF is not set so (in ifup-eth you can see): 
# Add Zeroconf route.
if [ -z "${NOZEROCONF}" -a "${ISALIAS}" = "no" -a "${REALDEVICE}" != "lo" ]; then
    ip route add 169.254.0.0/16 dev ${REALDEVICE} metric $((1000 + $(cat /sys/class/net/${REALDEVICE}/ifindex))) scope link
fi

- Now the metadata route is hosed.

So, how are we gonna fix it?

cloud-init-17.1-3.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-86137b2be8

The fix for this was pulled into the Beta-1.1 (Beta RC1) compose. Can anyone confirm the fix in the Cloud images from that compose? Thanks.

cloud-init-17.1-4.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-86137b2be8

I tested: 

http://dl.fedoraproject.org/pub/alt/stage/28_Beta-1.1/Cloud/x86_64/images/Fedora-Cloud-Base-28_Beta-1.1.x86_64.qcow2
and
http://dl.fedoraproject.org/pub/alt/stage/28_Beta-1.1/Cloud/ppc64le/images/Fedora-Cloud-Base-28_Beta-1.1.ppc64le.qcow2

and they boot fine and I can login to them. So, fix works here as far as I can tell.

Tested with Fedora-Cloud-Base-28_Beta-1.1.x86_64 on EC2 and locally with testcloud, boot and login work OK!

cloud-init-17.1-4.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.