Bug 1558641
| Summary: | cloud-init creates bogus metadata route preventing metadata setup | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Kevin Fenzi <kevin> | ||||
| Component: | cloud-init | Assignee: | Garrett Holmstrom <gholms> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 28 | CC: | adimania, apevec, awilliam, gholms, gmarr, kevin, lars, mboddu, puiterwijk, robatino, sgallagh, shardy, s | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | AcceptedBlocker | ||||||
| Fixed In Version: | cloud-init-17.1-4.fc28 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-03-29 19:22:03 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1469204 | ||||||
| Attachments: |
|
||||||
|
Description
Kevin Fenzi
2018-03-20 16:34:01 UTC
Proposed as a Blocker for 28-beta by Fedora user kevin using the blocker tracking app because: "The cloud-init package must be functional for release blocking cloud images. " +1 blocker, obviously hits the criterion right on the nose. cloud-init has a disable_metadata switch one can use to block access to the EC2 metadata service, but it shouldn't be enabled by default and it *definitely* shouldn't be applied this early in the boot process. Do you happen to have the rest of the VM's boot-time output handy? Created attachment 1410821 [details]
boot logs from cloud instance
Here's the logs...
+1 blocker if this is indeed true for all installations. If it turns out that it's limited to old versions of OpenStack, I'll revise that. Agreed with Stephen, there, I'm +1 assuming it's a general failure (has anyone tested EC2 yet?) Has anyone tested with other/newer clouds, if its happening in every instance then +1 Blocker That's +3, setting accepted - if it's shown that this doesn't happen on other clouds, I will drop accepted status for a revote. The route also gets added on EC2, although due to networking differences between openstack and EC2, it does not actually break the metadata gathering there. Regardless, this means that the networking behaviour changed and it breaks on clouds that aren't setup for these routes. Cloud image from Fedora-28-20180321.n.0: Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++ Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: +-------+-------------+------------+---------------+-----------+-------+ Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: +-------+-------------+------------+---------------+-----------+-------+ Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: | 0 | 0.0.0.0 | 172.30.2.1 | 0.0.0.0 | eth0 | UG | Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: | 1 | 169.254.0.0 | 0.0.0.0 | 255.255.0.0 | eth0 | U | Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: | 2 | 172.30.2.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U | Mar 21 22:52:14 ip-172-30-2-30.ec2.internal cloud-init[957]: ci-info: +-------+-------------+------------+---------------+-----------+-------+ Current live Fedora 27 cloud image: Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++ Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: +-------+-------------+------------+---------------+-----------+-------+ Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: +-------+-------------+------------+---------------+-----------+-------+ Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: | 0 | 0.0.0.0 | 172.30.2.1 | 0.0.0.0 | eth0 | UG | Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: | 1 | 172.30.2.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U | Mar 21 22:56:42 ip-172-30-2-41.ec2.internal cloud-init[825]: ci-info: +-------+-------------+------------+---------------+-----------+-------+ So, after some debug images and poking around I am pretty sure the bug is this:
- cloud-init overwrites our /etc/sysconfig/network file with:
# Created by cloud-init on instance boot automatically, do not edit.
#
NETWORKING=yes
- This means that 2 lines we add in the kickstart are gone:
NOZEROCONF=yes
DEVTIMEOUT=10
- network starts, NOZEROCONF is not set so (in ifup-eth you can see):
# Add Zeroconf route.
if [ -z "${NOZEROCONF}" -a "${ISALIAS}" = "no" -a "${REALDEVICE}" != "lo" ]; then
ip route add 169.254.0.0/16 dev ${REALDEVICE} metric $((1000 + $(cat /sys/class/net/${REALDEVICE}/ifindex))) scope link
fi
- Now the metadata route is hosed.
So, how are we gonna fix it? cloud-init-17.1-3.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-86137b2be8 The fix for this was pulled into the Beta-1.1 (Beta RC1) compose. Can anyone confirm the fix in the Cloud images from that compose? Thanks. cloud-init-17.1-4.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-86137b2be8 I tested: http://dl.fedoraproject.org/pub/alt/stage/28_Beta-1.1/Cloud/x86_64/images/Fedora-Cloud-Base-28_Beta-1.1.x86_64.qcow2 and http://dl.fedoraproject.org/pub/alt/stage/28_Beta-1.1/Cloud/ppc64le/images/Fedora-Cloud-Base-28_Beta-1.1.ppc64le.qcow2 and they boot fine and I can login to them. So, fix works here as far as I can tell. Tested with Fedora-Cloud-Base-28_Beta-1.1.x86_64 on EC2 and locally with testcloud, boot and login work OK! cloud-init-17.1-4.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. |