Bug 747692 - Networking startup failure on EC2, ami-7f5a063a (Cloud test day)
Summary: Networking startup failure on EC2, ami-7f5a063a (Cloud test day)
Alias: None
Product: Fedora
Classification: Fedora
Component: cloud-init
Version: 16
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Garrett Holmstrom
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2011-10-20 19:31 UTC by Adam Huffman
Modified: 2011-12-21 17:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2011-12-21 08:54:52 UTC

Attachments (Terms of Use)
cloud-init.log from a working instance (2.79 KB, text/plain)
2011-10-24 16:34 UTC, Garrett Holmstrom
no flags Details
System Log of F16 boot on EC2 (EU-West-1) (14.69 KB, text/plain)
2011-12-16 10:13 UTC, Sandro Mathys
no flags Details

Description Adam Huffman 2011-10-20 19:31:08 UTC
Description of problem:

Following the Cloud test day, I started ami-7f5a063a in US-West.

Networking didn't start properly, with the result that I could not connect to the instance.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
Actual results:

Expected results:

Additional info:

Fedora release 16 (Verne)
Kernel 3.1.0-0.rc6.git0.3.fc16.x86_64 on an x86_64 (hvc0)

localhost login: Bringing up loopback interface:  [  OK  ]

Bringing up interface eth0:  
Determining IP information for eth0.../etc/init.d/functions: line 58: /dev/stderr: Permission denied
/etc/rc.d/init.d/functions: line 58: /dev/stderr: Permission denied
/etc/init.d/functions: line 58: /dev/stderr: Permission denied
/etc/rc.d/init.d/functions: line 58: /dev/stderr: Permission denied
[  OK  ]

Comment 1 Adam Huffman 2011-10-20 20:07:03 UTC
Also happens with ami-d6fbc9a2 (EU-West) and ami-6d4d1128 (US-West).

Comment 2 Richard Marko 2011-10-21 13:22:27 UTC
Exactly the same output but no problem to connect to the instance.

Comment 3 Dennis Gilmore 2011-10-22 02:08:25 UTC
I suspect that this is actually an issue with the hypervisor. the output is misleading it actually happens on every boot. its also possible that cloud-init failed to get the keys and so login could not happen. im going to move it to there. i uploaded the exact same images to each of the zones. so failing in some instances is due to something intermittently going wonky. it could be ec2 it could be something in the guest, maybe a timing issue. anyways thats some random thoughts i had.

going to reassign to cloud-init because its the first thing we should look at on our side.

Comment 4 Garrett Holmstrom 2011-10-24 16:30:27 UTC
The error messages at the console appear to always occur.  They also appear to be benign.

I am having trouble reproducing this issue with either architecture.  Are you able to do so?  If so, could you please stop (not terminate) an affected instance, detach the / volume from it, attach that volume to a working instance, fetch /var/log/cloud-init.log from it, and attach that file to this bug?  The command you use to run broken instances may also be useful to know.

I will attach cloud-init.log from a working instance for comparison.

Comment 5 Garrett Holmstrom 2011-10-24 16:34:38 UTC
Created attachment 529925 [details]
cloud-init.log from a working instance

Comment 6 Adam Huffman 2011-10-24 16:53:31 UTC
Did you say during the test day that these AMIs should work with the Micro tier, just booting rather slowly?  If they don't work with Micro, then it will be difficult for me to test.  If they do, I'll try it.

Comment 7 Garrett Holmstrom 2011-10-24 17:01:12 UTC
Yes, the images should work with any instance type.  I failed to reproduce the issue with both t1.micro and m1.small.

Comment 8 Garrett Holmstrom 2011-11-28 21:23:23 UTC
I will close this bug for now.  If you manage to reproduce the issue then feel free to re-open it.

Comment 9 Sandro Mathys 2011-12-16 10:13:42 UTC
Created attachment 547713 [details]
System Log of F16 boot on EC2 (EU-West-1)

I just started an instance with the x86_64 image on a micro instance in EU-West-1 but when I try to connect over ssh, the connection times out.

Please find the bootup log attached.

Comment 10 Garrett Holmstrom 2011-12-16 18:38:19 UTC
Are you able to reproduce this issue consistently?  If so, what did you use to run the instance?  Did you supply any user-data to the instance?

The console log alone is of little use.  If possible, could you please post /var/log/cloud-init.log and anything from /var/log/messages that appears to be relevant?  You can access them by terminating the instance and attaching the EBS volume that contains the instance's / filesystem to another instance.

Comment 11 Sandro Mathys 2011-12-21 08:54:52 UTC
Okay, the problem clearly sits between keyboard and chair :) I thought the default security settings would be "all open" but it's actually "no connections allowed at all". So once you open 22/TCP everything's working fine.

Sorry for the trouble I've caused! ;) AWS documentation really sucks, at least for people new to EC2.

Note You need to log in before you can comment on or make changes to this bug.