Bug 1644692

Summary: f29/f30 unusable on gce
Product: [Fedora] Fedora Reporter: Maciej Żenczykowski <zenczykowski>
Component: cloud-initAssignee: Dusty Mabe <dustymabe>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: adimania, apevec, david_paterson, eterrell, gholms, lars, shardy, s, yago.riveiro
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-26 16:49:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Maciej Żenczykowski 2018-10-31 12:30:16 UTC
Trying to bring up an f29 instance from scratch.
Using image from:

IMG_VER=29
IMG_DATE=1.2
IMG_ARCH=x86_64
IMG="Fedora-Cloud-Base-${IMG_VER}-${IMG_DATE}.${IMG_ARCH}"
IMG_SRC="https://dl.fedoraproject.org/pub/fedora/linux/releases/${IMG_VER}/Cloud/${IMG_ARCH}/images/${IMG}.raw.xz"

serial console log shows:

         Starting Network Manager...
[   18.209735] cloud-init[523]: Cloud-init v. 17.1 running 'init' at Wed, 31 Oct 2018 12:22:03 +0000. Up 17.97 seconds.
[   18.223277] cloud-init[523]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[   18.236239] cloud-init[523]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   18.249238] cloud-init[523]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
[   18.262238] cloud-init[523]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   18.275280] cloud-init[523]: ci-info: | eth0:  | False |     .     |     .     |   .   | 42:01:0a:01:00:04 |
[   18.288279] cloud-init[523]: ci-info: |  lo:   |  True | 127.0.0.1 | 255.0.0.0 |   .   |         .         |
[   18.301237] cloud-init[523]: ci-info: |  lo:   |  True |     .     |     .     |   d   |         .         |
[   18.314243] cloud-init[523]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[   18.328284] cloud-init[523]: 2018-10-31 12:22:03,619 - DataSourceGCE.py[WARNING]: address "http://metadata.google.internal/computeMetadata/v1/" is not resolvable
[   18.346540] cloud-init[523]: 2018-10-31 12:22:03,687 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f78e9206b38>: Failed to establish a new connection: [Errno 101] Network is unreachable'))]
[  OK  ] Started Network Manager.
[  OK  ] Reached target Network.
         Starting Network Manager Wait Online...
         Starting Hostname Service...
[  OK  ] Started Hostname Service.
         Starting Network Manager Script Dispatcher Service...
[  OK  ] Started Network Manager Script Dispatcher Service.
[   18.933693] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
         Starting Authorization Manager...
[   19.283647] cloud-init[523]: 2018-10-31 12:22:04,691 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f78e922c400>: Failed to establish a new connection: [Errno 101] Network is unreachable'))]
[  OK  ] Started Network Manager Wait Online.
[  OK  ] Started Authorization Manager.
[   20.292188] cloud-init[523]: 2018-10-31 12:22:05,702 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: bad status code [404]
[   21.299699] cloud-init[523]: 2018-10-31 12:22:06,709 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/120s]: bad status code [404]
...
[  137.695629] cloud-init[523]: 2018-10-31 12:24:03,105 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: unexpected error [Attempted to set connect timeout to 0.0, but the timeout cannot be set to a value less than or equal to 0.]
[  144.703352] cloud-init[523]: 2018-10-31 12:24:10,113 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 126 seconds
[  194.758810] cloud-init[523]: 2018-10-31 12:25:00,168 - url_helper.py[WARNING]: Calling 'http://10.1.0.1/latest/meta-data/instance-id' failed [50/120s]: request error [HTTPConnectionPool(host='10.1.0.1', port=80): Max retries exceeded with url: /latest/meta-data/instance-id (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f78e922c0b8>, 'Connection to 10.1.0.1 timed out. (connect timeout=50.0)'))]
[  245.815112] cloud-init[523]: 2018-10-31 12:25:51,224 - url_helper.py[WARNING]: Calling 'http://10.1.0.1/latest/meta-data/instance-id' failed [101/120s]: request error [HTTPConnectionPool(host='10.1.0.1', port=80): Max retries exceeded with url: /latest/meta-data/instance-id (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f78e9241198>, 'Connection to 10.1.0.1 timed out. (connect timeout=50.0)'))]
[  263.836924] cloud-init[523]: 2018-10-31 12:26:09,246 - url_helper.py[WARNING]: Calling 'http://10.1.0.1/latest/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='10.1.0.1', port=80): Max retries exceeded with url: /latest/meta-data/instance-id (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f78e92418d0>, 'Connection to 10.1.0.1 timed out. (connect timeout=17.0)'))]
[  264.838606] cloud-init[523]: 2018-10-31 12:26:10,248 - DataSourceCloudStack.py[CRITICAL]: Giving up on waiting for the metadata from ['http://10.1.0.1/latest/meta-data/instance-id'] after 120 seconds
[  266.720886] cloud-init[523]: /usr/lib/python3.7/site-packages/cloudinit/config/cc_rsyslog.py:205: FutureWarning: Possible nested set at position 23
[  266.737336] cloud-init[523]:   r'^(?P<proto>[@]{0,2})'
[  OK  ] Started Initial cloud-init job (metadata service crawler).

and we never pick up ssh keys so can't ssh in.
(also not clear how to login, what's the login name/password?)

Comment 1 Maciej Żenczykowski 2018-10-31 12:31:05 UTC
Oh, I guess I should say f28 instances dnf --releasever=29 distro-sync'ed are fine, although I haven't checked if there cloud-init works or not, since it's only really needed on first boot...

Comment 2 Maciej Żenczykowski 2018-10-31 12:34:16 UTC
(oh, machine does become ssh-reachable, just don't have any keys/passwords that would work, so network does come up, it's just metadata which fails)

Comment 3 Maciej Żenczykowski 2018-10-31 12:42:43 UTC
Possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=1398388

Comment 4 Maciej Żenczykowski 2018-10-31 12:47:01 UTC
Maybe this is using the entirely wrong way of fetching metadata...

wget -O - --header 'Metadata-Flavor: Google' http://169.254.169.254/computeMetadata/v1/?recursive=true

is more like it...

Comment 5 Maciej Żenczykowski 2018-10-31 12:52:38 UTC
Maybe relevant is that:

[   17.986291] cloud-init[515]: 2018-10-31 12:50:02,268 - DataSourceGCE.py[WARNING]: address "http://metadata.google.internal/computeMetadata/v1/" is not resolvable

appears to be printed out prior to network startup...

maybe cloud-init needs to wait for network manager to bring things up first?

Comment 6 Yago Riveiro 2019-05-03 21:33:53 UTC
Any idea how to fix this?

Fedora 30 has the same issue.

Comment 7 Maciej Żenczykowski 2019-05-04 22:26:23 UTC
(my workaround is to use a much older version of fedora (I think possibly 28) and then upgrade (via dnf --releasever=29 distro-sync) to 29+)

Comment 8 Ben Cotton 2020-04-30 21:14:47 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 9 Ben Cotton 2020-05-26 16:49:53 UTC
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.