Bug 1687241
| Summary: | [OSP] Sometimes bootstrap and api fail to dial openstack metadata server during booting even with correct security group | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | weiwei jiang <wjiang> |
| Component: | Installer | Assignee: | Tomas Sedovic <tsedovic> |
| Installer sub component: | openshift-installer | QA Contact: | Tomas Sedovic <tsedovic> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | bleanhar, eduen, mifiedle, wsun, xtian |
| Version: | 4.1.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-09 09:28:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
weiwei jiang
2019-03-11 03:16:18 UTC
This has been fixed upstream: https://github.com/openshift/installer/pull/1390 Unfortunately, I got bootstrap and api is working will this time, but masters got this isssue. [openshift@dhcp-140-70 installer]$ bin/openshift-install version bin/openshift-install unreleased-master-560-g974d9b0848866f03d4dd8c577d8b7ef28756a1d5-dirty built from commit 974d9b0848866f03d4dd8c577d8b7ef28756a1d5 [ 3.590429] iscsid[467]: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi [ 3.598874] iscsid[467]: can't open iscsid.safe_logout configuration file /etc/iscsi/iscsid.conf [ 3.995768] ignition[481]: GET http://169.254.169.254/openstack/latest/user_data: attempt #4 [ 4.000615] ignition[481]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 4.224919] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 5.596229] ignition[481]: GET http://169.254.169.254/openstack/latest/user_data: attempt #5 [ 5.601071] ignition[481]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 8.796609] ignition[481]: GET http://169.254.169.254/openstack/latest/user_data: attempt #6 [ 8.801560] ignition[481]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 11.453595] dhclient[1584]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 20 (xid=0x7e0ed866) [ 13.796972] ignition[481]: GET http://169.254.169.254/openstack/latest/user_data: attempt #7 [ 13.802551] ignition[481]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 18.797357] ignition[481]: GET http://169.254.169.254/openstack/latest/user_data: attempt #8 [ 18.802735] ignition[481]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 23.797705] ignition[481]: GET http://169.254.169.254/openstack/latest/user_data: attempt #9 [ 23.802374] ignition[481]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 28.798044] ignition[481]: GET http://169.254.169.254/openstack/latest/user_data: attempt #10 [ 28.802690] ignition[481]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 30.606979] dhclient[1584]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8 (xid=0x7e0ed866) [ 32.594697] ignition[481]: neither config drive nor metadata service were available in time. Continuing without a config... [ 32.600272] ignition[481]: failed to fetch config: not a config (empty) Checked again and met this for bootstrap and api this time [openshift@dhcp-140-70 installer]$ bin/openshift-install version bin/openshift-install unreleased-master-560-g974d9b0848866f03d4dd8c577d8b7ef28756a1d5 built from commit 974d9b0848866f03d4dd8c577d8b7ef28756a1d5 (openstack) server list --name wjiang +--------------------------------------+----------------------------+--------+-------------------------------------------------------+-------+----------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------------------------+--------+-------------------------------------------------------+-------+----------------+ | 78cdfb63-cd5e-4fc2-8f0c-e14be3e6d91f | wjiang-ocp-fvkd5-master-1 | ACTIVE | wjiang-ocp-fvkd5-openshift=192.168.0.11 | rhcos | ci.m1.medlarge | | 984783ba-5303-4d35-a26a-fe7e9b784e3d | wjiang-ocp-fvkd5-master-2 | ACTIVE | wjiang-ocp-fvkd5-openshift=192.168.0.5 | rhcos | ci.m1.medlarge | | 0f4812e9-9f18-4336-b1c8-5a356a90a8e1 | wjiang-ocp-fvkd5-master-0 | ACTIVE | wjiang-ocp-fvkd5-openshift=192.168.0.9 | rhcos | ci.m1.medlarge | | e4347058-6889-4dc7-a5ad-d98115e468f4 | wjiang-ocp-fvkd5-api | ACTIVE | wjiang-ocp-fvkd5-openshift=192.168.128.13, 10.0.77.71 | rhcos | ci.m1.medlarge | | bfb430a2-a6b7-4899-9117-6bb3bca7a181 | wjiang-ocp-fvkd5-bootstrap | ACTIVE | wjiang-ocp-fvkd5-openshift=192.168.0.10 | rhcos | ci.m1.medlarge | +--------------------------------------+----------------------------+--------+-------------------------------------------------------+-------+----------------+ part of boot log of wjiang-ocp-fvkd5-bootstrap: [ 5.147991] ignition[466]: GET http://169.254.169.254/openstack/latest/user_data: attempt #4 [ 5.154008] ignition[466]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 5.601541] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.748673] ignition[466]: GET http://169.254.169.254/openstack/latest/user_data: attempt #5 [ 6.755664] ignition[466]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 9.840795] dhclient[1583]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6 (xid=0x7a39d5d2) [ 9.949077] ignition[466]: GET http://169.254.169.254/openstack/latest/user_data: attempt #6 [ 9.955439] ignition[466]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 14.949788] ignition[466]: GET http://169.254.169.254/openstack/latest/user_data: attempt #7 [ 14.957273] ignition[466]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 15.455825] dhclient[1583]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 7 (xid=0x7a39d5d2) [ 19.950196] ignition[466]: GET http://169.254.169.254/openstack/latest/user_data: attempt #8 [ 19.957614] ignition[466]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 22.451638] dhclient[1583]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 9 (xid=0x7a39d5d2) [ 24.950728] ignition[466]: GET http://169.254.169.254/openstack/latest/user_data: attempt #9 [ 24.958736] ignition[466]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 29.951109] ignition[466]: GET http://169.254.169.254/openstack/latest/user_data: attempt #10 [ 29.959172] ignition[466]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 31.595989] dhclient[1583]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 19 (xid=0x7a39d5d2) [ 33.745242] ignition[466]: neither config drive nor metadata service were available in time. Continuing without a config... [ 33.754528] ignition[466]: timed out while fetching config from config drive (config-2) [[32m OK [0m] Started Ignition (disks). [ 33.762476] ignition[466]: timed out while fetching config from config drive (CONFIG-2) [ 33.768096] systemd[1]: Started Ignition (disks). [ 33.771753] ignition[466]: failed to fetch config: not a config (empty) [ 33.775737] ignition[466]: not a config (empty): ignoring user-provided config [ 33.781088] ignition[466]: reading system config file "/usr/lib/ignition/default.ign" [ 33.786469] ignition[466]: no config at "/usr/lib/ignition/default.ign" also wjiang-ocp-fvkd5-api [ 4.018893] ignition[468]: GET http://169.254.169.254/openstack/latest/user_data: attempt #4 [ 4.023585] ignition[468]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 4.216511] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 5.619149] ignition[468]: GET http://169.254.169.254/openstack/latest/user_data: attempt #5 [ 5.623783] ignition[468]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 8.819580] ignition[468]: GET http://169.254.169.254/openstack/latest/user_data: attempt #6 [ 8.824254] ignition[468]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 10.700005] dhclient[1582]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 13 (xid=0x48a379fd) [ 13.819940] ignition[468]: GET http://169.254.169.254/openstack/latest/user_data: attempt #7 [ 13.824619] ignition[468]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 18.820319] ignition[468]: GET http://169.254.169.254/openstack/latest/user_data: attempt #8 [ 18.824988] ignition[468]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 23.676860] dhclient[1582]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 (xid=0x48a379fd) [ 23.820812] ignition[468]: GET http://169.254.169.254/openstack/latest/user_data: attempt #9 [ 23.825591] ignition[468]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 28.821256] ignition[468]: GET http://169.254.169.254/openstack/latest/user_data: attempt #10 [ 28.828608] ignition[468]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 32.611715] ignition[468]: neither config drive nor metadata service were available in time. Continuing without a config... [ 32.617872] ignition[468]: failed to fetch config: not a config (empty) [[32m OK [0m] Started Ignition (disks). [ 32.622242] ignition[468]: not a config (empty): ignoring user-provided config [ 32.626316] systemd[1]: Started Ignition (disks). [ 32.628965] ignition[468]: reading system config file "/usr/lib/ignition/default.ign" [ 32.633212] ignition[468]: no config at "/usr/lib/ignition/default.ign" Might be a transient issue with Upshift. May also require a fix from CoreOS team to help mitigate time outs. Checked with 4.2.0-0.nightly-2019-07-31-162901 and also recent versions and we do not meet this anymore recently. Moving to ON_QA to since weiwei already verified. Move to verified, since it's more like a unstable upshift OpenStack issue. |