Bug 1776155

Summary: OpenStack Config-Drive supportability
Product: OpenShift Container Platform Reporter: Juan Luis de Sousa-Valadas <jdesousa>
Component: InstallerAssignee: Mike Fedosin <mfedosin>
Installer sub component: openshift-installer QA Contact: Johnny Liu <jialiu>
Status: CLOSED WONTFIX Docs Contact:
Severity: urgent    
Priority: urgent CC: fgarciad, mfedosin, pprinett
Version: 4.2.z   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-14 17:10:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Juan Luis de Sousa-Valadas 2019-11-25 09:06:22 UTC
Description of problem:
Customer runs OCP 4.2 on openstack without metadata service.
Ignition doesn't take the configuration from config-drive, even though the disk is attached. RHEL instances have it attached on /dev/disk/by-label/config-2


Version-Release number of the following components:
Red Hat Enterprise Linux CoreOS - OpenStack Image (QCOW) 4.2.0
ignition-0.33.0-5.rhaos4.2.gitc65e95c.el8.x86_64

How reproducible:
Apparently always

Steps to Reproduce:
1. Deploy 
2.
3.

Actual results:[    0.000000] Kernel command line: BOOT_IMAGE=/ostree/rhcos-1f8a02066bf1850bb60e814e7ffa9c7066494bd88f097eae08b37781f980cefe/vmlinuz-4.18.0-80.11.2.el8_0.x86_64 console=tty0 console=ttyS0,115200n8 rootflags=defaults,prjquota rw ignition.firstboot rd.neednet=1 ip=dhcp root=UUID=477c3d77-20c6-4ff3-8bb3-dc2543eedfbd ostree=/ostree/boot.0/rhcos/1f8a02066bf1850bb60e814e7ffa9c7066494bd88f097eae08b37781f980cefe/0 coreos.oem.id=qemu coreos.oem.id=openstack ignition.platform.id=openstack
[   10.013580] ignition[657]: GET http://169.254.169.254/openstack/latest/user_data: attempt #6
[   10.020468] ignition[657]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable
[   10.054660] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #6
[   10.068642] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:55422->[::1]:53: read: connection refused
[   15.014348] ignition[657]: GET http://169.254.169.254/openstack/latest/user_data: attempt #7
[   15.020966] ignition[657]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable
[   15.055878] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #7
[   15.069962] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:33365->[::1]:53: read: connection refused
[   20.014908] ignition[657]: GET http://169.254.169.254/openstack/latest/user_data: attempt #8
[   20.021356] ignition[657]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable
[...]
[  375.136910] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #79
[  375.151295] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:40593->[::1]:53: read: connection refused
Press Enter for emergency shell or wait 3 minutes for reboot.                 [  380.137801] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #80
[  380.152156] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:45821->[::1]:53: read: connection refused
[  385.138747] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #81
[  385.153039] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:36824->[::1]:53: read: connection refused

Expected results:
Ignition gets the metadata from configdrive

Additional info:
https://github.com/coreos/ignition/issues/769

Comment 3 Mike Fedosin 2019-11-27 18:35:24 UTC
Hello! In OCP we use the metadata service in three components:
1. In the installer, to inject ignition configs in the bootstrap machine and initial masters.
2. In cluster-api-provider-openstack, to inject ignition configs in workers and new masters.
3. In the in-tree cloud provider to get metadata for kubelet.

We rely heavily on this service, and I'm afraid it will be hard to fix it in 4.2, since a lot of components are affected. We will investigate what changes need to be done in 4.3 to enable config-drive metadata and whether it is possible to backport the patches to 4.2.

Meanwhile, until it's fixed, I think we should add the metadata service requirement to our documentation.

Comment 4 Juan Luis de Sousa-Valadas 2019-11-28 11:07:59 UTC
Added a documentation bug https://bugzilla.redhat.com/show_bug.cgi?id=1777804

Comment 5 Mike Fedosin 2019-11-29 11:12:15 UTC
Documenation fix: https://github.com/openshift/installer/pull/2729
Installer fix: https://github.com/openshift/installer/pull/2734

Comment 8 Mike Fedosin 2019-11-29 16:31:18 UTC
Cloud provider fix:
  Upstream: https://github.com/kubernetes/kubernetes/pull/85755
  Backport: https://github.com/openshift/origin/pull/24241

Comment 9 Mike Fedosin 2020-02-14 17:10:05 UTC
I have to say that it is impossible to use the config drive with OpenStack's cloud provider because it can't provide information about attached volumes.
https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go#L518-L519

We documented that Nova's Metadata service is a hard requirement for us: https://github.com/openshift/installer/blob/master/docs/user/openstack/README.md#nova-metadata-service