Description of problem: Customer runs OCP 4.2 on openstack without metadata service. Ignition doesn't take the configuration from config-drive, even though the disk is attached. RHEL instances have it attached on /dev/disk/by-label/config-2 Version-Release number of the following components: Red Hat Enterprise Linux CoreOS - OpenStack Image (QCOW) 4.2.0 ignition-0.33.0-5.rhaos4.2.gitc65e95c.el8.x86_64 How reproducible: Apparently always Steps to Reproduce: 1. Deploy 2. 3. Actual results:[ 0.000000] Kernel command line: BOOT_IMAGE=/ostree/rhcos-1f8a02066bf1850bb60e814e7ffa9c7066494bd88f097eae08b37781f980cefe/vmlinuz-4.18.0-80.11.2.el8_0.x86_64 console=tty0 console=ttyS0,115200n8 rootflags=defaults,prjquota rw ignition.firstboot rd.neednet=1 ip=dhcp root=UUID=477c3d77-20c6-4ff3-8bb3-dc2543eedfbd ostree=/ostree/boot.0/rhcos/1f8a02066bf1850bb60e814e7ffa9c7066494bd88f097eae08b37781f980cefe/0 coreos.oem.id=qemu coreos.oem.id=openstack ignition.platform.id=openstack [ 10.013580] ignition[657]: GET http://169.254.169.254/openstack/latest/user_data: attempt #6 [ 10.020468] ignition[657]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 10.054660] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #6 [ 10.068642] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:55422->[::1]:53: read: connection refused [ 15.014348] ignition[657]: GET http://169.254.169.254/openstack/latest/user_data: attempt #7 [ 15.020966] ignition[657]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [ 15.055878] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #7 [ 15.069962] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:33365->[::1]:53: read: connection refused [ 20.014908] ignition[657]: GET http://169.254.169.254/openstack/latest/user_data: attempt #8 [ 20.021356] ignition[657]: GET error: Get http://169.254.169.254/openstack/latest/user_data: dial tcp 169.254.169.254:80: connect: network is unreachable [...] [ 375.136910] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #79 [ 375.151295] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:40593->[::1]:53: read: connection refused Press Enter for emergency shell or wait 3 minutes for reboot. [ 380.137801] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #80 [ 380.152156] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:45821->[::1]:53: read: connection refused [ 385.138747] ignition[657]: GET https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: attempt #81 [ 385.153039] ignition[657]: GET error: Get https://<removed>:13808/v1/AUTH_<removed>/<removed>/bootstrap.ign?temp_url_sig=<removed>&temp_url_expires=1573659478: dial tcp: lookup <removed> on [::1]:53: read udp [::1]:36824->[::1]:53: read: connection refused Expected results: Ignition gets the metadata from configdrive Additional info: https://github.com/coreos/ignition/issues/769
Hello! In OCP we use the metadata service in three components: 1. In the installer, to inject ignition configs in the bootstrap machine and initial masters. 2. In cluster-api-provider-openstack, to inject ignition configs in workers and new masters. 3. In the in-tree cloud provider to get metadata for kubelet. We rely heavily on this service, and I'm afraid it will be hard to fix it in 4.2, since a lot of components are affected. We will investigate what changes need to be done in 4.3 to enable config-drive metadata and whether it is possible to backport the patches to 4.2. Meanwhile, until it's fixed, I think we should add the metadata service requirement to our documentation.
Added a documentation bug https://bugzilla.redhat.com/show_bug.cgi?id=1777804
Documenation fix: https://github.com/openshift/installer/pull/2729 Installer fix: https://github.com/openshift/installer/pull/2734
Cloud provider fix: Upstream: https://github.com/kubernetes/kubernetes/pull/85755 Backport: https://github.com/openshift/origin/pull/24241
I have to say that it is impossible to use the config drive with OpenStack's cloud provider because it can't provide information about attached volumes. https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go#L518-L519 We documented that Nova's Metadata service is a hard requirement for us: https://github.com/openshift/installer/blob/master/docs/user/openstack/README.md#nova-metadata-service