It is now possible to install on IPI without using floating IPs. Change the UPI documentation so that it is possible to install with the provided Ansible playbooks without using floating IPs.
Since the setup of a UPI installation can vary wildly, we don't want to be prescriptive as to how the installation process is carried over; even more so in exotic network topologies where FIPs are not available. What about spinning an additional VM inside the cluster network, and run openshift-install wait-for bootstrap-complete (and the following steps) from that machine, with the API VIP in /etc/hosts?
The playbooks now allow to disable just the FIPs individually, or the external network. If the external network is not set, the router is not created; this means that the connectivity must be provided by the user (possibly, by creating the router manually).
And with external_network, the installation work well. [core@wj46uos1010f-5hlt2-bootstrap ~]$ ./openshift-install-4.6 wait-for bootstrap-complete --dir install-dir --log-level debug DEBUG OpenShift Installer 4.6.0-0.nightly-2020-10-10-041109 DEBUG Built from commit ebdbda57fc18d3b73e69f0f2cc499ddfca7e6593 INFO Waiting up to 20m0s for the Kubernetes API at https://api.wj46uos1010f.qe.devcluster.openshift.com:6443... INFO API v1.19.0+d59ce34 up INFO Waiting up to 30m0s for bootstrapping to complete... DEBUG Bootstrap status: complete INFO It is now safe to remove the bootstrap resources DEBUG Time elapsed per stage: DEBUG Bootstrap Complete: 1s DEBUG API: 1s INFO Time elapsed: 1s [core@wj46uos1010f-5hlt2-bootstrap ~]$ ./openshift-install-4.6 wait-for install-complete --dir install-dir --log-level debug DEBUG OpenShift Installer 4.6.0-0.nightly-2020-10-10-041109 DEBUG Built from commit ebdbda57fc18d3b73e69f0f2cc499ddfca7e6593 DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Using Install Config loaded from state file INFO Waiting up to 40m0s for the cluster at https://api.wj46uos1010f.qe.devcluster.openshift.com:6443 to initialize... DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, kube-storage-version-migrator, monitoring DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.nightly-2020-10-10-041109: 99% complete DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.nightly-2020-10-10-041109: 100% complete DEBUG Still waiting for the cluster to initialize: Cluster operator authentication is reporting a failure: WellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://192.168.0.202:6443/.well-known/oauth-authorization-server $s not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance) DEBUG Cluster is initialized INFO Waiting up to 10m0s for the openshift-console route to be created... DEBUG Route found in openshift-console namespace: console DEBUG Route found in openshift-console namespace: downloads DEBUG OpenShift console route is created INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/var/home/core/install-dir/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.wj46uos1010f.qe.devcluster.openshift.com INFO Login to the console with user: "kubeadmin", and password: "YnnNB-KI9XX-VYYTj-pcEqc" DEBUG Time elapsed per stage: DEBUG Cluster Operators: 8m6s DEBUG Console: 1s INFO Time elapsed: 8m7s [core@wj46uos1010f-5hlt2-bootstrap ~]$ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME wj46uos1010f-5hlt2-master-0 Ready master 75m v1.19.0+d59ce34 192.168.1.204 <none> Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa) 4.18.0-193.24.1.el8_2.dt1.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e. el8 wj46uos1010f-5hlt2-master-1 Ready master 73m v1.19.0+d59ce34 192.168.0.36 <none> Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa) 4.18.0-193.24.1.el8_2.dt1.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e. el8 wj46uos1010f-5hlt2-master-2 Ready master 71m v1.19.0+d59ce34 192.168.0.202 <none> Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa) 4.18.0-193.24.1.el8_2.dt1.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e. el8 wj46uos1010f-5hlt2-worker-0 Ready worker 10m v1.19.0+d59ce34 192.168.1.244 <none> Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa) 4.18.0-193.24.1.el8_2.dt1.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e.el8 wj46uos1010f-5hlt2-worker-1 Ready worker 10m v1.19.0+d59ce34 192.168.2.111 <none> Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa) 4.18.0-193.24.1.el8_2.dt1.x86_64 cri-o://1.19.0-20.rhaos4.6.git97d715e.el8 # openstack server list --name wj46uos +--------------------------------------+------------------------------+--------+------------------------------------------+----------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------+--------+------------------------------------------+----------------------------+-----------+ | 5ee762b1-a064-42a4-87df-25baff0b2dbc | wj46uos1010f-5hlt2-worker-1 | ACTIVE | wj46uos1010f-5hlt2-network=192.168.2.111 | rhcos-46.82.202008260918-0 | m1.large | | 85cd1f8d-91f9-457c-b2d5-991758cf7dfb | wj46uos1010f-5hlt2-worker-0 | ACTIVE | wj46uos1010f-5hlt2-network=192.168.1.244 | rhcos-46.82.202008260918-0 | m1.large | | a1701e90-3c99-47cc-9955-323a1d21ea24 | wj46uos1010f-5hlt2-master-2 | ACTIVE | wj46uos1010f-5hlt2-network=192.168.0.202 | rhcos-46.82.202008260918-0 | m1.xlarge | | 50143f05-7225-4ae5-aaf3-07d601b89be5 | wj46uos1010f-5hlt2-master-1 | ACTIVE | wj46uos1010f-5hlt2-network=192.168.0.36 | rhcos-46.82.202008260918-0 | m1.xlarge | | 7e710c67-e955-45e8-89b4-e8a9a4ad70d0 | wj46uos1010f-5hlt2-master-0 | ACTIVE | wj46uos1010f-5hlt2-network=192.168.1.204 | rhcos-46.82.202008260918-0 | m1.xlarge | | f0393345-aa20-42cb-a736-dcd5cf456a36 | wj46uos1010f-5hlt2-bootstrap | ACTIVE | wj46uos1010f-5hlt2-network=192.168.3.164 | rhcos-46.82.202008260918-0 | m1.xlarge | +--------------------------------------+------------------------------+--------+------------------------------------------+----------------------------+-----------+
(In reply to weiwei jiang from comment #9) > (In reply to Pierre Prinetti from comment #7) > > The playbooks now allow to disable just the FIPs individually, or the > > external network. If the external network is not set, the router is not > > created; this means that the connectivity must be provided by the user > > (possibly, by creating the router manually). > > Hi, now if we do not give external network, then the network will not have > gateway, and bootstrap server will never fetch it's ignition file, and OCP > cluster will never setup. > Any thoughts? Run the network playbook, then create the router and create and attach the Control plane FIP: ``` subnet_id='<cluster subnet UUID or name>' external_network='<external network UUID or name>' router_id="$(openstack router create -f value -c id ocp_router)" openstack router add subnet "$router_id" "$subnet_id" openstack router set --external-gateway "$external_network" "$router_id" fip_id="$(openstack floating ip create -f value -c id --description "OpenShift API" "$external_network")" openstack floating ip set --port "${INFRA_ID}-api-port" "$fip_id" ``` Then run the rest of the playbooks; the installation should complete.
(In reply to Pierre Prinetti from comment #11) > (In reply to weiwei jiang from comment #9) > > (In reply to Pierre Prinetti from comment #7) > > > The playbooks now allow to disable just the FIPs individually, or the > > > external network. If the external network is not set, the router is not > > > created; this means that the connectivity must be provided by the user > > > (possibly, by creating the router manually). > > > > Hi, now if we do not give external network, then the network will not have > > gateway, and bootstrap server will never fetch it's ignition file, and OCP > > cluster will never setup. > > Any thoughts? > > Run the network playbook, then create the router and create and attach the > Control plane FIP: > > ``` > subnet_id='<cluster subnet UUID or name>' > external_network='<external network UUID or name>' > > router_id="$(openstack router create -f value -c id ocp_router)" > openstack router add subnet "$router_id" "$subnet_id" > openstack router set --external-gateway "$external_network" "$router_id" > > fip_id="$(openstack floating ip create -f value -c id --description > "OpenShift API" "$external_network")" > openstack floating ip set --port "${INFRA_ID}-api-port" "$fip_id" > ``` > > Then run the rest of the playbooks; the installation should complete. This should be equivalent with the result in https://bugzilla.redhat.com/show_bug.cgi?id=1878758#c10 which install with valid os_external_network.
(In reply to weiwei jiang from comment #13) > (In reply to Pierre Prinetti from comment #11) > > (In reply to weiwei jiang from comment #9) > > > (In reply to Pierre Prinetti from comment #7) > > > > The playbooks now allow to disable just the FIPs individually, or the > > > > external network. If the external network is not set, the router is not > > > > created; this means that the connectivity must be provided by the user > > > > (possibly, by creating the router manually). > > > > > > Hi, now if we do not give external network, then the network will not have > > > gateway, and bootstrap server will never fetch it's ignition file, and OCP > > > cluster will never setup. > > > Any thoughts? > > > > Run the network playbook, then create the router and create and attach the > > Control plane FIP: > > > > ``` > > subnet_id='<cluster subnet UUID or name>' > > external_network='<external network UUID or name>' > > > > router_id="$(openstack router create -f value -c id ocp_router)" > > openstack router add subnet "$router_id" "$subnet_id" > > openstack router set --external-gateway "$external_network" "$router_id" > > > > fip_id="$(openstack floating ip create -f value -c id --description > > "OpenShift API" "$external_network")" > > openstack floating ip set --port "${INFRA_ID}-api-port" "$fip_id" > > ``` > > > > Then run the rest of the playbooks; the installation should complete. > > This should be equivalent with the result in > https://bugzilla.redhat.com/show_bug.cgi?id=1878758#c10 which install with > valid os_external_network. Precisely.
(In reply to weiwei jiang from comment #15) > How about comment https://bugzilla.redhat.com/show_bug.cgi?id=1878758#c12? I want to address that with explicitly requiring Ansible 2.9 (and adjusting the CI to use that) cf https://github.com/openshift/installer/pull/4261
I dont think we should suggest attaching a floating IP to customers who are trying to install without one. Also this has implications for IPI as well. I think we should make a note to customers using IPI that the cluster needs to be able to reach glance. This probably means that most IPI deployments need to be on a network attached to a router with an external interface. However, for UPI because the user has the freedom to serve the bootstrap ignition bundle however they choose, we should approach this much more broadly. For UPI, we should inform users of this issue, and explain to them why it happens on a technical level, but really just leave the door open for them to come up with their own solution. We don't need to hold their hand for UPI. Thats what IPI is for
(In reply to egarcia from comment #17) > I dont think we should suggest attaching a floating IP to customers who are > trying to install without one. This is what I meant in #3: > Since the setup of a UPI installation can vary wildly, we don't want to be prescriptive as to how the installation process is carried over; even more so in exotic network topologies where FIPs are not available. Comment #11 is one way to test the rest of the playbooks, not the recommended way users should use them. I agree that it would not make sense to copy paste in the docs the steps I describe in #11.
The remaining docs work for this is tracked in https://issues.redhat.com/browse/OSDOCS-1275. I wouldn't call this a bug, yet, as the 4.6 docs are unreleased and UPI w/out FIPs is covered in content that's under review. Closing.