Bug 1878758 - Add UPI instructions for FIPless install
Summary: Add UPI instructions for FIPless install
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.6.0
Assignee: Max Bridges
QA Contact: weiwei jiang
Vikram Goyal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-14 13:17 UTC by Pierre Prinetti
Modified: 2020-10-14 17:53 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-14 17:53:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 3755 0 None closed Bug 1878758: openstack UPI: Optional floating IPs 2021-02-09 14:43:30 UTC
Github openshift installer pull 4219 0 None closed Bug 1878758: openstack UPI: Allow for no FIP or router 2021-02-09 14:43:30 UTC

Description Pierre Prinetti 2020-09-14 13:17:32 UTC
It is now possible to install on IPI without using floating IPs.

Change the UPI documentation so that it is possible to install with the provided Ansible playbooks without using floating IPs.

Comment 3 Pierre Prinetti 2020-09-24 07:26:04 UTC
Since the setup of a UPI installation can vary wildly, we don't want to be prescriptive as to how the installation process is carried over; even more so in exotic network topologies where FIPs are not available.

What about spinning an additional VM inside the cluster network, and run openshift-install wait-for bootstrap-complete (and the following steps) from that machine, with the API VIP in /etc/hosts?

Comment 7 Pierre Prinetti 2020-10-02 13:55:24 UTC
The playbooks now allow to disable just the FIPs individually, or the external network. If the external network is not set, the router is not created; this means that the connectivity must be provided by the user (possibly, by creating the router manually).

Comment 10 weiwei jiang 2020-10-10 08:49:34 UTC
And with external_network, the installation work well.

[core@wj46uos1010f-5hlt2-bootstrap ~]$ ./openshift-install-4.6 wait-for bootstrap-complete --dir install-dir --log-level debug                                                                                                                
DEBUG OpenShift Installer 4.6.0-0.nightly-2020-10-10-041109                                                                                                                                                                                   
DEBUG Built from commit ebdbda57fc18d3b73e69f0f2cc499ddfca7e6593                                                                                                                                                                              
INFO Waiting up to 20m0s for the Kubernetes API at https://api.wj46uos1010f.qe.devcluster.openshift.com:6443... 
INFO API v1.19.0+d59ce34 up                       
INFO Waiting up to 30m0s for bootstrapping to complete... 
DEBUG Bootstrap status: complete                   
INFO It is now safe to remove the bootstrap resources 
DEBUG Time elapsed per stage:                      
DEBUG Bootstrap Complete: 1s                       
DEBUG                API: 1s                       
INFO Time elapsed: 1s                             
[core@wj46uos1010f-5hlt2-bootstrap ~]$ ./openshift-install-4.6 wait-for install-complete --dir install-dir --log-level debug
DEBUG OpenShift Installer 4.6.0-0.nightly-2020-10-10-041109  
DEBUG Built from commit ebdbda57fc18d3b73e69f0f2cc499ddfca7e6593 
DEBUG Loading Install Config...                    
DEBUG   Loading SSH Key...                         
DEBUG   Loading Base Domain...                     
DEBUG     Loading Platform...                      
DEBUG   Loading Cluster Name...                    
DEBUG     Loading Base Domain...                   
DEBUG     Loading Platform...                      
DEBUG   Loading Pull Secret...                     
DEBUG   Loading Platform...                        
DEBUG Using Install Config loaded from state file  
INFO Waiting up to 40m0s for the cluster at https://api.wj46uos1010f.qe.devcluster.openshift.com:6443 to initialize...  
DEBUG Still waiting for the cluster to initialize: Some cluster operators are still updating: authentication, console, image-registry, ingress, kube-storage-version-migrator, monitoring 
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.nightly-2020-10-10-041109: 99% complete 
DEBUG Still waiting for the cluster to initialize: Working towards 4.6.0-0.nightly-2020-10-10-041109: 100% complete 
DEBUG Still waiting for the cluster to initialize: Cluster operator authentication is reporting a failure: WellKnownReadyControllerDegraded: kube-apiserver oauth endpoint https://192.168.0.202:6443/.well-known/oauth-authorization-server $s not yet served and authentication operator keeps waiting (check kube-apiserver operator, and check that instances roll out successfully, which can take several minutes per instance) 
DEBUG Cluster is initialized                       
INFO Waiting up to 10m0s for the openshift-console route to be created... 
DEBUG Route found in openshift-console namespace: console 
DEBUG Route found in openshift-console namespace: downloads  
DEBUG OpenShift console route is created           
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/var/home/core/install-dir/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.wj46uos1010f.qe.devcluster.openshift.com 
INFO Login to the console with user: "kubeadmin", and password: "YnnNB-KI9XX-VYYTj-pcEqc" 
DEBUG Time elapsed per stage:                      
DEBUG Cluster Operators: 8m6s                      
DEBUG           Console: 1s                        
INFO Time elapsed: 8m7s                           

[core@wj46uos1010f-5hlt2-bootstrap ~]$ oc get nodes -o wide                                                                                                                                                                                   
NAME                          STATUS   ROLES    AGE   VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                     CONTAINER-RUNTIME
wj46uos1010f-5hlt2-master-0   Ready    master   75m   v1.19.0+d59ce34   192.168.1.204   <none>        Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa)   4.18.0-193.24.1.el8_2.dt1.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.
el8                                                
wj46uos1010f-5hlt2-master-1   Ready    master   73m   v1.19.0+d59ce34   192.168.0.36    <none>        Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa)   4.18.0-193.24.1.el8_2.dt1.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.
el8                                               
wj46uos1010f-5hlt2-master-2   Ready    master   71m   v1.19.0+d59ce34   192.168.0.202   <none>        Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa)   4.18.0-193.24.1.el8_2.dt1.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.
el8                                                                                                                                                                                                                                           
wj46uos1010f-5hlt2-worker-0   Ready    worker   10m   v1.19.0+d59ce34   192.168.1.244   <none>        Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa)   4.18.0-193.24.1.el8_2.dt1.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.el8
wj46uos1010f-5hlt2-worker-1   Ready    worker   10m   v1.19.0+d59ce34   192.168.2.111   <none>        Red Hat Enterprise Linux CoreOS 46.82.202010091720-0 (Ootpa)   4.18.0-193.24.1.el8_2.dt1.x86_64   cri-o://1.19.0-20.rhaos4.6.git97d715e.el8


# openstack server list --name wj46uos
+--------------------------------------+------------------------------+--------+------------------------------------------+----------------------------+-----------+
| ID                                   | Name                         | Status | Networks                                 | Image                      | Flavor    |
+--------------------------------------+------------------------------+--------+------------------------------------------+----------------------------+-----------+
| 5ee762b1-a064-42a4-87df-25baff0b2dbc | wj46uos1010f-5hlt2-worker-1  | ACTIVE | wj46uos1010f-5hlt2-network=192.168.2.111 | rhcos-46.82.202008260918-0 | m1.large  |
| 85cd1f8d-91f9-457c-b2d5-991758cf7dfb | wj46uos1010f-5hlt2-worker-0  | ACTIVE | wj46uos1010f-5hlt2-network=192.168.1.244 | rhcos-46.82.202008260918-0 | m1.large  |
| a1701e90-3c99-47cc-9955-323a1d21ea24 | wj46uos1010f-5hlt2-master-2  | ACTIVE | wj46uos1010f-5hlt2-network=192.168.0.202 | rhcos-46.82.202008260918-0 | m1.xlarge |
| 50143f05-7225-4ae5-aaf3-07d601b89be5 | wj46uos1010f-5hlt2-master-1  | ACTIVE | wj46uos1010f-5hlt2-network=192.168.0.36  | rhcos-46.82.202008260918-0 | m1.xlarge |
| 7e710c67-e955-45e8-89b4-e8a9a4ad70d0 | wj46uos1010f-5hlt2-master-0  | ACTIVE | wj46uos1010f-5hlt2-network=192.168.1.204 | rhcos-46.82.202008260918-0 | m1.xlarge |
| f0393345-aa20-42cb-a736-dcd5cf456a36 | wj46uos1010f-5hlt2-bootstrap | ACTIVE | wj46uos1010f-5hlt2-network=192.168.3.164 | rhcos-46.82.202008260918-0 | m1.xlarge |
+--------------------------------------+------------------------------+--------+------------------------------------------+----------------------------+-----------+

Comment 11 Pierre Prinetti 2020-10-10 17:15:19 UTC
(In reply to weiwei jiang from comment #9)
> (In reply to Pierre Prinetti from comment #7)
> > The playbooks now allow to disable just the FIPs individually, or the
> > external network. If the external network is not set, the router is not
> > created; this means that the connectivity must be provided by the user
> > (possibly, by creating the router manually).
> 
> Hi, now if we do not give external network, then the network will not have
> gateway, and bootstrap server will never fetch it's ignition file, and OCP
> cluster will never setup.
> Any thoughts?

Run the network playbook, then create the router and create and attach the Control plane FIP:

```
subnet_id='<cluster subnet UUID or name>'
external_network='<external network UUID or name>'

router_id="$(openstack router create -f value -c id ocp_router)"
openstack router add subnet "$router_id" "$subnet_id"
openstack router set --external-gateway "$external_network" "$router_id"

fip_id="$(openstack floating ip create -f value -c id --description "OpenShift API" "$external_network")"
openstack floating ip set --port "${INFRA_ID}-api-port" "$fip_id"
```

Then run the rest of the playbooks; the installation should complete.

Comment 13 weiwei jiang 2020-10-12 05:10:44 UTC
(In reply to Pierre Prinetti from comment #11)
> (In reply to weiwei jiang from comment #9)
> > (In reply to Pierre Prinetti from comment #7)
> > > The playbooks now allow to disable just the FIPs individually, or the
> > > external network. If the external network is not set, the router is not
> > > created; this means that the connectivity must be provided by the user
> > > (possibly, by creating the router manually).
> > 
> > Hi, now if we do not give external network, then the network will not have
> > gateway, and bootstrap server will never fetch it's ignition file, and OCP
> > cluster will never setup.
> > Any thoughts?
> 
> Run the network playbook, then create the router and create and attach the
> Control plane FIP:
> 
> ```
> subnet_id='<cluster subnet UUID or name>'
> external_network='<external network UUID or name>'
> 
> router_id="$(openstack router create -f value -c id ocp_router)"
> openstack router add subnet "$router_id" "$subnet_id"
> openstack router set --external-gateway "$external_network" "$router_id"
> 
> fip_id="$(openstack floating ip create -f value -c id --description
> "OpenShift API" "$external_network")"
> openstack floating ip set --port "${INFRA_ID}-api-port" "$fip_id"
> ```
> 
> Then run the rest of the playbooks; the installation should complete.

This should be equivalent with the result in https://bugzilla.redhat.com/show_bug.cgi?id=1878758#c10 which install with valid os_external_network.

Comment 14 Pierre Prinetti 2020-10-12 14:01:27 UTC
(In reply to weiwei jiang from comment #13)
> (In reply to Pierre Prinetti from comment #11)
> > (In reply to weiwei jiang from comment #9)
> > > (In reply to Pierre Prinetti from comment #7)
> > > > The playbooks now allow to disable just the FIPs individually, or the
> > > > external network. If the external network is not set, the router is not
> > > > created; this means that the connectivity must be provided by the user
> > > > (possibly, by creating the router manually).
> > > 
> > > Hi, now if we do not give external network, then the network will not have
> > > gateway, and bootstrap server will never fetch it's ignition file, and OCP
> > > cluster will never setup.
> > > Any thoughts?
> > 
> > Run the network playbook, then create the router and create and attach the
> > Control plane FIP:
> > 
> > ```
> > subnet_id='<cluster subnet UUID or name>'
> > external_network='<external network UUID or name>'
> > 
> > router_id="$(openstack router create -f value -c id ocp_router)"
> > openstack router add subnet "$router_id" "$subnet_id"
> > openstack router set --external-gateway "$external_network" "$router_id"
> > 
> > fip_id="$(openstack floating ip create -f value -c id --description
> > "OpenShift API" "$external_network")"
> > openstack floating ip set --port "${INFRA_ID}-api-port" "$fip_id"
> > ```
> > 
> > Then run the rest of the playbooks; the installation should complete.
> 
> This should be equivalent with the result in
> https://bugzilla.redhat.com/show_bug.cgi?id=1878758#c10 which install with
> valid os_external_network.

Precisely.

Comment 16 Pierre Prinetti 2020-10-13 09:50:48 UTC
(In reply to weiwei jiang from comment #15)
> How about comment https://bugzilla.redhat.com/show_bug.cgi?id=1878758#c12?

I want to address that with explicitly requiring Ansible 2.9 (and adjusting the CI to use that)

cf https://github.com/openshift/installer/pull/4261

Comment 17 egarcia 2020-10-13 19:41:20 UTC
I dont think we should suggest attaching a floating IP to customers who are trying to install without one. Also this has implications for IPI as well. I think we should make a note to customers using IPI that the cluster needs to be able to reach glance. This probably means that most IPI deployments need to be on a network attached to a router with an external interface. However, for UPI because the user has the freedom to serve the bootstrap ignition bundle however they choose, we should approach this much more broadly. For UPI, we should inform users of this issue, and explain to them why it happens on a technical level, but really just leave the door open for them to come up with their own solution. We don't need to hold their hand for UPI. Thats what IPI is for

Comment 20 Pierre Prinetti 2020-10-14 08:51:08 UTC
(In reply to egarcia from comment #17)
> I dont think we should suggest attaching a floating IP to customers who are
> trying to install without one.

This is what I meant in #3:
> Since the setup of a UPI installation can vary wildly, we don't want to be prescriptive as to how the installation process is carried over; even more so in exotic network topologies where FIPs are not available.

Comment #11 is one way to test the rest of the playbooks, not the recommended way users should use them. I agree that it would not make sense to copy paste in the docs the steps I describe in #11.

Comment 21 Max Bridges 2020-10-14 17:53:25 UTC
The remaining docs work for this is tracked in https://issues.redhat.com/browse/OSDOCS-1275. I wouldn't call this a bug, yet, as the 4.6 docs are unreleased and UPI w/out FIPs is covered in content that's under review. Closing.


Note You need to log in before you can comment on or make changes to this bug.