Description of problem: The playbook already spin up a LB for a multi-master deployment. But the same thing should happen in case we have multiple infra nodes. In case we run multiple masters and multiple infra nodes, the same load balancer VM should be configured for both. Expected results:
This thing is blocked with the status: "the use case requires clarification and dod expectations adjusted" More details on that: what is the use case for a lb node and multi infra nodes? I can't find any docs or details on that. Which configuration the lb node is expected to get for multi infra nodes? What is the test case for it? Shall the lb node's haproxy config provide a load balance for a docker registry, when deploying a multiple infra nodes? I can see only a single logical frontend/backend definition stanza for master lb: ``` frontend atomic-openshift-api bind *:8443 default_backend atomic-openshift-api mode tcp option tcplog backend atomic-openshift-api balance source mode tcp server master0 192.168.99.12:8443 check ``` so blocked to know more for supported use cases of lb for registry and routers, and what else hosted at the infra nodes?..
Info needed from openshift-ansible and/or reference-architecture folks.
The load balancer deployed by openshift-ansible seems to only support master node loadbalancing. I don't think we should spend time working on that. Let's limit the scope of this RFE to Octavia only. I.e. when you use the Neutron LBaaSv2/Octavia, we should set up load balancing for ports 80 and 443 for the infra nodes **in addition** to the master ones. The reference architecture doesn't use the node set up by openshift-ansible either, but since there are some issues with lbaas on OSP 10, they create their own load balancer and configure it themselves: https://access.redhat.com/documentation/en-us/reference_architectures/2017/html-single/deploying_and_managing_red_hat_openshift_container_platform_3.6_on_red_hat_openstack_platform_10/#haproxy If we want to do that as well (for cases where octavia is unavailable), that should require another RFE.
Fuller description: Description of problem: OpenShift on OpenStack should support load balancing for a multi-master/infra node deployment of Openshift-on-OpenStack. This RFE adds an option to the openshift-ansible playbooks to use Octavia to create a load balancer to do so. Steps to Reproduce: 1. Configure the inventory as documented in https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/advanced-configuration.md 2. Run the playbooks as documented in https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/README.md Actual results: Not implemented Expected results: Validation steps are documented in https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/advanced-configuration.md Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated
How to test: Prerequisites: 1. A tenant access to an OpenStack with Octavia Steps: 1. Configure the inventory as described in: https://bugzilla.redhat.com/show_bug.cgi?id=1503667#c2 2. Add `openshift_openstack_use_lbaas_load_balancer: true` to your inventory/group_vars/all.yml 3. Set `openshift_openstack_num_infra = 2` in all.yml 4. Run the provision_install playbook * The playbook will print out `openshift_openstack_public_router_ip` at the end * Note the IP address Validation: 1. The playbook must finish without any errors 2. The `router_lb` load balancer was created: `openstack loadbalancer list` 3. The `openshift_openstack_public_router_ip` is NOT an IP address of any of the servers in `openstack server list` but it corresponds to a floating IP address attached to a port of the load balancer 4. Log in to the cluster: oc login 5. Create a new project: oc new-project test 6. Launch an openshift app: oc new-app --template=cakephp-mysql-example 7. Wait for all pods to be running: oc status --suggest 8. Update your DNS or /etc/hosts so that the app route resolves to the `openshift_openstack_public_router_ip` 9. Verify that the app is accessible at its route/url
Okay, I don't think steps 5 and onwards are really necessary. They test the end-to-end but that's not what this RFE is about anyway. Let's do this instead: 5. SSH into the master VM 6. Run: oc get pod -n default -o wide | grep router - there should be two router pods Running, one on each Infra node 7. Run: oc describe svc router -n default - the router service should exist - note it's IP value (not Endpoints) 8. From the ansible host, run: openstack loadbalancer list - there should be a load balancer called `default/router` - its `vip_address` should be equal to the router svc IP
Verified in openshift-ansible-3.10.0-0.67.0 over OSP 13 2018-05-23.1 puddle with Octavia. Verification steps: 1. Deploy OpenStack (OSP13) with Octavia 2. Deploy an Ansible-host and a DNS server on the overcloud 3. Get OCP openshift-ansible downstream rpm 4. Configure OSP (all.yml) and OCP (OSEv3.yml) inventory files Set: - 'openshift_openstack_use_lbaas_load_balancer: true' and - 'openshift_openstack_num_infra: 2' in inventory/group_vars/all.yml 5. Run from the Ansible-host: ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/prerequisites.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory red-hat-ca.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/repos.yml ansible-playbook --user openshift -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory /usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/install.yml 6. Check the installer finishes without errors, and note the `openshift_openstack_public_router_ip` at the end of the playbook print-out TASK [Print the OpenShift Router Public IP Address] *************************************************************************************************************************************************************** ok: [localhost] => { "openshift_openstack_public_router_ip": "172.20.0.234" } 7. Check vms deployed in the overcloud (shiftstack) [cloud-user@ansible-host ~]$ openstack server list +--------------------------------------+------------------------------------+--------+-------------------------------------------------------------------------+--------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------------+--------+-------------------------------------------------------------------------+--------+-----------+ | 40b470c7-5229-4fba-a8fc-6c5e885450ba | infra-node-1.openshift.example.com | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.19, 172.20.0.222 | rhel75 | m1.node | | b6c0d43b-14d8-46a0-8a6d-9fdf062aad74 | infra-node-0.openshift.example.com | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.14, 172.20.0.240 | rhel75 | m1.node | | af6b6db0-c9bf-4111-a347-7d61d63e7975 | master-0.openshift.example.com | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.16, 172.20.0.236 | rhel75 | m1.master | | eb1eb7ae-5a76-407a-b546-ac0281af5cd0 | app-node-1.openshift.example.com | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.15, 172.20.0.237 | rhel75 | m1.node | | 3f877066-2083-4fa8-ad2b-ab90de38c4b5 | app-node-0.openshift.example.com | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.11, 172.20.0.235 | rhel75 | m1.node | +--------------------------------------+------------------------------------+--------+-------------------------------------------------------------------------+--------+-----------+ 8. Check the `router_lb` load balancer was created (`openstack loadbalancer list`) (shiftstack) [cloud-user@ansible-host ~]$ openstack loadbalancer list +--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+ | b97a6f5d-f8ab-4ff0-9ae9-bebe0d24a5d9 | openshift-ansible-openshift.example.com-api-lb | a02185177ac246529e69bb252f021683 | 172.30.0.1 | ACTIVE | octavia | | b096546e-6d94-42bd-a3f0-aa827ba54435 | openshift-cluster-router_lb-4f53nds4cg75 | a02185177ac246529e69bb252f021683 | 192.168.99.6 | ACTIVE | octavia | | 4e340890-1075-4123-bf1d-9b1f9a3ecafc | default/router | a02185177ac246529e69bb252f021683 | 172.30.108.168 | ACTIVE | octavia | | 37f2f27b-478c-4287-ad72-6c62a720ce91 | default/docker-registry | a02185177ac246529e69bb252f021683 | 172.30.247.20 | ACTIVE | octavia | | 9c8d6e8e-09aa-4729-9641-de2ee71706dd | default/registry-console | a02185177ac246529e69bb252f021683 | 172.30.217.98 | ACTIVE | octavia | +--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+ 9. Check the `openshift_openstack_public_router_ip` is NOT an IP address of any of the servers in `openstack server list` but it corresponds to a floating IP address attached to a port of the load balancer (shiftstack) [cloud-user@ansible-host ~]$ openstack floating ip list | grep 192.168.99.6 | 23d04d26-f760-43c2-b2d6-ed8b9dccf429 | 172.20.0.234 | 192.168.99.6 | 9ac772da-69b4-47a8-8c56-daf3ac4fec4d | dd5a700a-a0bf-4e18-b6db-a59f4063f7b4 | a02185177ac246529e69bb252f021683 | LB's fixed IP (192.168.99.6) and floating IP (172.20.0.234). The floating IP is the one showed in the playbook print-out. 10. SSH into the master VM (shiftstack) [cloud-user@ansible-host ~]$ ssh -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" openshift.0.236 11. Run: oc get pod -n default -o wide | grep router - there should be two router pods Running, one on each Infra node [openshift@master-0 ~]$ oc get pod -n default -o wide | grep router router-1-7htzk 1/1 Running 0 3h 192.168.99.14 infra-node-0.openshift.example.com router-1-7qlcx 1/1 Running 0 3h 192.168.99.19 infra-node-1.openshift.example.com 12. Run: oc describe svc router -n default - the router service should exist - note it's IP value (not Endpoints) [openshift@master-0 ~]$ oc describe svc router -n default Name: router Namespace: default Labels: router=router Annotations: openstack.org/kuryr-lbaas-spec={"versioned_object.data": {"ip": "172.30.108.168", "lb_ip": null, "ports": [{"versioned_object.data": {"name": "80-tcp", "port": 80, "protocol": "TCP"}, "versioned_objec... prometheus.openshift.io/password=Oha11kU2EC prometheus.openshift.io/username=admin service.alpha.openshift.io/serving-cert-secret-name=router-metrics-tls service.alpha.openshift.io/serving-cert-signed-by=openshift-service-serving-signer@1530010068 Selector: router=router Type: ClusterIP IP: 172.30.108.168 Port: 80-tcp 80/TCP TargetPort: 80/TCP Endpoints: 192.168.99.14:80,192.168.99.19:80 Port: 443-tcp 443/TCP TargetPort: 443/TCP Endpoints: 192.168.99.14:443,192.168.99.19:443 Port: 1936-tcp 1936/TCP TargetPort: 1936/TCP Endpoints: 192.168.99.14:1936,192.168.99.19:1936 Session Affinity: None Events: <none> 13. From the ansible host, run: openstack loadbalancer list - there should be a load balancer called `default/router` - its `vip_address` should be equal to the router svc IP (shiftstack) [cloud-user@ansible-host ~]$ openstack loadbalancer list +--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+ | b97a6f5d-f8ab-4ff0-9ae9-bebe0d24a5d9 | openshift-ansible-openshift.example.com-api-lb | a02185177ac246529e69bb252f021683 | 172.30.0.1 | ACTIVE | octavia | | b096546e-6d94-42bd-a3f0-aa827ba54435 | openshift-cluster-router_lb-4f53nds4cg75 | a02185177ac246529e69bb252f021683 | 192.168.99.6 | ACTIVE | octavia | | 4e340890-1075-4123-bf1d-9b1f9a3ecafc | default/router | a02185177ac246529e69bb252f021683 | 172.30.108.168 | ACTIVE | octavia | | 37f2f27b-478c-4287-ad72-6c62a720ce91 | default/docker-registry | a02185177ac246529e69bb252f021683 | 172.30.247.20 | ACTIVE | octavia | | 9c8d6e8e-09aa-4729-9641-de2ee71706dd | default/registry-console | a02185177ac246529e69bb252f021683 | 172.30.217.98 | ACTIVE | octavia | +--------------------------------------+------------------------------------------------+----------------------------------+----------------+---------------------+----------+ The `vip_address` 172.30.108.168 indeed matches with the router svc IP.