Bug 1573128 - Openshift-on-OpenStack playbook support for kuryr pool driver selection
Summary: Openshift-on-OpenStack playbook support for kuryr pool driver selection
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 3.11.z
Assignee: Luis Tomas Bolivar
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-30 09:32 UTC by Jon Uriarte
Modified: 2019-01-10 09:04 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The openshift ansible openstack playbook defaulted to the kuryr-kubernetes multi pool driver but that funcionality was not merged on stable/queens kuryr-controller. This fix adds the option to select the pool driver to use for versions older than stable/queens. For newer version it will suffice with setting the kuryr_openstack_pool_driver to multi as describe in the documentation.
Clone Of:
Environment:
Last Closed: 2019-01-10 09:03:57 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift openshift-ansible pull 8103 None closed Enable kuryr pool driver selection 2020-03-10 02:09:40 UTC
Red Hat Product Errata RHBA-2019:0024 None None None 2019-01-10 09:04:04 UTC

Description Jon Uriarte 2018-04-30 09:32:39 UTC
Description of problem:

Support for kuryr pool driver selection is needed, i.e., neutron for baremetal deployments or nested for deployments on top of VMs, or even a multi driver for hybrid deployments. 

Version-Release number of the following components:
rpm -q openshift-ansible
 openshift-ansible-roles-3.10.0-0.27.0.git.0.abed3b7.el7.noarch
 openshift-ansible-docs-3.10.0-0.27.0.git.0.abed3b7.el7.noarch
 openshift-ansible-3.10.0-0.27.0.git.0.abed3b7.el7.noarch
 openshift-ansible-playbooks-3.10.0-0.27.0.git.0.abed3b7.el7.noarch

rpm -q ansible
 ansible-2.4.3.0-1.el7ae.noarch

ansible --version
 ansible 2.4.3.0                                                                                                                                                                               
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Feb  9 2018, 09:51:13) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

Comment 1 Scott Dodson 2018-08-14 21:24:33 UTC
Should be in openshift-ansible-3.11.0-0.15.0

Comment 2 N. Harrison Ripps 2018-09-21 20:16:11 UTC
Per OCP program call on 21-SEP-2018 we are deferring Kuryr-related bugs to 3.11.z

Comment 3 Jon Uriarte 2018-10-11 14:48:12 UTC
Verified on openshift-ansible-3.11.18-1.git.0.033596c.el7.noarch, with:
  . openstack-kuryr-controller 14.0-80 container image
  . openstack-kuryr-cni 14.0-75 container image

Note it only works for OSP 14 Kuryr containers.

Verification steps:

1. Install OSP 14, or OSP 13 but use OSP 14 kuryr containers
2. Install OCP 3.11, with the default config for kuryr_openstack_pool_driver (no need to specify it)
3. Check nested-vlan label is not present for all the nodes:
$ oc get nodes --show-labels                                                                                                                                                                 
NAME                                 STATUS    ROLES     AGE       VERSION           LABELS
app-node-0.openshift.example.com     Ready     compute   3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=f5cfa48f-4428-421d-8938-f22b40e66636,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=app-node-0.openshift.example.com,node-role.kubernetes.io/compute=true
app-node-1.openshift.example.com     Ready     compute   3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=f5cfa48f-4428-421d-8938-f22b40e66636,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=app-node-1.openshift.example.com,node-role.kubernetes.io/compute=true
infra-node-0.openshift.example.com   Ready     infra     3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=f5cfa48f-4428-421d-8938-f22b40e66636,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=infra-node-0.openshift.example.com,node-role.kubernetes.io/infra=true
master-0.openshift.example.com       Ready     master    3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=c4885424-715b-4f78-8849-2fb3287d753e,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=master-0.openshift.example.com,node-role.kubernetes.io/master=true

4. Check basic OCP operation
5. Uninstall OCP
6. Install OCP 3.11, enabling Kuryr multi driver in OSEv3.yml:
# Kuryr label configuration
kuryr_openstack_pool_driver: multi

openshift_node_groups:
  - name: node-config-master
    labels:
      - 'node-role.kubernetes.io/master=true'
      - 'pod_vif=nested-vlan'
    edits: []
  - name: node-config-infra
    labels:
      - 'node-role.kubernetes.io/infra=true'
      - 'pod_vif=nested-vlan'
    edits: []
  - name: node-config-compute
    labels:
      - 'node-role.kubernetes.io/compute=true'
      - 'pod_vif=nested-vlan'
    edits: []

7. Check nested-vlan label is present for all the nodes:

$ oc get nodes --show-labels                                                                                                                                                               
NAME                                 STATUS    ROLES     AGE       VERSION           LABELS
app-node-0.openshift.example.com     Ready     compute   3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=f5cfa48f-4428-421d-8938-f22b40e66636,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=app-node-0.openshift.example.com,node-role.kubernetes.io/compute=true,pod_vif=nested-vlan
app-node-1.openshift.example.com     Ready     compute   3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=f5cfa48f-4428-421d-8938-f22b40e66636,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=app-node-1.openshift.example.com,node-role.kubernetes.io/compute=true,pod_vif=nested-vlan
infra-node-0.openshift.example.com   Ready     infra     3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=f5cfa48f-4428-421d-8938-f22b40e66636,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=infra-node-0.openshift.example.com,node-role.kubernetes.io/infra=true,pod_vif=nested-vlan
master-0.openshift.example.com       Ready     master    3h        v1.11.0+d4cacc0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=c4885424-715b-4f78-8849-2fb3287d753e,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=master-0.openshift.example.com,node-role.kubernetes.io/master=true,pod_vif=nested-vlan

8. Create and expose and application:
$ oc new-project test
$ oc run --image=kuryr/demo demo
$ oc scale deploymentconfig demo --replicas=3

$ oc get pods -o wide
NAME           READY     STATUS    RESTARTS   AGE       IP           NODE                               NOMINATED NODE
demo-1-4wlbj   1/1       Running   0          1h        10.11.0.8    app-node-1.openshift.example.com   <none>
demo-1-c2px6   1/1       Running   0          1h        10.11.0.34   app-node-1.openshift.example.com   <none>
demo-1-qwmnv   1/1       Running   0          1h        10.11.0.22   app-node-0.openshift.example.com   <none>

$ oc expose deploymentconfig demo --port=80 --target-port=8080 --type=LoadBalancer

$ oc get svc
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
demo      LoadBalancer   172.30.39.33   10.46.22.53   80:32638/TCP   2m

9. Check connectivity between pods:
$ oc rsh demo-1-4wlbj ping 10.11.0.34 -c 1
PING 10.11.0.34 (10.11.0.34): 56 data bytes
64 bytes from 10.11.0.34: seq=0 ttl=64 time=0.675 ms

--- 10.11.0.34 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.675/0.675/0.675 ms

$ oc rsh demo-1-4wlbj ping 10.11.0.22 -c 1
PING 10.11.0.22 (10.11.0.22): 56 data bytes
64 bytes from 10.11.0.22: seq=0 ttl=64 time=0.835 ms

--- 10.11.0.22 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.835/0.835/0.835 ms

10. Check pod ports belong to trunk ports:
(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack network trunk list
+--------------------------------------+------------------------------------+--------------------------------------+-------------+
| ID                                   | Name                               | Parent Port                          | Description |
+--------------------------------------+------------------------------------+--------------------------------------+-------------+
| 2a2ade3b-6d1e-44ff-8a63-b441125b2978 | master-0.openshift.example.com     | 1e3ac9a8-f212-4ed8-a0d1-b7005c1440f5 |             |
| 3c38db50-88b9-4027-9c82-30c983658a2e | app-node-1.openshift.example.com   | bc59357b-902a-4739-8b43-ede9587e6e6e |             |
| b015057f-55cc-4e54-b34c-ae270a5b8d46 | app-node-0.openshift.example.com   | 7095a643-38ad-4d80-aecd-fa91b7ae9d64 |             |
| c8dd02e5-c4e1-4abc-8b42-978cb100a3af | infra-node-0.openshift.example.com | 8ad0440b-d0e5-437d-a909-16d73d5a05f8 |             |
+--------------------------------------+------------------------------------+--------------------------------------+-------------+

(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack network trunk show app-node-1.openshift.example.com
+-----------------+--------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                            |
+-----------------+--------------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                               |
| created_at      | 2018-10-11T11:06:50Z                                                                             |
| description     |                                                                                                  |
| id              | 3c38db50-88b9-4027-9c82-30c983658a2e                                                             |
| name            | app-node-1.openshift.example.com                                                                 |
| port_id         | bc59357b-902a-4739-8b43-ede9587e6e6e                                                             |
| project_id      | 0127ae37e08b4ef9972632e71f190b14                                                                 |
| revision_number | 4                                                                                                |
| status          | ACTIVE                                                                                           |
| sub_ports       | port_id='43ef4aa9-038b-4aa1-bb16-3d72f43c01fa', segmentation_id='260', segmentation_type='vlan'  |
|                 | port_id='a02dc715-f45e-472d-8867-65695c55ed8f', segmentation_id='1079', segmentation_type='vlan' |
|                 | port_id='1fc61706-2eed-49aa-b343-a936e2b4b279', segmentation_id='2529', segmentation_type='vlan' |
|                 | port_id='2218fa93-c66f-4c7c-b67b-698fa27d0785', segmentation_id='3441', segmentation_type='vlan' |
|                 | port_id='c049c8af-75ac-4599-b0da-0889d7ce70ef', segmentation_id='3528', segmentation_type='vlan' |
| tags            | []                                                                                               |
| tenant_id       | 0127ae37e08b4ef9972632e71f190b14                                                                 |
| updated_at      | 2018-10-11T13:03:22Z                                                                             |
+-----------------+--------------------------------------------------------------------------------------------------+

(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack port list | grep 10.11.0.34
| c049c8af-75ac-4599-b0da-0889d7ce70ef |                                                                               | fa:16:3e:2c:0a:89 | ip_address='10.11.0.34', subnet_id='bf8a0880-a81b-4698-a4cc-2be6bca67512'     | ACTIVE |

(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack port list | grep 10.11.0.8
| a02dc715-f45e-472d-8867-65695c55ed8f |                                                                               | fa:16:3e:43:71:be | ip_address='10.11.0.8', subnet_id='bf8a0880-a81b-4698-a4cc-2be6bca67512'      | ACTIVE |


(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack network trunk show app-node-0.openshift.example.com
+-----------------+--------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                            |
+-----------------+--------------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                               |
| created_at      | 2018-10-11T11:06:50Z                                                                             |
| description     |                                                                                                  |
| id              | b015057f-55cc-4e54-b34c-ae270a5b8d46                                                             |
| name            | app-node-0.openshift.example.com                                                                 |
| port_id         | 7095a643-38ad-4d80-aecd-fa91b7ae9d64                                                             |
| project_id      | 0127ae37e08b4ef9972632e71f190b14                                                                 |
| revision_number | 4                                                                                                |
| status          | ACTIVE                                                                                           |
| sub_ports       | port_id='a8339996-e716-464e-9897-fb09bdb9b615', segmentation_id='794', segmentation_type='vlan'  |
|                 | port_id='91c37ebd-0682-4540-8497-247896ec2f99', segmentation_id='1030', segmentation_type='vlan' |
|                 | port_id='439f3f86-cf30-4744-a020-2b74c60ebc0f', segmentation_id='1999', segmentation_type='vlan' |
|                 | port_id='31cd2a4a-6540-4eb7-ad33-86568d85b988', segmentation_id='2368', segmentation_type='vlan' |
|                 | port_id='0ff74019-3837-4a8a-b50f-46c73cf1d8f6', segmentation_id='2740', segmentation_type='vlan' |
| tags            | []                                                                                               |
| tenant_id       | 0127ae37e08b4ef9972632e71f190b14                                                                 |
| updated_at      | 2018-10-11T13:04:02Z                                                                             |
+-----------------+--------------------------------------------------------------------------------------------------+

(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack port list | grep 10.11.0.22
| 91c37ebd-0682-4540-8497-247896ec2f99 |                                                                               | fa:16:3e:4d:67:34 | ip_address='10.11.0.22', subnet_id='bf8a0880-a81b-4698-a4cc-2be6bca67512'     | ACTIVE |

11. Delete the test project:
$ oc delete project test

Comment 5 errata-xmlrpc 2019-01-10 09:03:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0024


Note You need to log in before you can comment on or make changes to this bug.