Bug 1857865 - Unable to access load balancer members through the vip
Summary: Unable to access load balancer members through the vip
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Assaf Muller
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On: 1858191
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-16 17:05 UTC by Maysa Macedo
Modified: 2020-07-28 13:36 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-28 13:36:17 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Maysa Macedo 2020-07-16 17:05:54 UTC
Description of problem:

A pod/port is unable to access a load balancer member trough the lb VIP, but works fine when directly accessing the load balancer members.

$ oc get po -A -o wide|grep network-policy
network-policy-1337                                client-can-connect-80-k2jvk                               1/1     Running     0          69m     10.128.114.32    ostest-m6tzl-worker-w4zw2   <none>           <none>
network-policy-1337                                server-ktc66                                              1/1     Running     0          70m     10.128.115.64    ostest-m6tzl-worker-l5fhk   <none>           <none>

$ oc exec -it client-can-connect-80-k2jvk -n network-policy-1337 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
13: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1442 qdisc noqueue qlen 1000
    link/ether fa:16:3e:7a:49:89 brd ff:ff:ff:ff:ff:ff
    inet 10.128.114.32/23 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe7a:4989/64 scope link 
       valid_lft forever preferred_lft forever
/ # nc -vz 10.128.115.64 80
10.128.115.64 (10.128.115.64:80) open
/ # nc -vz 172.30.86.89 80
nc: 172.30.86.89 (172.30.86.89:80): Connection timed out


$ openstack port show 669fe3ec-6968-4766-914d-77ae3ce1cc3e
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                 |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                    |
| allowed_address_pairs   |                                                                                                                                                                       |
| binding_host_id         | None                                                                                                                                                                  |
| binding_profile         | None                                                                                                                                                                  |
| binding_vif_details     | None                                                                                                                                                                  |
| binding_vif_type        | None                                                                                                                                                                  |
| binding_vnic_type       | normal                                                                                                                                                                |
| created_at              | 2020-07-16T15:31:27Z                                                                                                                                                  |
| data_plane_status       | None                                                                                                                                                                  |
| description             |                                                                                                                                                                       |
| device_id               |                                                                                                                                                                       |
| device_owner            | trunk:subport                                                                                                                                                         |
| dns_assignment          | fqdn='host-10-128-114-32.shiftstack.com.', hostname='host-10-128-114-32', ip_address='10.128.114.32'                                                                  |
| dns_domain              |                                                                                                                                                                       |
| dns_name                |                                                                                                                                                                       |
| extra_dhcp_opts         |                                                                                                                                                                       |
| fixed_ips               | ip_address='10.128.114.32', subnet_id='0c9e644d-3177-4d9d-814b-20579de04e7e'                                                                                          |
| id                      | 669fe3ec-6968-4766-914d-77ae3ce1cc3e                                                                                                                                  |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='d4ca2d6c3db14fca92bff757d3d63439', project.name='shiftstack', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:7a:49:89                                                                                                                                                     |
| name                    |                                                                                                                                                                       |
| network_id              | 60557190-aa31-4e1f-b3ef-b29f224d141b                                                                                                                                  |
| port_security_enabled   | True                                                                                                                                                                  |
| project_id              | d4ca2d6c3db14fca92bff757d3d63439                                                                                                                                      |
| propagate_uplink_status | None                                                                                                                                                                  |
| qos_policy_id           | None                                                                                                                                                                  |
| resource_request        | None                                                                                                                                                                  |
| revision_number         | 5                                                                                                                                                                     |
| security_group_ids      | 9e60aa72-11b9-45ba-98bf-efe3e66f10c3                                                                                                                                  |
| status                  | ACTIVE                                                                                                                                                                |
| tags                    | openshiftClusterID=ostest-m6tzl


$ openstack port show f44d9cc5-6047-4dfa-b97b-49b4e392373c
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                 |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                    |
| allowed_address_pairs   |                                                                                                                                                                       |
| binding_host_id         | None                                                                                                                                                                  |
| binding_profile         | None                                                                                                                                                                  |
| binding_vif_details     | None                                                                                                                                                                  |
| binding_vif_type        | None                                                                                                                                                                  |
| binding_vnic_type       | normal                                                                                                                                                                |
| created_at              | 2020-07-16T15:30:34Z                                                                                                                                                  |
| data_plane_status       | None                                                                                                                                                                  |
| description             |                                                                                                                                                                       |
| device_id               |                                                                                                                                                                       |
| device_owner            | trunk:subport                                                                                                                                                         |
| dns_assignment          | fqdn='host-10-128-115-64.shiftstack.com.', hostname='host-10-128-115-64', ip_address='10.128.115.64'                                                                  |
| dns_domain              |                                                                                                                                                                       |
| dns_name                |                                                                                                                                                                       |
| extra_dhcp_opts         |                                                                                                                                                                       |
| fixed_ips               | ip_address='10.128.115.64', subnet_id='0c9e644d-3177-4d9d-814b-20579de04e7e'                                                                                          |
| id                      | f44d9cc5-6047-4dfa-b97b-49b4e392373c                                                                                                                                  |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='d4ca2d6c3db14fca92bff757d3d63439', project.name='shiftstack', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:20:24:80                                                                                                                                                     |
| name                    |                                                                                                                                                                       |
| network_id              | 60557190-aa31-4e1f-b3ef-b29f224d141b                                                                                                                                  |
| port_security_enabled   | True                                                                                                                                                                  |
| project_id              | d4ca2d6c3db14fca92bff757d3d63439                                                                                                                                      |
| propagate_uplink_status | None                                                                                                                                                                  |
| qos_policy_id           | None                                                                                                                                                                  |
| resource_request        | None                                                                                                                                                                  |
| revision_number         | 5                                                                                                                                                                     |
| security_group_ids      | 9e60aa72-11b9-45ba-98bf-efe3e66f10c3                                                                                                                                  |
| status                  | ACTIVE                                                                                                                                                                |
| tags                    | openshiftClusterID=ostest-m6tzl  

$ openstack security group show 9e60aa72-11b9-45ba-98bf-efe3e66f10c3 -c rules
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value                                                                                                                                                                                |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| rules | created_at='2020-07-16T12:54:37Z', direction='egress', ethertype='IPv6', id='083a38fb-f292-4f97-9145-4320915a76d2', updated_at='2020-07-16T12:54:37Z'                                |
|       | created_at='2020-07-16T12:54:38Z', direction='ingress', ethertype='IPv4', id='c0432409-982b-47a1-aa1d-718a56a86a00', remote_ip_prefix='0.0.0.0/0', updated_at='2020-07-16T12:54:38Z' |
|       | created_at='2020-07-16T12:54:37Z', direction='egress', ethertype='IPv4', id='e12b609b-62c4-49bb-8b10-9df30df4ecab', updated_at='2020-07-16T12:54:37Z'                                |
+-------+-----------

$ openstack loadbalancer show network-policy-1337/svc-server
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-07-16T15:31:25                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 8d3dd6d7-06ce-40f3-9ea5-9ceeb27ba5bb |
| listeners           | 974e6545-20e1-4fd5-a4d9-801eac381f7a |
| name                | network-policy-1337/svc-server       |
| operating_status    | ONLINE                               |
| pools               | 95a2d203-7ae9-4a59-b67b-4158c8542ef9 |
| project_id          | d4ca2d6c3db14fca92bff757d3d63439     |
| provider            | ovn                                  |
| provisioning_status | ACTIVE                               |
| updated_at          | 2020-07-16T15:31:45                  |
| vip_address         | 172.30.86.89                         |
| vip_network_id      | 4ae1668f-4755-494a-bd84-643d08732005 |
| vip_port_id         | 87cc8a6e-9b80-4303-8d64-ffb1d90a53b9 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 12001018-e6e0-4c2b-8541-1530a9e0da25 |
+---------------------+--------------------------------------+

$ openstack port show 87cc8a6e-9b80-4303-8d64-ffb1d90a53b9
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                 |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                    |
| allowed_address_pairs   |                                                                                                                                                                       |
| binding_host_id         | None                                                                                                                                                                  |
| binding_profile         | None                                                                                                                                                                  |
| binding_vif_details     | None                                                                                                                                                                  |
| binding_vif_type        | None                                                                                                                                                                  |
| binding_vnic_type       | normal                                                                                                                                                                |
| created_at              | 2020-07-16T15:31:26Z                                                                                                                                                  |
| data_plane_status       | None                                                                                                                                                                  |
| description             |                                                                                                                                                                       |
| device_id               |                                                                                                                                                                       |
| device_owner            |                                                                                                                                                                       |
| dns_assignment          | fqdn='host-172-30-86-89.shiftstack.com.', hostname='host-172-30-86-89', ip_address='172.30.86.89'                                                                     |
| dns_domain              |                                                                                                                                                                       |
| dns_name                |                                                                                                                                                                       |
| extra_dhcp_opts         |                                                                                                                                                                       |
| fixed_ips               | ip_address='172.30.86.89', subnet_id='12001018-e6e0-4c2b-8541-1530a9e0da25'                                                                                           |
| id                      | 87cc8a6e-9b80-4303-8d64-ffb1d90a53b9                                                                                                                                  |
| location                | cloud='', project.domain_id=, project.domain_name='Default', project.id='d4ca2d6c3db14fca92bff757d3d63439', project.name='shiftstack', region_name='regionOne', zone= |
| mac_address             | fa:16:3e:3f:d1:e7                                                                                                                                                     |
| name                    | ovn-lb-vip-8d3dd6d7-06ce-40f3-9ea5-9ceeb27ba5bb                                                                                                                       |
| network_id              | 4ae1668f-4755-494a-bd84-643d08732005                                                                                                                                  |
| port_security_enabled   | True                                                                                                                                                                  |
| project_id              | d4ca2d6c3db14fca92bff757d3d63439                                                                                                                                      |
| propagate_uplink_status | None                                                                                                                                                                  |
| qos_policy_id           | None                                                                                                                                                                  |
| resource_request        | None                                                                                                                                                                  |
| revision_number         | 2                                                                                                                                                                     |
| security_group_ids      |                                                                                                                                                                       |
| status                  | DOWN                                                                                                                                                                  |
| tags                    |                                                                                                                                                                       |
| trunk_details           | None                                                                                                                                                                  |
| updated_at              | 2020-07-16T15:31:41Z      

$ openstack loadbalancer listener list |grep svc-server
| 974e6545-20e1-4fd5-a4d9-801eac381f7a | 95a2d203-7ae9-4a59-b67b-4158c8542ef9 | network-policy-1337/svc-server:TCP:80                                               | d4ca2d6c3db14fca92bff757d3d63439 | TCP      |            80 | True           |

$ openstack loadbalancer pool list |grep svc-server
| 95a2d203-7ae9-4a59-b67b-4158c8542ef9 | network-policy-1337/svc-server:TCP:80                                               | d4ca2d6c3db14fca92bff757d3d63439 | ACTIVE              | TCP      | SOURCE_IP_PORT | True           |

$ openstack loadbalancer member list 95a2d203-7ae9-4a59-b67b-4158c8542ef9
+--------------------------------------+-------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| id                                   | name                                | project_id                       | provisioning_status | address       | protocol_port | operating_status | weight |
+--------------------------------------+-------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| 783f5a54-7bbd-4182-b3a3-ff09ff212c47 | network-policy-1337/server-ktc66:80 | d4ca2d6c3db14fca92bff757d3d63439 | ACTIVE              | 10.128.115.64 |            80 | NO_MONITOR       |      1 |
+--------------------------------------+-------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+----

The gateway of both subnets are attached to the router:

$ openstack router show ostest-m6tzl-external-router |grep -e 12001018-e6e0-4c2b-8541-1530a9e0da25 -e 0c9e644d-3177-4d9d-814b-20579de04e7e

| interfaces_info         | {"port_id": "2824f41b-282e-4e72-9131-e0ccb3bf2641", "ip_address": "10.128.114.1", "subnet_id": "0c9e644d-3177-4d9d-814b-20579de04e7e"}, {"port_id": "49eb0f46-6698-4412-a530-04f18c64e538", "ip_address": "172.31.255.254", "subnet_id": "12001018-e6e0-4c2b-8541-1530a9e0da25"}

Version-Release number of selected component (if applicable):

OVN version: 2.13-20.06.1-3
Red Hat OpenStack Platform release 16.1.0 Beta (Train)

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:


Note You need to log in before you can comment on or make changes to this bug.