Description of problem: LB request takes too long to respond: [root@networker-0 ~]# time ip net ex qdhcp-546a606b-a58b-4612-8fe4-0fdc45104712 curl -g -6 'http://[2001::f816:3eff:fee1:72fb]:80/' lbtree-server1-oyze5u6hfttj real 0m15.072s user 0m0.005s sys 0m0.005s [root@networker-0 ~]# time ip net ex qdhcp-546a606b-a58b-4612-8fe4-0fdc45104712 curl -g -6 'http://[2001::f816:3eff:fee1:72fb]:80/' lbtree-server2-dmo3gp5ehlvo real 0m15.067s user 0m0.004s sys 0m0.005s [root@networker-0 ~]# Version-Release number of selected component (if applicable): [2019-02-27 06:12:51] (tester) [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 13 -p 2019-02-25.2 Steps to Reproduce: 1. Deploy overcloud with octavia on network node ( roles data- http://pastebin.test.redhat.com/656153 ) 2. we need ipv4 and ipv6 networks [2019-02-27 06:12:57] (tester) [stack@undercloud-0 ~]$ openstack network list +--------------------------------------+-----------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+-----------+----------------------------------------------------------------------------+ | 247ab3ac-de0b-4752-b03d-b179cf473aa1 | nova | 1e4006db-9206-4077-b8e6-cb3a972f1a0e | | 546a606b-a58b-4612-8fe4-0fdc45104712 | int_net_1 | 009fa910-c7a5-4d06-a3ca-83dac93a926f, ef2dcd07-0116-4974-9a95-f31a2cdfc5bc | +--------------------------------------+-----------+----------------------------------------------------------------------------+ [2019-02-27 06:15:26] (tester) [stack@undercloud-0 ~]$ openstack subnet list +--------------------------------------+---------------+--------------------------------------+-------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------------+--------------------------------------+-------------+ | 009fa910-c7a5-4d06-a3ca-83dac93a926f | subnet_ipv4_1 | 546a606b-a58b-4612-8fe4-0fdc45104712 | 10.0.1.0/24 | | ef2dcd07-0116-4974-9a95-f31a2cdfc5bc | subnet_ipv6_1 | 546a606b-a58b-4612-8fe4-0fdc45104712 | 2001::/64 | +--------------------------------------+---------------+--------------------------------------+-------------+ [2019-02-27 06:15:49] (tester) [stack@undercloud-0 ~]$ 3. Create lb tree with the following action ( openstack stack create -t filename.yaml lbtree) http://paste.openstack.org/show/746413/ Actual results: u will have 2 ipv4 members in the create pool. delete those and create 2 members with the ipv6 addresses of the existing VMs. test curl ipv6_VIP (via the subnet namespace on the networker node) curl response takes too long Expected results: curl response should be faster (significantly)
Alex, could you also reproduce this issue on standalone topology and non-composable?
(In reply to Carlos Goncalves from comment #2) > Alex, could you also reproduce this issue on standalone topology and > non-composable? Non composable setup = not reproduces. [root@controller-0 ~]# time ip net exe qdhcp-10efe81b-17c7-40cf-87e5-7873e80ccb28 curl -g -6 'http://[2001::f816:3eff:fec5:afb4]:80/' lbtree-server1-tjdzejeodzv6 real 0m0.042s user 0m0.001s sys 0m0.008s [root@controller-0 ~]# time ip net exe qdhcp-10efe81b-17c7-40cf-87e5-7873e80ccb28 curl -g -6 'http://[2001::f816:3eff:fec5:afb4]:80/' lbtree-server2-wjqnnog2fw7m real 0m0.038s user 0m0.002s sys 0m0.006s [root@controller-0 ~]# time ip net exe qdhcp-10efe81b-17c7-40cf-87e5-7873e80ccb28 curl -g -6 'http://[2001::f816:3eff:fec5:afb4]:80/' lbtree-server1-tjdzejeodzv6 real 0m0.044s user 0m0.002s sys 0m0.009s [root@controller-0 ~]# time ip net exe qdhcp-10efe81b-17c7-40cf-87e5-7873e80ccb28 curl -g -6 'http://[2001::f816:3eff:fec5:afb4]:80/' lbtree-server2-wjqnnog2fw7m real 0m0.039s user 0m0.003s sys 0m0.006s [root@controller-0 ~]#
Thanks, Alex. Any chance we could get access to that deployment? Otherwise I'll try to deploy on one of my machines.
(In reply to Carlos Goncalves from comment #4) > Thanks, Alex. Any chance we could get access to that deployment? Otherwise > I'll try to deploy on one of my machines. Hi Carlos, Sorry I do not have the deployment already and do not have an extra host to provide one =\ Will be happy to help if needed :)
Greg will try to reproduce on OSP 13 composable and share any findings that he may find interesting.
Greg, I was able to reproduce it on my machine, it seems like the long response happens only when LB is deployed as ACTIVE/BACKUP (in a IPV6 Composable environment). I tried both SINGLE and ACTIVE/BACKUP modes. When working as a SINGLE Octavia load balancer: (overcloud) [stack@undercloud-0 ~]$ openstack network list +--------------------------------------+-------------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+-------------+----------------------------------------------------------------------------+ | 2f511b25-b462-4d87-94f3-c83cf5c1fda3 | nova | c0b7ec83-d0b9-4fcf-87f5-6b9c7670f8a9 | | aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f | int_net_1 | 0875b382-7089-4828-9329-6ab66cc6baf3, 417c3625-ecf7-41a2-aa6c-4ca98cea1476 | | cded3b69-932c-40e3-bab5-18ac22e395c3 | lb-mgmt-net | 75b26dd6-a00d-4604-8cab-54861e6838ee | +--------------------------------------+-------------+----------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack subnet list +--------------------------------------+-----------------+--------------------------------------+---------------+ | ID | Name | Network | Subnet | +--------------------------------------+-----------------+--------------------------------------+---------------+ | 0875b382-7089-4828-9329-6ab66cc6baf3 | subnet_ipv4_1 | aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f | 10.0.1.0/24 | | 417c3625-ecf7-41a2-aa6c-4ca98cea1476 | subnet_ipv6_1 | aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f | 2001::/64 | | 75b26dd6-a00d-4604-8cab-54861e6838ee | lb-mgmt-subnet | cded3b69-932c-40e3-bab5-18ac22e395c3 | 172.24.0.0/16 | | c0b7ec83-d0b9-4fcf-87f5-6b9c7670f8a9 | external_subnet | 2f511b25-b462-4d87-94f3-c83cf5c1fda3 | 10.0.0.0/24 | +--------------------------------------+-----------------+--------------------------------------+---------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list +--------------------------------------+-----------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+-----------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | 7d48c505-8bc7-4715-bf11-e624e2ade2f8 | lbtreevmshttp-loadbalancer-bwfrm6il2xe7 | ac822737436343e7b1353d5421ad1eeb | 2001::f816:3eff:fe2e:b235 | ACTIVE | octavia | +--------------------------------------+-----------------------------------------+----------------------------------+---------------------------+---------------------+----------+ From the Networker node: Curl directly to server1 and server2 (members): [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe22:2407]:8080/' lbtreevmshttp-server2-xup5l6tgo6us real 0m0.031s user 0m0.003s sys 0m0.004s [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe94:7478]:8080/' lbtreevmshttp-server1-l5gccnbbnzmy real 0m0.021s user 0m0.004s sys 0m0.005s Curl several times to load balancer's VIP: [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe2e:b235]:80/' lbtreevmshttp-server2-xup5l6tgo6us real 0m0.040s user 0m0.000s sys 0m0.009s [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe2e:b235]:80/' lbtreevmshttp-server1-l5gccnbbnzmy real 0m0.039s user 0m0.002s sys 0m0.006s [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe2e:b235]:80/' lbtreevmshttp-server2-xup5l6tgo6us real 0m0.037s user 0m0.004s sys 0m0.005s [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe2e:b235]:80/' lbtreevmshttp-server1-l5gccnbbnzmy real 0m0.038s user 0m0.004s sys 0m0.004s [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe2e:b235]:80/' lbtreevmshttp-server2-xup5l6tgo6us real 0m0.037s user 0m0.003s sys 0m0.007s [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe2e:b235]:80/' lbtreevmshttp-server1-l5gccnbbnzmy real 0m0.040s user 0m0.003s sys 0m0.006s [root@networker-0 ~]# time ip net ex qdhcp-aaa6a97c-3ad6-4c50-80c8-ea6fc8bffe7f curl -g -6 'http://[2001::f816:3eff:fe2e:b235]:80/' lbtreevmshttp-server2-xup5l6tgo6us real 0m0.039s user 0m0.002s sys 0m0.006s The load balancer functions as expected. ------------------ When working as a ACTIVE/BACKUP Octavia load balancer: From the Networker node: (overcloud) [stack@undercloud-0 ~]$ openstack server list --all +--------------------------------------+----------------------------------------------+--------+--------------------------------------------------------------+----------------------------------------+---------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------------------------------------------+--------+--------------------------------------------------------------+----------------------------------------+---------------+ | a3c607b9-1e91-414e-84b4-55091a00028c | amphora-811a0989-bf09-4c6e-a22c-3a4c692e0308 | ACTIVE | lb-mgmt-net=172.24.0.3; int_net_1=2001::f816:3eff:fef5:f17b | octavia-amphora-13.0-20190304.1.x86_64 | | | 74ddcea9-af58-4cc4-bd5f-84580e27c389 | amphora-58e736e5-e3fd-4ff8-a5b0-037bfd1489bd | ACTIVE | lb-mgmt-net=172.24.0.11; int_net_1=2001::f816:3eff:feec:44bb | octavia-amphora-13.0-20190304.1.x86_64 | | | 3637dd26-a504-403d-b235-293983b18b38 | lbtree-server2-l5oomv55523k | ACTIVE | int_net_1=2001::f816:3eff:fefb:cd50, 10.0.1.13 | cirros35 | cirros_flavor | | f43de376-35dd-4e85-a9b7-a08aca68262e | lbtree-server1-o455x6ntmxju | ACTIVE | int_net_1=2001::f816:3eff:fe9e:64b4, 10.0.1.19 | cirros35 | cirros_flavor | +--------------------------------------+----------------------------------------------+--------+--------------------------------------------------------------+----------------------------------------+---------------+ Curl directly to member servers: [root@networker-0 ~]# time ip net ex qdhcp-c91c9379-feea-4259-9c9e-6fab84f1dafe curl -g -6 http://[2001::f816:3eff:fefb:cd50]:8080/ lbtree-server2-l5oomv55523k real 0m0.023s user 0m0.004s sys 0m0.008s [root@networker-0 ~]# time ip net ex qdhcp-c91c9379-feea-4259-9c9e-6fab84f1dafe curl -g -6 http://[2001::f816:3eff:fefb:cd50]:8080/ lbtree-server2-l5oomv55523k real 0m0.019s user 0m0.003s sys 0m0.005s [root@networker-0 ~]# time ip net ex qdhcp-c91c9379-feea-4259-9c9e-6fab84f1dafe curl -g -6 http://[2001::f816:3eff:fe9e:64b4]:8080/ lbtree-server1-o455x6ntmxju real 0m0.036s user 0m0.003s sys 0m0.006s [root@networker-0 ~]# time ip net ex qdhcp-c91c9379-feea-4259-9c9e-6fab84f1dafe curl -g -6 http://[2001::f816:3eff:fe9e:64b4]:8080/ lbtree-server1-o455x6ntmxju real 0m0.033s user 0m0.002s sys 0m0.007s Looks fast. Curl to the load balancer's VIP: [root@networker-0 ~]# time ip net ex qdhcp-c91c9379-feea-4259-9c9e-6fab84f1dafe curl -g -6 'http://[2001::f816:3eff:feae:f0d9]:80/' lbtree-server1-o455x6ntmxju real 0m4.325s user 0m0.005s sys 0m0.007s [root@networker-0 ~]# [root@networker-0 ~]# [root@networker-0 ~]# time ip net ex qdhcp-c91c9379-feea-4259-9c9e-6fab84f1dafe curl -g -6 'http://[2001::f816:3eff:feae:f0d9]:80/' lbtree-server2-l5oomv55523k real 0m15.180s user 0m0.004s sys 0m0.005s [root@networker-0 ~]# [root@networker-0 ~]# time ip net ex qdhcp-c91c9379-feea-4259-9c9e-6fab84f1dafe curl -g -6 'http://[2001::f816:3eff:feae:f0d9]:80/' lbtree-server1-o455x6ntmxju real 0m15.069s user 0m0.004s sys 0m0.005s [root@networker-0 ~]# The response here is very slow. More info on the environment: (undercloud) [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 13 -p 2019-03-18.1 (overcloud) [stack@undercloud-0 ~]$ openstack network list +--------------------------------------+-------------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+-------------+----------------------------------------------------------------------------+ | 86c9cac6-3bd1-4a32-98e1-4d5e8f75d498 | nova | da253722-b55a-4a43-856b-34f2526dcc61 | | c91c9379-feea-4259-9c9e-6fab84f1dafe | int_net_1 | 0cf92b53-8ec3-4254-9d21-348cddef5d2f, 9547bcb7-86b9-4265-8225-7b355167ff39 | | edeec527-95b0-4f5d-8b62-c46fa9071f53 | lb-mgmt-net | d02aeb45-82e1-454d-ba9d-900116f97058 | +--------------------------------------+-------------+----------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack subnet list +--------------------------------------+-----------------+--------------------------------------+---------------+ | ID | Name | Network | Subnet | +--------------------------------------+-----------------+--------------------------------------+---------------+ | 0cf92b53-8ec3-4254-9d21-348cddef5d2f | subnet_ipv6_1 | c91c9379-feea-4259-9c9e-6fab84f1dafe | 2001::/64 | | 9547bcb7-86b9-4265-8225-7b355167ff39 | subnet_ipv4_1 | c91c9379-feea-4259-9c9e-6fab84f1dafe | 10.0.1.0/24 | | d02aeb45-82e1-454d-ba9d-900116f97058 | lb-mgmt-subnet | edeec527-95b0-4f5d-8b62-c46fa9071f53 | 172.24.0.0/16 | | da253722-b55a-4a43-856b-34f2526dcc61 | external_subnet | 86c9cac6-3bd1-4a32-98e1-4d5e8f75d498 | 10.0.0.0/24 | +--------------------------------------+-----------------+--------------------------------------+---------------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list +--------------------------------------+----------------------------------+----------------------------------+---------------------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+----------------------------------+----------------------------------+---------------------------+---------------------+----------+ | 625f00a7-c83f-4469-b013-28d8f2e8310e | lbtree-loadbalancer-yfw5ljnywcs4 | 8061e7bc5c704a48abbc6a66e56a736a | 2001::f816:3eff:feae:f0d9 | ACTIVE | octavia | +--------------------------------------+----------------------------------+----------------------------------+---------------------------+---------------------+----------+ (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ | 58e736e5-e3fd-4ff8-a5b0-037bfd1489bd | 625f00a7-c83f-4469-b013-28d8f2e8310e | ALLOCATED | BACKUP | 172.24.0.11 | 2001::f816:3eff:feae:f0d9 | | 811a0989-bf09-4c6e-a22c-3a4c692e0308 | 625f00a7-c83f-4469-b013-28d8f2e8310e | ALLOCATED | MASTER | 172.24.0.3 | 2001::f816:3eff:feae:f0d9 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ Greg, you are welcome to connect to my server and check it out :)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1744