Please include the following upstream fix for OSP13: crash with a segfault due to a known bug[1]. This patch resolves both issues and allows load balancers with IPv6 VIP addresses to be created in active/standby topology. Conflicts: octavia/amphorae/backends/agent/agent_jinja_cfg.py octavia/amphorae/backends/agent/api_server/listener.py octavia/amphorae/backends/agent/api_server/plug.py octavia/amphorae/backends/agent/api_server/templates/plug_vip_ethX.conf.j2 octavia/amphorae/backends/agent/templates/amphora_agent_conf.template octavia/tests/functional/amphorae/backend/agent/api_server/test_server.py octavia/tests/unit/amphorae/backends/agent/test_agent_jinja_cfg.py Note that this patch was adapted for the stable/queens branch. [1] https://github.com/acassen/keepalived/issues/457
Bruna, Thank you for bringing this to our attention. At first, I suspected that this is being caused by a lack of resources in your setup. I noticed Nova rejects the additional instances creation and Octavia reported an exception that matches this situation. However, I was able to synthetically increase the amount of available compute nodes RAM by reducing reserved_host_memory_mb to 512MB (find it on nova.conf). You can either alter this value in the customized Job you use to deploy or pre-allocate more RAM to the compute nodes. On that point, I noticed that the SELinux policies running on the amphora instances block keepalived from working correctly: Mar 10 08:07:53 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 Keepalived_vrrp[3467]: Sending unsolicited Neighbour Advert on eth1 for 2001::f816:3eff:fee4:6cf0 Mar 10 08:07:55 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 systemd: Reloading. Mar 10 08:07:55 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 systemd: Reloading. Mar 10 08:07:56 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 systemd: Reloading Keepalive Daemon (LVS and VRRP). Mar 10 08:07:56 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 systemd: Failed at step EXEC spawning /bin/kill: Permission denied Mar 10 08:07:56 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 systemd: octavia-keepalived.service: control process exited, code=exited status=203 Mar 10 08:07:56 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 systemd: Reload failed for Keepalive Daemon (LVS and VRRP). Mar 10 08:07:56 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 amphora-agent: 2019-03-10 08:07:56.036 3162 DEBUG octavia.amphorae.backends.agent.api_server.keepalived [-] Failed to reload octavia-keepalived service: Command '['/usr/sbin/service', 'octavia-keepalived', 'reload']' returned non-zero exit status 1 Redirecting to /bin/systemctl reload octavia-keepalived.service Mar 10 08:07:56 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 amphora-agent: Job for octavia-keepalived.service failed because the control process exited with error code. See "systemctl status octavia-keepalived.service" and "journalctl -xe" for details. Mar 10 08:07:56 amphora-7ebca2dd-a11f-4198-bad3-85e9c6bef457 amphora-agent: manager_keepalived_service /usr/lib/python2.7/site-packages/octavia/amphorae/backends/agent/api_server/keepalived.py:158 To confirm this indeed resolves the issue, I made a modified copy of the amphora image loaded to your setup and switched SELinux from enforcing to permissive (image named rh76amp-noselinux). I created an IPv6 based loadbalancer and triggered a failover, which now works as expected. I filed a bug[1] (and a fix[2]) against openstack-selinux to mitigate the issue. In the meantime, I marked this bug as blocked by bug 1687321 (check if for more details about the missing policy). [1] https://bugzilla.redhat.com/show_bug.cgi?id=1687321 [2] https://github.com/redhat-openstack/openstack-selinux/pull/30
Moving to ON_QA since bug 1687321 moved to ON_QA The fix I mentioned in comment 21, only changed openstack-selinux. Please test with openstack-selinux-0.8.18-1.el7ost
According to our records, this should be resolved by openstack-octavia-2.0.3-2.el7ost. This build is available now.
[2020-01-02 12:00:55] (overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed 13 -p 2019-11-04.1 [2020-01-02 08:48:35] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer list +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | treeipv6_3-loadbalancer-ymtukqf43uhy | 7929cfc9da1140d8ad5a686de911a367 | 2001::f816:3eff:fe6f:e5f9 | ACTIVE | octavia | +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ [2020-01-02 08:48:42] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer show 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2020-01-02T13:42:34 | | description | | | flavor | | | id | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | | listeners | 1cfcc8e4-ced5-43fc-b7b7-32f2efa4b1c6 | | name | treeipv6_3-loadbalancer-ymtukqf43uhy | | operating_status | ONLINE | | pools | b56055d2-a063-4d10-b601-010a822b92d1 | | project_id | 7929cfc9da1140d8ad5a686de911a367 | | provider | octavia | | provisioning_status | ACTIVE | | updated_at | 2020-01-02T13:44:39 | | vip_address | 2001::f816:3eff:fe6f:e5f9 | | vip_network_id | 6b96fb0f-19fc-49df-9ef0-a9b78af3412a | | vip_port_id | 9f906d1c-b36c-4ca0-b1eb-b936390e75a4 | | vip_qos_policy_id | None | | vip_subnet_id | 46473cfa-391a-4b40-bbfb-1435f8567014 | +---------------------+--------------------------------------+ [2020-01-02 08:48:54] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer listener list +--------------------------------------+--------------------------------------+----------------------------------+----------------------------------+----------+---------------+----------------+ | id | default_pool_id | name | project_id | protocol | protocol_port | admin_state_up | +--------------------------------------+--------------------------------------+----------------------------------+----------------------------------+----------+---------------+----------------+ | 1cfcc8e4-ced5-43fc-b7b7-32f2efa4b1c6 | b56055d2-a063-4d10-b601-010a822b92d1 | treeipv6_3-listener-6jt2zsgeysss | 7929cfc9da1140d8ad5a686de911a367 | HTTP | 80 | True | +--------------------------------------+--------------------------------------+----------------------------------+----------------------------------+----------+---------------+----------------+ [2020-01-02 08:49:13] (tester) [stack@undercloud-0 ~]$ . overcloudrc ; openstack loadbalancer amphora list +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ | 43a2e15d-e5e4-4970-a0b4-b271c2f21458 | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | ALLOCATED | MASTER | 172.24.0.7 | 2001::f816:3eff:fe6f:e5f9 | | 7bb257b5-45aa-4824-953d-a3899755eb19 | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | ALLOCATED | BACKUP | 172.24.0.9 | 2001::f816:3eff:fe6f:e5f9 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ ############## FAILOVER Load Balancer: ############## [2020-01-02 09:30:18] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer failover 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 [2020-01-02 09:30:23] (overcloud) [stack@undercloud-0 ~]$ [2020-01-02 09:31:57] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | treeipv6_3-loadbalancer-ymtukqf43uhy | 7929cfc9da1140d8ad5a686de911a367 | 2001::f816:3eff:fe6f:e5f9 | PENDING_UPDATE | octavia | +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ [2020-01-02 09:32:04] (overcloud) [stack@undercloud-0 ~]$ [2020-01-02 09:32:05] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | treeipv6_3-loadbalancer-ymtukqf43uhy | 7929cfc9da1140d8ad5a686de911a367 | 2001::f816:3eff:fe6f:e5f9 | ACTIVE | octavia | +--------------------------------------+--------------------------------------+----------------------------------+---------------------------+---------------------+----------+ [2020-01-02 09:33:04] (overcloud) [stack@undercloud-0 ~]$ [2020-01-02 09:33:04] (overcloud) [stack@undercloud-0 ~]$ . overcloudrc ; openstack loadbalancer amphora list +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ | 808c3fe3-cf1e-4de8-bc35-b326c960744b | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | ALLOCATED | BACKUP | 172.24.0.12 | 2001::f816:3eff:fe6f:e5f9 | | f69f7eb0-d6af-43ca-af56-6ecea88e493b | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | ALLOCATED | MASTER | 172.24.0.7 | 2001::f816:3eff:fe6f:e5f9 | +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ +--------------------------------------+--------------------------------------+-----------+--------+---------------+---------------------------+ [2020-01-02 11:46:57] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora show 808c3fe3-cf1e-4de8-bc35-b326c960744b +-----------------+--------------------------------------+ | Field | Value | +-----------------+--------------------------------------+ | id | 808c3fe3-cf1e-4de8-bc35-b326c960744b | | loadbalancer_id | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | | compute_id | f0e3b010-b326-45a0-9e4c-1d6975829faa | | lb_network_ip | 172.24.0.12 | | vrrp_ip | 2001::f816:3eff:fe2e:10b5 | | ha_ip | 2001::f816:3eff:fe6f:e5f9 | | vrrp_port_id | 2b722516-a7a3-4b74-bd06-a4b1710213e9 | | ha_port_id | 9f906d1c-b36c-4ca0-b1eb-b936390e75a4 | | cert_expiration | 2022-01-01T14:30:30 | | cert_busy | False | | role | BACKUP | | status | ALLOCATED | | vrrp_interface | eth1 | | vrrp_id | 1 | | vrrp_priority | 90 | | cached_zone | nova | +-----------------+--------------------------------------+ [2020-01-02 11:47:09] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora show f69f7eb0-d6af-43ca-af56-6ecea88e493b +-----------------+--------------------------------------+ | Field | Value | +-----------------+--------------------------------------+ | id | f69f7eb0-d6af-43ca-af56-6ecea88e493b | | loadbalancer_id | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | | compute_id | f226b152-a5f4-4263-ba8c-cfb2c3b5567b | | lb_network_ip | 172.24.0.7 | | vrrp_ip | 2001::f816:3eff:feb4:7e73 | | ha_ip | 2001::f816:3eff:fe6f:e5f9 | | vrrp_port_id | d54b3cb8-f652-42fe-96c7-60a15573125f | | ha_port_id | 9f906d1c-b36c-4ca0-b1eb-b936390e75a4 | | cert_expiration | 2022-01-01T14:31:43 | | cert_busy | False | | role | MASTER | | status | ALLOCATED | | vrrp_interface | eth1 | | vrrp_id | 1 | | vrrp_priority | 100 | | cached_zone | nova | +-----------------+--------------------------------------+ [2020-01-02 11:46:32] (tester) [stack@undercloud-0 ~]$ openstack loadbalancer show 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2020-01-02T13:42:34 | | description | | | flavor | | | id | 2e85532c-2bd0-4a03-9ddb-703541e6b5e0 | | listeners | 1cfcc8e4-ced5-43fc-b7b7-32f2efa4b1c6 | | name | treeipv6_3-loadbalancer-ymtukqf43uhy | | operating_status | ONLINE | | pools | b56055d2-a063-4d10-b601-010a822b92d1 | | project_id | 7929cfc9da1140d8ad5a686de911a367 | | provider | octavia | | provisioning_status | ACTIVE | | updated_at | 2020-01-02T16:35:23 | | vip_address | 2001::f816:3eff:fe6f:e5f9 | | vip_network_id | 6b96fb0f-19fc-49df-9ef0-a9b78af3412a | | vip_port_id | 9f906d1c-b36c-4ca0-b1eb-b936390e75a4 | | vip_qos_policy_id | None | | vip_subnet_id | 46473cfa-391a-4b40-bbfb-1435f8567014 | +---------------------+--------------------------------------+