Bug 1717470

Summary: Active/Standby fails with RHEL8 based amphora images
Product: Red Hat OpenStack Reporter: Nir Magnezi <nmagnezi>
Component: openstack-octaviaAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED WORKSFORME QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: cgoncalves, ihrachys, lpeer, majopela, scohen
Target Milestone: betaKeywords: Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-02 07:03:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1698576    

Description Nir Magnezi 2019-06-05 14:31:37 UTC 
Description of problem:
=======================
While working on bug 1623857, I noticed that when:

loadbalancer_topology = ACTIVE_STANDBY

loadbalancer creation fails with the following:
HA

DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] Post-VIP-Plugging with vrrp_ip 10.0.0.29 vrrp_port 97eac854-ea20-4bdd-a0f1-6c6a543a1bcd {{(pid=30963) post_vip_plug /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:248}}
> /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py(466)request()
-> cfg_ha_amp = CONF.haproxy_amphora
(Pdb) c
DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url plug/vip/10.0.0.58 {{(pid=30963) request /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:466}}
DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] request url https://192.168.0.22:9443/0.5/plug/vip/10.0.0.58 {{(pid=30963) request /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:466}}
DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connected to amphora. Response: <Response [500]> {{(pid=30963) request /opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py:466}}
ERROR octavia.amphorae.drivers.haproxy.exceptions [-] Amphora agent returned unexpected result code 500 with response {u'message': u'Error plugging VIP', u'details': u"Could not load file '/etc/sysconfig/network-scripts/ifcfg-eth1'\nError: unknown connection '/etc/sysconfig/network-scripts/ifcfg-eth1'.\n"}


Version-Release number of selected component (if applicable):
=============================================================
OSP15

How reproducible:
=================
Always

Steps to Reproduce:
===================
1. Build an RHEL8 based amphora.
2. Upload to glance and make sure to tag it
3. Create a loadbalancer via Octavia API


Additional info:
================
Fails with SELinux permissive as well.
Currently set to medium severity, since active/standby is not currently supported in OSP.
Comment 4 Carlos Goncalves 2019-10-02 07:03:44 UTC 
I cannot reproduce this on an OSP 15 GA deployment. It might have been fixed between the time the issue was reported and OSP 15 GA.

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer show a79600ff-5b9f-4d05-9920-f8cbfc1f37cb
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2019-10-02T06:54:44                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | a79600ff-5b9f-4d05-9920-f8cbfc1f37cb |
| listeners           |                                      |
| name                |                                      |
| operating_status    | ONLINE                               |
| pools               |                                      |
| project_id          | 7a761c01f27f4dd69d8020af23f4b8b0     |
| provider            | amphora                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2019-10-02T06:57:22                  |
| vip_address         | 10.99.0.96                           |
| vip_network_id      | 7d376e9b-5324-423f-8fde-7c67131028c4 |
| vip_port_id         | 957948ff-d797-4227-94a0-49805ff269b7 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 9ae559d5-7d76-4179-bfd7-2599df4b631d |
+---------------------+--------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+--------+---------------+------------+
| id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip      |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+------------+
| 20bd77cd-cc04-4a82-b059-33e215845c53 | a79600ff-5b9f-4d05-9920-f8cbfc1f37cb | ALLOCATED | BACKUP | 172.24.3.151  | 10.99.0.96 |
| 7c571878-6097-439d-992d-dfeec01520f4 | a79600ff-5b9f-4d05-9920-f8cbfc1f37cb | ALLOCATED | MASTER | 172.24.2.138  | 10.99.0.96 |
+--------------------------------------+--------------------------------------+-----------+--------+---------------+------------+

[cloud-user@amphora-7c571878-6097-439d-992d-dfeec01520f4 ~]$ sudo ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:f9:73:e6 brd ff:ff:ff:ff:ff:ff
    inet 172.24.2.138/16 brd 172.24.255.255 scope global dynamic noprefixroute eth0
       valid_lft 42914sec preferred_lft 42914sec
    inet6 fe80::f816:3eff:fef9:73e6/64 scope link 
       valid_lft forever preferred_lft forever

[cloud-user@amphora-7c571878-6097-439d-992d-dfeec01520f4 ~]$ sudo ip netns exec amphora-haproxy ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:db:96:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.99.0.221/24 brd 10.99.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 10.99.0.96/32 scope global eth1
       valid_lft forever preferred_lft forever