Bug 1522711

Summary: healthmonitor did not take effect
Product: Red Hat OpenStack Reporter: Carlos Goncalves <cgoncalves>
Component: openstack-neutron-lbaasAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: high Docs Contact:
Priority: high    
Version: 11.0 (Ocata)CC: amuller, apevec, cgoncalves, jlibosva, lhh, mzheng, nyechiel, tfreger
Target Milestone: asyncKeywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-lbaas-10.0.1-8.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1512375 Environment:
Last Closed: 2018-02-13 16:40:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1512375, 1522713    
Bug Blocks:    

Description Carlos Goncalves 2017-12-06 10:01:07 UTC
+++ This bug was initially created as a clone of Bug #1512375 +++

Description of problem:

Create a healthmonitor and add to a pool, shutdown or delete one member but member state is not set to inactive when the max of failed retries is reached for the instance.



Version-Release number of selected component (if applicable):
RHOSP10


How reproducible:


Steps to Reproduce:
1. Create Lbaasv2: 

$ nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks            |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| e7e10318-ac85-4191-8a56-fed46f2422f9 | node1 | ACTIVE | -          | Running     | private=10.10.1.103 |
| 8d691b56-cedb-4723-82f4-65938e2f71a8 | node2 | ACTIVE | -          | Running     | private=10.10.1.102 |
+--------------------------------------+-------+--------+------------+-------------+---------------------+

$ neutron lbaas-loadbalancer-status  041b24a3-55e1-4880-b04f-cf22ab057b30
{
    "loadbalancer": {
        "name": "lb1", 
        "provisioning_status": "ACTIVE", 
        "listeners": [
            {
                "name": "listener1", 
                "provisioning_status": "ACTIVE", 
                "pools": [
                    {
                        "name": "pool1", 
                        "provisioning_status": "ACTIVE", 
                        "healthmonitor": {
                            "provisioning_status": "ACTIVE", 
                            "type": "HTTP", 
                            "id": "083f068f-3ee1-471b-a6ba-2438648078c0", 
                            "name": ""
                        }, 
                        "members": [
                            {
                                "name": "", 
                                "provisioning_status": "ACTIVE", 
                                "address": "10.10.1.103", 
                                "protocol_port": 80, 
                                "id": "97e4de09-7e88-41fb-9e7f-5b061fd4d44f", 
                                "operating_status": "ONLINE"
                            }, 
                            {
                                "name": "", 
                                "provisioning_status": "ACTIVE", 
                                "address": "10.10.1.102", 
                                "protocol_port": 80, 
                                "id": "9bdc780f-2553-40cf-82bf-aa43ab94ce8a", 
                                "operating_status": "ONLINE"
                            }
                        ], 
                        "id": "17c63199-b63e-4f22-83a3-baea98345923", 
                        "operating_status": "ONLINE"
                    }
                ], 
                "l7policies": [], 
                "id": "2c1f8311-d834-45af-8efb-05ff1684e2a6", 
                "operating_status": "ONLINE"
            }
        ], 
        "pools": [
            {
                "name": "pool1", 
                "provisioning_status": "ACTIVE", 
                "healthmonitor": {
                    "provisioning_status": "ACTIVE", 
                    "type": "HTTP", 
                    "id": "083f068f-3ee1-471b-a6ba-2438648078c0", 
                    "name": ""
                }, 
                "members": [
                    {
                        "name": "", 
                        "provisioning_status": "ACTIVE", 
                        "address": "10.10.1.103", 
                        "protocol_port": 80, 
                        "id": "97e4de09-7e88-41fb-9e7f-5b061fd4d44f", 
                        "operating_status": "ONLINE"
                    }, 
                    {
                        "name": "", 
                        "provisioning_status": "ACTIVE", 
                        "address": "10.10.1.102", 
                        "protocol_port": 80, 
                        "id": "9bdc780f-2553-40cf-82bf-aa43ab94ce8a", 
                        "operating_status": "ONLINE"
                    }
                ], 
                "id": "17c63199-b63e-4f22-83a3-baea98345923", 
                "operating_status": "ONLINE"
            }
        ], 
        "id": "041b24a3-55e1-4880-b04f-cf22ab057b30", 
        "operating_status": "ONLINE"
    }
}



2. Delete one of the member:
  
$ nova delete e7e10318-ac85-4191-8a56-fed46f2422f9

$ nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks            |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| 8d691b56-cedb-4723-82f4-65938e2f71a8 | node2 | ACTIVE | -          | Running     | private=10.10.1.102 |
+--------------------------------------+-------+--------+------------+-------------+---------------------+


3. Check lbaas status again:

$ neutron lbaas-loadbalancer-status  041b24a3-55e1-4880-b04f-cf22ab057b30
{
    "loadbalancer": {
        "name": "lb1", 
        "provisioning_status": "ACTIVE", 
        "listeners": [
            {
                "name": "listener1", 
                "provisioning_status": "ACTIVE", 
                "pools": [
                    {
                        "name": "pool1", 
                        "provisioning_status": "ACTIVE", 
                        "healthmonitor": {
                            "provisioning_status": "ACTIVE", 
                            "type": "HTTP", 
                            "id": "083f068f-3ee1-471b-a6ba-2438648078c0", 
                            "name": ""
                        }, 
                        "members": [
                            {
                                "name": "", 
                                "provisioning_status": "ACTIVE", 
                                "address": "10.10.1.103", 
                                "protocol_port": 80, 
                                "id": "97e4de09-7e88-41fb-9e7f-5b061fd4d44f", 
                                "operating_status": "ONLINE"
                            }, 
                            {
                                "name": "", 
                                "provisioning_status": "ACTIVE", 
                                "address": "10.10.1.102", 
                                "protocol_port": 80, 
                                "id": "9bdc780f-2553-40cf-82bf-aa43ab94ce8a", 
                                "operating_status": "ONLINE"
                            }
                        ], 
                        "id": "17c63199-b63e-4f22-83a3-baea98345923", 
                        "operating_status": "ONLINE"
                    }
                ], 
                "l7policies": [], 
                "id": "2c1f8311-d834-45af-8efb-05ff1684e2a6", 
                "operating_status": "ONLINE"
            }
        ], 
        "pools": [
            {
                "name": "pool1", 
                "provisioning_status": "ACTIVE", 
                "healthmonitor": {
                    "provisioning_status": "ACTIVE", 
                    "type": "HTTP", 
                    "id": "083f068f-3ee1-471b-a6ba-2438648078c0", 
                    "name": ""
                }, 
                "members": [
                    {
                        "name": "", 
                        "provisioning_status": "ACTIVE", 
                        "address": "10.10.1.103", 
                        "protocol_port": 80, 
                        "id": "97e4de09-7e88-41fb-9e7f-5b061fd4d44f", 
                        "operating_status": "ONLINE"
                    }, 
                    {
                        "name": "", 
                        "provisioning_status": "ACTIVE", 
                        "address": "10.10.1.102", 
                        "protocol_port": 80, 
                        "id": "9bdc780f-2553-40cf-82bf-aa43ab94ce8a", 
                        "operating_status": "ONLINE"
                    }
                ], 
                "id": "17c63199-b63e-4f22-83a3-baea98345923", 
                "operating_status": "ONLINE"
            }
        ], 
        "id": "041b24a3-55e1-4880-b04f-cf22ab057b30", 
        "operating_status": "ONLINE"
    }
}


Actual results:
The operating_status for member 10.10.1.103 is still online.

Expected results:
The operating_status for member 10.10.1.103 should be offline or error. 


Additional info:
There are some similar issue reported in upstream:
https://bugs.launchpad.net/neutron/+bug/1548774
https://bugs.launchpad.net/octavia/+bug/1607309

--- Additional comment from Jakub Libosvar on 2017-11-13 09:35:13 EST ---

Nir is going to look at this one

Comment 7 Alexander Stafeyev 2018-01-28 08:30:17 UTC
[root@controller-0 ~]# rpm -qa | grep neutron-lba
python-neutron-lbaas-10.0.1-10.el7ost.noarch
openstack-neutron-lbaas-10.0.1-10.el7ost.noarch
[root@controller-0 ~]# 




[stack@undercloud-0 ~]$ nova list --all
+--------------------------------------+---------+----------------------------------+---------+------------+-------------+--------------------------------------------------------------------------+
| ID                                   | Name    | Tenant ID                        | Status  | Task State | Power State | Networks                                                                 |
+--------------------------------------+---------+----------------------------------+---------+------------+-------------+--------------------------------------------------------------------------+
| d92ed31d-f6cf-4c04-b82b-52ac607a043b | server1 | dfb1bbe50c264e2d93c18158f2f19aac | SHUTOFF | -          | Shutdown    | tempest-TestHealthMonitorBasic-1264904937-network=10.100.0.6, 10.0.0.221 |
| 07c570ea-a052-42e0-b189-c3ff6e4d592b | server2 | dfb1bbe50c264e2d93c18158f2f19aac | ACTIVE  | -          | Running     | tempest-TestHealthMonitorBasic-1264904937-network=10.100.0.8, 10.0.0.211 |
+--------------------------------------+---------+----------------------------------+---------+------------+-------------+--------------------------------------------------------------------------+
[stack@undercloud-0 ~]$ 
[stack@undercloud-0 ~]$ 
[stack@undercloud-0 ~]$ 
[stack@undercloud-0 ~]$ 
[stack@undercloud-0 ~]$ 
[stack@undercloud-0 ~]$ neutron lbaas-loadbalancer-status b8b843b9-2e3c-4bd1-9b75-fed558ef901c 
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
{
    "loadbalancer": {
        "name": "", 
        "provisioning_status": "ACTIVE", 
        "listeners": [
            {
                "name": "", 
                "provisioning_status": "ACTIVE", 
                "pools": [
                    {
                        "name": "", 
                        "provisioning_status": "ACTIVE", 
                        "healthmonitor": {
                            "provisioning_status": "ACTIVE", 
                            "type": "HTTP", 
                            "id": "76c9bc96-d3a0-4f58-bf97-618ab8f0a802", 
                            "name": ""
                        }, 
                        "members": [
                            {
                                "name": "", 
                                "provisioning_status": "ACTIVE", 
                                "address": "10.100.0.6", 
                                "protocol_port": 80, 
                                "id": "ab99fbfc-b5a6-4d31-b885-e1722b403709", 
                                "operating_status": "OFFLINE"
                            }, 
                            {
                                "name": "", 
                                "provisioning_status": "ACTIVE", 
                                "address": "10.100.0.8", 
                                "protocol_port": 80, 
                                "id": "ea2f6a6c-16a2-4887-b76d-cc07ca96075a", 
                                "operating_status": "ONLINE"
                            }
                        ], 
                        "id": "bfe5bd5c-87de-4351-b2ac-818364d862f4", 
                        "operating_status": "DEGRADED"
                    }
                ], 
                "l7policies": [], 
                "id": "b664e460-fd82-4123-9664-8d38862fb272", 
                "operating_status": "DEGRADED"
            }
        ], 
        "pools": [
            {
                "name": "", 
                "provisioning_status": "ACTIVE", 
                "healthmonitor": {
                    "provisioning_status": "ACTIVE", 
                    "type": "HTTP", 
                    "id": "76c9bc96-d3a0-4f58-bf97-618ab8f0a802", 
                    "name": ""
                }, 
                "members": [
                    {
                        "name": "", 
                        "provisioning_status": "ACTIVE", 
                        "address": "10.100.0.6", 
                        "protocol_port": 80, 
                        "id": "ab99fbfc-b5a6-4d31-b885-e1722b403709", 
                        "operating_status": "OFFLINE"
                    }, 
                    {
                        "name": "", 
                        "provisioning_status": "ACTIVE", 
                        "address": "10.100.0.8", 
                        "protocol_port": 80, 
                        "id": "ea2f6a6c-16a2-4887-b76d-cc07ca96075a", 
                        "operating_status": "ONLINE"
                    }
                ], 
                "id": "bfe5bd5c-87de-4351-b2ac-818364d862f4", 
                "operating_status": "DEGRADED"
            }
        ], 
        "id": "b8b843b9-2e3c-4bd1-9b75-fed558ef901c", 
        "operating_status": "DEGRADED"
    }

Comment 10 errata-xmlrpc 2018-02-13 16:40:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0304