Description of problem: Loadbalancer pool members in provisioning_status of ERROR, services are inaccessible. Version-Release number of selected component (if applicable): 3.11.394 How reproducible: Unclear at this stage. It doesn't occur all the time. Steps to Reproduce: 1. Restart/recreate pods 2. Check if OpenShfit service is working, if not: 3. Check loadbalancer pool member's provisioning_status Actual results: Service is not accessible Expected results: Service is accessible Additional info:
Some info: ========== Check service in question (not responding to requests): ------------------------------------------------------- $ openstack loadbalancer list |grep org1-abcservices | d43094a6-6944-4304-bb3c-ccfbde4acd60 | env-myorg-projectabc/org1-abcservices | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.130.132 | ACTIVE | amphora | Check pools and their status: ----------------------------- $ openstack loadbalancer pool list --loadbalancer d43094a6-6944-4304-bb3c-ccfbde4acd60 +--------------------------------------+-------------------------------------------------+----------------------------------+---------------------+----------+--------------+----------------+ | id | name | project_id | provisioning_status | protocol | lb_algorithm | admin_state_up | +--------------------------------------+-------------------------------------------------+----------------------------------+---------------------+----------+--------------+----------------+ | 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 | env-myorg-projectabc/org1-abcservices:TCP:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ACTIVE | TCP | ROUND_ROBIN | True | | dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f | env-myorg-projectabc/org1-abcservices:TCP:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ACTIVE | TCP | ROUND_ROBIN | True | +--------------------------------------+-------------------------------------------------+----------------------------------+---------------------+----------+--------------+----------------+ $ openstack loadbalancer pool show 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 +----------------------+-------------------------------------------------+ | Field | Value | +----------------------+-------------------------------------------------+ | admin_state_up | True | | created_at | 2021-03-15T06:27:14 | | description | | | healthmonitor_id | | | id | 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 | | lb_algorithm | ROUND_ROBIN | | listeners | 31f6bdcc-70a3-483d-8a31-dfaf48ff0f48 | | loadbalancers | d43094a6-6944-4304-bb3c-ccfbde4acd60 | | members | df192470-8525-4140-8786-2208f8e2ba0b | | | 19234407-7358-48b2-b788-a0868c335983 | | | 3ed08da5-95db-4db0-acfd-d9f3b58d3acc | | name | env-myorg-projectabc/org1-abcservices:TCP:8080 | | operating_status | ONLINE | | project_id | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | | protocol | TCP | | provisioning_status | ACTIVE | | session_persistence | None | | updated_at | 2021-04-27T21:41:08 | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | +----------------------+-------------------------------------------------+ $ openstack loadbalancer pool show dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f +----------------------+-------------------------------------------------+ | Field | Value | +----------------------+-------------------------------------------------+ | admin_state_up | True | | created_at | 2021-03-15T06:27:16 | | description | | | healthmonitor_id | | | id | dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f | | lb_algorithm | ROUND_ROBIN | | listeners | 4950c925-be6e-453d-805c-306aec00bb32 | | loadbalancers | d43094a6-6944-4304-bb3c-ccfbde4acd60 | | members | 86d83316-9698-4e71-9e78-1fc5eb26c99d | | | 335f7491-d921-4270-aafe-adb8161dbccd | | | 096f247d-efb3-401d-95b2-22c7d487b37e | | | d5ca17d9-c7eb-41e0-bcd0-f1a1ab04cdf4 | | name | env-myorg-projectabc/org1-abcservices:TCP:9090 | | operating_status | ONLINE | | project_id | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | | protocol | TCP | | provisioning_status | ACTIVE | | session_persistence | None | | updated_at | 2021-04-22T04:03:27 | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | +----------------------+-------------------------------------------------+ Check the status of members of the pools: ----------------------------------------- $ openstack loadbalancer member list dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | 86d83316-9698-4e71-9e78-1fc5eb26c99d | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-c2fq7:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.156 | 9090 | NO_MONITOR | 1 | | 335f7491-d921-4270-aafe-adb8161dbccd | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-67jh5:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.219 | 9090 | NO_MONITOR | 1 | | 096f247d-efb3-401d-95b2-22c7d487b37e | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-8xhhj:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.231 | 9090 | NO_MONITOR | 1 | | d5ca17d9-c7eb-41e0-bcd0-f1a1ab04cdf4 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-dx222:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.245 | 9090 | NO_MONITOR | 1 | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ $ openstack loadbalancer member list 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | df192470-8525-4140-8786-2208f8e2ba0b | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-c2fq7:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.156 | 8080 | NO_MONITOR | 1 | | 19234407-7358-48b2-b788-a0868c335983 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-67jh5:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.219 | 8080 | NO_MONITOR | 1 | | 3ed08da5-95db-4db0-acfd-d9f3b58d3acc | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-8xhhj:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.231 | 8080 | NO_MONITOR | 1 | | d982799b-6241-4760-b6b7-774c6f4fb796 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-dx222:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.245 | 8080 | NO_MONITOR | 1 | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ Attempt to manually delete and add a member to see if the provisioning_status can be changed: ---------------------------------------------------------------------------------------------- $ openstack loadbalancer member delete 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 d982799b-6241-4760-b6b7-774c6f4fb796 $ openstack loadbalancer member list 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | df192470-8525-4140-8786-2208f8e2ba0b | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-c2fq7:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.156 | 8080 | NO_MONITOR | 1 | | 19234407-7358-48b2-b788-a0868c335983 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-67jh5:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.219 | 8080 | NO_MONITOR | 1 | | 3ed08da5-95db-4db0-acfd-d9f3b58d3acc | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-8xhhj:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.231 | 8080 | NO_MONITOR | 1 | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ $ openstack loadbalancer member create --address XX.XX.9.245 --name env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-dx222:8080 --protocol-port 8080 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 Pool 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 is immutable and cannot be updated. (HTTP 409) (Request-ID: req-45b61c91-4bf5-4580-b18c-25769ab0c8a7)
Because there is no easy way to reproduce is we simulated a case where one of the members of the pool is in ERROR state 1. Used the following to create pods and a service deployment.yaml =================== apiVersion: apps/v1 kind: Deployment metadata: name: demo labels: app: demo spec: replicas: 3 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: demo image: kuryr/demo ports: - containerPort: 8080 svc.yaml ======== apiVersion: v1 kind: Service metadata: name: demo labels: app: demo spec: selector: app: demo ports: - port: 80 protocol: TCP targetPort: 8080 2. Check service connectivity from the pods to the service 3. Get the the pool ID $ openstack loadbalancer pool list |grep demo | 084de820-0945-4aaf-a5f9-e259973bf98a | default/demo:TCP:80 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | TCP | ROUND_ROBIN | True | 4 Get one of the members ID $ openstack loadbalancer member list 084de820-0945-4aaf-a5f9-e259973bf98a +--------------------------------------+-----------------------------------+----------------------------------+---------------------+-------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+-----------------------------------+----------------------------------+---------------------+-------------+---------------+------------------+--------+ | ecb0230e-db93-49bd-88a0-e526189295c3 | default/demo-68dbc445d-zt9vw:8080 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | 10.11.4.108 | 8080 | NO_MONITOR | 1 | | 7975067a-828f-4f21-9807-530385051763 | default/demo-68dbc445d-w5tx2:8080 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | 10.11.4.112 | 8080 | NO_MONITOR | 1 | | 1f8b2119-eb67-4c82-a312-8fbe37bc47b0 | default/demo-68dbc445d-75mzk:8080 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | 10.11.4.199 | 8080 | NO_MONITOR | 1 | +--------------------------------------+-----------------------------------+----------------------------------+---------------------+-------------+---------------+------------------+--------+ 5. Updated the state of the member to ERRRO . ~/stackrc && ssh heat-admin@$(openstack server list -f value -c Name -c Networks | grep controller-0 | awk -F= '{print $2}')# On osp13: sudo docker exec -uroot -it galera-bundle-docker-0 mysql MariaDB [octavia]> update member set provisioning_status='ERROR' where id='ecb0230e-db93-49bd-88a0-e526189295c3'; 6. Removed openstack.org/kuryr-lbaas-state from the ep to trigger Kuryr $ oc edit ep demo 7. Verified the service is avaialble from the pods and all the members are ACTIVE Verified using OCPv3.11.448 OSP13 2021-03-24.1/RH7-RHOS-13.0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 3.11.452 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2150