Bug 1954353
| Summary: | Kuryr controller not creating healthy pool members for service | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mohammad <mahmad> |
| Component: | Networking | Assignee: | Maysa Macedo <mdemaced> |
| Networking sub component: | kuryr | QA Contact: | Itzik Brown <itbrown> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | medium | CC: | dsedgmen, mdemaced |
| Version: | 3.11.0 | Keywords: | Triaged |
| Target Milestone: | --- | ||
| Target Release: | 3.11.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-09 17:06:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Mohammad
2021-04-28 00:18:35 UTC
Some info: ========== Check service in question (not responding to requests): ------------------------------------------------------- $ openstack loadbalancer list |grep org1-abcservices | d43094a6-6944-4304-bb3c-ccfbde4acd60 | env-myorg-projectabc/org1-abcservices | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | XX.XX.130.132 | ACTIVE | amphora | Check pools and their status: ----------------------------- $ openstack loadbalancer pool list --loadbalancer d43094a6-6944-4304-bb3c-ccfbde4acd60 +--------------------------------------+-------------------------------------------------+----------------------------------+---------------------+----------+--------------+----------------+ | id | name | project_id | provisioning_status | protocol | lb_algorithm | admin_state_up | +--------------------------------------+-------------------------------------------------+----------------------------------+---------------------+----------+--------------+----------------+ | 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 | env-myorg-projectabc/org1-abcservices:TCP:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ACTIVE | TCP | ROUND_ROBIN | True | | dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f | env-myorg-projectabc/org1-abcservices:TCP:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ACTIVE | TCP | ROUND_ROBIN | True | +--------------------------------------+-------------------------------------------------+----------------------------------+---------------------+----------+--------------+----------------+ $ openstack loadbalancer pool show 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 +----------------------+-------------------------------------------------+ | Field | Value | +----------------------+-------------------------------------------------+ | admin_state_up | True | | created_at | 2021-03-15T06:27:14 | | description | | | healthmonitor_id | | | id | 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 | | lb_algorithm | ROUND_ROBIN | | listeners | 31f6bdcc-70a3-483d-8a31-dfaf48ff0f48 | | loadbalancers | d43094a6-6944-4304-bb3c-ccfbde4acd60 | | members | df192470-8525-4140-8786-2208f8e2ba0b | | | 19234407-7358-48b2-b788-a0868c335983 | | | 3ed08da5-95db-4db0-acfd-d9f3b58d3acc | | name | env-myorg-projectabc/org1-abcservices:TCP:8080 | | operating_status | ONLINE | | project_id | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | | protocol | TCP | | provisioning_status | ACTIVE | | session_persistence | None | | updated_at | 2021-04-27T21:41:08 | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | +----------------------+-------------------------------------------------+ $ openstack loadbalancer pool show dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f +----------------------+-------------------------------------------------+ | Field | Value | +----------------------+-------------------------------------------------+ | admin_state_up | True | | created_at | 2021-03-15T06:27:16 | | description | | | healthmonitor_id | | | id | dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f | | lb_algorithm | ROUND_ROBIN | | listeners | 4950c925-be6e-453d-805c-306aec00bb32 | | loadbalancers | d43094a6-6944-4304-bb3c-ccfbde4acd60 | | members | 86d83316-9698-4e71-9e78-1fc5eb26c99d | | | 335f7491-d921-4270-aafe-adb8161dbccd | | | 096f247d-efb3-401d-95b2-22c7d487b37e | | | d5ca17d9-c7eb-41e0-bcd0-f1a1ab04cdf4 | | name | env-myorg-projectabc/org1-abcservices:TCP:9090 | | operating_status | ONLINE | | project_id | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | | protocol | TCP | | provisioning_status | ACTIVE | | session_persistence | None | | updated_at | 2021-04-22T04:03:27 | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | +----------------------+-------------------------------------------------+ Check the status of members of the pools: ----------------------------------------- $ openstack loadbalancer member list dd39bf03-14c3-4ed6-b27d-e8f9a95ac08f +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | 86d83316-9698-4e71-9e78-1fc5eb26c99d | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-c2fq7:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.156 | 9090 | NO_MONITOR | 1 | | 335f7491-d921-4270-aafe-adb8161dbccd | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-67jh5:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.219 | 9090 | NO_MONITOR | 1 | | 096f247d-efb3-401d-95b2-22c7d487b37e | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-8xhhj:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.231 | 9090 | NO_MONITOR | 1 | | d5ca17d9-c7eb-41e0-bcd0-f1a1ab04cdf4 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-dx222:9090 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.245 | 9090 | NO_MONITOR | 1 | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ $ openstack loadbalancer member list 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | df192470-8525-4140-8786-2208f8e2ba0b | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-c2fq7:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.156 | 8080 | NO_MONITOR | 1 | | 19234407-7358-48b2-b788-a0868c335983 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-67jh5:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.219 | 8080 | NO_MONITOR | 1 | | 3ed08da5-95db-4db0-acfd-d9f3b58d3acc | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-8xhhj:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.231 | 8080 | NO_MONITOR | 1 | | d982799b-6241-4760-b6b7-774c6f4fb796 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-dx222:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.245 | 8080 | NO_MONITOR | 1 | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ Attempt to manually delete and add a member to see if the provisioning_status can be changed: ---------------------------------------------------------------------------------------------- $ openstack loadbalancer member delete 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 d982799b-6241-4760-b6b7-774c6f4fb796 $ openstack loadbalancer member list 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | df192470-8525-4140-8786-2208f8e2ba0b | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-c2fq7:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.156 | 8080 | NO_MONITOR | 1 | | 19234407-7358-48b2-b788-a0868c335983 | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-67jh5:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.219 | 8080 | NO_MONITOR | 1 | | 3ed08da5-95db-4db0-acfd-d9f3b58d3acc | env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-8xhhj:8080 | f1f9eba4ac7c4f1bbd89cc5f4b68d720 | ERROR | XX.XX.9.231 | 8080 | NO_MONITOR | 1 | +--------------------------------------+--------------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ $ openstack loadbalancer member create --address XX.XX.9.245 --name env-myorg-projectabc/org1-abcservices-7bcb5cf9bb-dx222:8080 --protocol-port 8080 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 Pool 2d2f78a5-8cd7-48c4-8fb8-83e3bd1a87f8 is immutable and cannot be updated. (HTTP 409) (Request-ID: req-45b61c91-4bf5-4580-b18c-25769ab0c8a7) Because there is no easy way to reproduce is we simulated a case where one of the members of the pool is in ERROR state
1. Used the following to create pods and a service
deployment.yaml
===================
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo
labels:
app: demo
spec:
replicas: 3
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
containers:
- name: demo
image: kuryr/demo
ports:
- containerPort: 8080
svc.yaml
========
apiVersion: v1
kind: Service
metadata:
name: demo
labels:
app: demo
spec:
selector:
app: demo
ports:
- port: 80
protocol: TCP
targetPort: 8080
2. Check service connectivity from the pods to the service
3. Get the the pool ID
$ openstack loadbalancer pool list |grep demo
| 084de820-0945-4aaf-a5f9-e259973bf98a | default/demo:TCP:80 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | TCP | ROUND_ROBIN | True |
4 Get one of the members ID
$ openstack loadbalancer member list 084de820-0945-4aaf-a5f9-e259973bf98a
+--------------------------------------+-----------------------------------+----------------------------------+---------------------+-------------+---------------+------------------+--------+
| id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight |
+--------------------------------------+-----------------------------------+----------------------------------+---------------------+-------------+---------------+------------------+--------+
| ecb0230e-db93-49bd-88a0-e526189295c3 | default/demo-68dbc445d-zt9vw:8080 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | 10.11.4.108 | 8080 | NO_MONITOR | 1 |
| 7975067a-828f-4f21-9807-530385051763 | default/demo-68dbc445d-w5tx2:8080 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | 10.11.4.112 | 8080 | NO_MONITOR | 1 |
| 1f8b2119-eb67-4c82-a312-8fbe37bc47b0 | default/demo-68dbc445d-75mzk:8080 | 00724efa760541d2b940ecb038722ca4 | ACTIVE | 10.11.4.199 | 8080 | NO_MONITOR | 1 |
+--------------------------------------+-----------------------------------+----------------------------------+---------------------+-------------+---------------+------------------+--------+
5. Updated the state of the member to ERRRO
. ~/stackrc && ssh heat-admin@$(openstack server list -f value -c Name -c Networks | grep controller-0 | awk -F= '{print $2}')# On osp13:
sudo docker exec -uroot -it galera-bundle-docker-0 mysql
MariaDB [octavia]> update member set provisioning_status='ERROR' where id='ecb0230e-db93-49bd-88a0-e526189295c3';
6. Removed openstack.org/kuryr-lbaas-state from the ep to trigger Kuryr
$ oc edit ep demo
7. Verified the service is avaialble from the pods and all the members are ACTIVE
Verified using
OCPv3.11.448
OSP13 2021-03-24.1/RH7-RHOS-13.0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 3.11.452 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2150 |