Bug 1836018
| Summary: | AWS UPI: Both Internal and External load balancers for kube-apiserver should use /readyz | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> | ||||
| Component: | Installer | Assignee: | Russell Teague <rteague> | ||||
| Installer sub component: | openshift-installer | QA Contact: | Yunfei Jiang <yunjiang> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | medium | ||||||
| Priority: | high | CC: | adahiya, akashem, bleanhar, esimard, ffranz, mgahagan, mjudeiki, wking, xxia, yunjiang | ||||
| Version: | 4.5 | Keywords: | Upgrades | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.6.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
Cause: Healthcheck probes not defined on AWS UPI deployed load balancers
Consequence: Load balancers do not properly detect when an API endpoint is no longer available and may still direct traffic causing client failures.
Fix: Added Healthcheck probes to AWS UPI deployed load balancers
Result: Load balancers properly detect when API endpoints are not available and do not route traffic to offline nodes.
|
Story Points: | --- | ||||
| Clone Of: | 1828382 | Environment: | |||||
| Last Closed: | 2020-10-27 15:59:49 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1873280, 1876581 | ||||||
| Attachments: |
|
||||||
|
Description
W. Trevor King
2020-05-14 23:46:53 UTC
Won't be able to get to it this sprint. Created attachment 1696713 [details]
install log
> https://github.com/openshift/installer/pull/3709/files#diff-c60d83f04b0b1233e1739f06c356299fR221 I think that one should have been healthz https://github.com/openshift/installer/blob/12320ec6536b5145ef49c825205e4d487e0f3c4d/data/data/aws/vpc/master-elb.tf#L114-L120 verified on 4.6.0-0.nightly-2020-06-16-214732. FAILED. Same problem as described in comment #5 https://bugzilla.redhat.com/show_bug.cgi?id=1836018#c5 can you provide must-gather and info on the AWS target groups like the backends, health probe info etc. After several attempts to resolve this issue, a adequate fix has not yet been identified. Flagging with UpcomingSprint for review. Hello Abhinav,
following is the target group info, and log-bundle is attached.
vpc-id is vpc-00bb6a475a9898cd2, let me know if you need further information.
1. checking health check settings:
aws elbv2 describe-target-groups --names yunji-Inter-F33KVHQZNMLM
{
"TargetGroups": [
{
"TargetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:301721915996:targetgroup/yunji-Inter-F33KVHQZNMLM/0a5d717106179edd",
"TargetGroupName": "yunji-Inter-F33KVHQZNMLM",
"Protocol": "TCP",
"Port": 22623,
"VpcId": "vpc-00bb6a475a9898cd2",
"HealthCheckProtocol": "HTTPS",
"HealthCheckPort": "22623",
"HealthCheckEnabled": true,
"HealthCheckIntervalSeconds": 10,
"HealthCheckTimeoutSeconds": 10,
"HealthyThresholdCount": 2,
"UnhealthyThresholdCount": 2,
"HealthCheckPath": "/readyz",
"Matcher": {
"HttpCode": "200-399"
},
"LoadBalancerArns": [
"arn:aws:elasticloadbalancing:us-east-2:301721915996:loadbalancer/net/yunjiang-22bz0185-c4wkt-int/bd15404005f6e841"
],
"TargetType": "ip"
}
]
}
2. checking healthy status:
aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:us-east-2:301721915996:targetgroup/yunji-Inter-F33KVHQZNMLM/0a5d717106179edd | jq .TargetHealthDescriptions[].TargetHealth.State
"unhealthy"
"unhealthy"
"unhealthy"
> https://github.com/openshift/installer/pull/3757/files#diff-c60d83f04b0b1233e1739f06c356299fR221 We changed the health endpoint, but it looks like you are still using the old CF template. > > aws elbv2 describe-target-groups --names yunji-Inter-F33KVHQZNMLM > > { > "TargetGroups": [ > { > "TargetGroupArn": > "arn:aws:elasticloadbalancing:us-east-2:301721915996:targetgroup/yunji-Inter- > F33KVHQZNMLM/0a5d717106179edd", > "TargetGroupName": "yunji-Inter-F33KVHQZNMLM", > "Protocol": "TCP", > "Port": 22623, > "VpcId": "vpc-00bb6a475a9898cd2", > "HealthCheckProtocol": "HTTPS", > "HealthCheckPort": "22623", > "HealthCheckEnabled": true, > "HealthCheckIntervalSeconds": 10, > "HealthCheckTimeoutSeconds": 10, > "HealthyThresholdCount": 2, > "UnhealthyThresholdCount": 2, > "HealthCheckPath": "/readyz", > "Matcher": { > "HttpCode": "200-399" > }, > "LoadBalancerArns": [ > > "arn:aws:elasticloadbalancing:us-east-2:301721915996:loadbalancer/net/ > yunjiang-22bz0185-c4wkt-int/bd15404005f6e841" > ], > "TargetType": "ip" > } > ] > } > Please use the latest CF template and try again. verified on registry.svc.ci.openshift.org/ocp/release:4.6.0-0.ci-2020-06-28-020831. PASS. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |