Bug 1462397
| Summary: | EnsureLoadBalancer is spammy in large clusters | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Eric Paris <eparis> |
| Component: | Networking | Assignee: | Ivan Chavero <ichavero> |
| Status: | CLOSED ERRATA | QA Contact: | Meng Bo <bmeng> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.5.1 | CC: | aos-bugs, bbennett, eparis, trankin, xtian, zzhao |
| Target Milestone: | --- | ||
| Target Release: | 3.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
no doc update
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-11-28 21:58:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Eric Paris
2017-06-17 04:05:26 UTC
(In reply to Eric Paris from comment #0) > oc v3.5.5.19 > > /usr/bin/openshift start master controllers > --config=/etc/origin/master/master-config.yaml --loglevel=2 > --listen=https://0.0.0.0:8444 > > Every so often I see stuff like this in the logs. Do I really need to print > out every single node in the cluster? Make logs great again. > > Jun 16 22:09:34 ip-172-31-54-162.ec2.internal > atomic-openshift-master-controllers[48569]: ternal > ip-172-31-55-2.ec2.internal ip-172-31-62-64.ec2.internal > ip-172-31-55-86.ec2.internal ip-172-31-49-42.ec2.internal > ip-172-31-52-92.ec2.internal ip-172-31-56-76.ec2.internal > ip-172-31-48-8.ec2.internal ip-172-31-50-116.ec2.internal > ip-172-31-59-231.ec2.internal ip-172-31-54-77.ec2.internal > ip-172-31-58-211.ec2.internal ip-172-31-56-184.ec2.internal > ip-172-31-61-32.ec2.internal ip-172-31-55-219.ec2.internal > ip-172-31-51-63.ec2.internal ip-172-31-62-236.ec2.internal > ip-172-31-60-113.ec2.internal ip-172-31-53-190.ec2.internal > ip-172-31-58-188.ec2.internal ip-172-31-49-230.ec2.internal > ip-172-31-49-20.ec2.internal ip-172-31-59-113.ec2.internal > ip-172-31-54-226.ec2.internal ip-172-31-55-14.ec2.internal > ip-172-31-52-166.ec2.internal ip-172-31-63-19.ec2.internal > ip-172-31-55-15.ec2.internal ip-172-31-62-85.ec2.internal > ip-172-31-63-1.ec2.internal ip-172-31-54-67.ec2.internal > ip-172-31-49-165.ec2.internal ip-172-31-52-230.ec2.internal > ip-172-31-49-220.ec2.internal ip-172-31-52-181.ec2.internal], map[]) can you paste logs with --loglevel=5 please? Ivan, we just need to change these from V(2) to V(4)... but they are in Kubernetes, so you need to post the PR over there and get the change made there.
$ ack "EnsureLoadBalancer\(" | grep log.V\(2 | grep -v incubator
vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:588: glog.V(2).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v, %v, %v)", loadBalancerName, gce.region, loadBalancerIP, portStr, hostNames, serviceName, apiService.Annotations)
vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:633: glog.V(2).Infof("EnsureLoadBalancer(%v(%v)): released static IP %s", loadBalancerName, serviceName, ipAddress)
vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:2508: glog.V(2).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v, %v, %v, %v)",
$ ack "Ensuring LB for service" | grep -v incubator
vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:295: glog.V(2).Infof("Ensuring LB for service %s", key)
Is this information useful in a normal cluster? What should an admin do with the hundreds and hundreds of nodes being listed regularly? What makes sense to log for everyone? What makes sense to log only in some debug mode? Please fix that upstream and then backport to 3.6. If this message says nothing useful move it to V(4). If some of it is useful and some of it not, leave the useful part alone and move the rest to V(4).... (In reply to Ben Bennett from comment #2) > Ivan, we just need to change these from V(2) to V(4)... but they are in > Kubernetes, so you need to post the PR over there and get the change made > there. > > $ ack "EnsureLoadBalancer\(" | grep log.V\(2 | grep -v incubator > vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:588: > glog.V(2).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v, %v, %v)", > loadBalancerName, gce.region, loadBalancerIP, portStr, hostNames, > serviceName, apiService.Annotations) > vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:633: > glog.V(2).Infof("EnsureLoadBalancer(%v(%v)): released static IP %s", > loadBalancerName, serviceName, ipAddress) > vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:2508: > glog.V(2).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v, %v, %v, %v)", > > > $ ack "Ensuring LB for service" | grep -v incubator > vendor/k8s.io/kubernetes/pkg/controller/service/servicecontroller.go:295: > glog.V(2).Infof("Ensuring LB for service %s", key) ok, creating patch Is there an upstream PR? Can we get it and the origin backport PR linked? Upstream PR: https://github.com/kubernetes/kubernetes/pull/48145 This is being addressed on this pull requests: Upstream: https://github.com/kubernetes/kubernetes/pull/48145 Origin: https://github.com/openshift/origin/pull/14945 Verified this bug on v3.7.0-0.127.0
steps
1. Create ELB env on aws
2. Create Loadbalance service with the following file:
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "example-service"
},
"spec": {
"ports": [{
"port": 8765,
"targetPort": 9376
}],
"selector": {
"app": "example"
},
"type": "LoadBalancer"
}
}
3. Check the log on the master:
journalctl -fl | grep -i aws.go
...
...
Sep 28 06:03:24 ip-172-18-12-194.ec2.internal atomic-openshift-master-controllers[42348]: I0928 06:03:24.118359 42348 aws.go:2598] EnsureLoadBalancer(kubernetes, default, example-service, us-east-1, , [{ TCP 8765 {0 9376 } 32030}], map[])
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188 |