Hide Forgot
Description of problem: Checked some fresh install env on AWS (IPI and UPI) and found that the router pods are always running on same node, although no functional impact. And after scale down and scale up, then the two router pods running on different nodes. Version-Release number of selected component (if applicable): 4.1.0-0.nightly-2019-04-28-064010 How reproducible: 100% Steps to Reproduce: 1. fresh install on AWS 2. check the router pods $ oc get pod -o wide -n openshift-ingress 3. scale down $ oc -n openshift-ingress-operator patch ingresscontroller/default -p '{"spec":{"replicas": 0}}' --type=merge 4. scale up $ oc -n openshift-ingress-operator patch ingresscontroller/default -p '{"spec":{"replicas": 2}}' --type=merge 5. check the router pods again $ oc get pod -o wide -n openshift-ingress Actual results: step 2: $ oc get pod -o wide -n openshift-ingress NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES router-default-84c7f9d456-fgxqp 1/1 Running 0 4h35m 10.131.0.4 ip-172-31-134-171.ap-northeast-2.compute.internal <none> <none> router-default-84c7f9d456-sl7k2 1/1 Running 0 4h35m 10.131.0.3 ip-172-31-134-171.ap-northeast-2.compute.internal <none> <none> step 5: $ oc get pod -o wide -n openshift-ingress NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES router-default-84c7f9d456-2wvzv 1/1 Running 0 50s 10.128.2.9 ip-172-31-141-240.ap-northeast-2.compute.internal <none> <none> router-default-84c7f9d456-d9ndb 1/1 Running 0 50s 10.129.2.9 ip-172-31-150-49.ap-northeast-2.compute.internal <none> <none> Expected results: the two router pods are running on different nodes Additional info:
Because we don't redistribute pods unless they are evicted, maybe this is just: 1. Ingress requests pods, but we have no compute nodes yet. 2. Machine-API operator creates the first compute node. 3. Scheduler rejoices and drops both router pods on that node. 4. Machine API creates additional nodes, but since the router pods are already scheduled, it's too late for the scheduler to point an ingress pod at them. This seems like something that should have a generic Kubernetes rebalancing solution. I don't know if one exists or not, but if not, a short-term fix might be having your operator monitor for this and then kill one of the pods if it notices this condition.
verified with 4.1.0-0.nightly-2019-05-04-210601 on AWS and issue has been fixed. $ oc get pod -o wide -n openshift-ingress NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES router-default-75956b9c8d-6bmf2 1/1 Running 0 4h56m 10.128.2.3 ip-172-31-159-55.ap-southeast-1.compute.internal <none> <none> router-default-75956b9c8d-sg4z7 1/1 Running 0 4h56m 10.131.0.4 ip-172-31-169-47.ap-southeast-1.compute.internal <none> <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758