Bug 2026110

Summary: Altering the Schedule Profile configurations doesn't affect the placement of the pods
Product: OpenShift Container Platform Reporter: Mike Dame <mdame>
Component: kube-schedulerAssignee: Jan Chaloupka <jchaloup>
Status: CLOSED ERRATA QA Contact: RamaKasturi <knarra>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.8CC: aos-bugs, jchaloup, knarra, mfojtik, yhe
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2026109
: 2026111 (view as bug list) Environment:
Last Closed: 2022-04-11 20:04:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2026109    
Bug Blocks: 2026111    

Comment 1 Michal Fojtik 2021-12-25 05:22:35 UTC 
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Whiteboard if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.
Comment 3 Jan Chaloupka 2022-03-11 14:23:10 UTC 
Waiting for the 4.9 bug verification.
Comment 6 RamaKasturi 2022-03-18 11:17:55 UTC 
Verified with the build below and i see that pods gets placed on to a high utilized node.

[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2022-03-18-025702   True        False         60m     Cluster version is 4.8.0-0.nightly-2022-03-18-025702

Below are the steps i have followed to verify the bug and tested this on a 5 & 7 node worker cluster so that i can see the diff clearly before and after altering the profile.

5 node worker cluster:
===========================
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc new-app httpd --name httpd
--> Found image 28666a5 (8 days old) in image stream "openshift/httpd" under tag "2.4-el8" for "httpd"

    Apache httpd 2.4 
    ---------------- 
    Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites.

    Tags: builder, httpd, httpd-24


--> Creating resources ...
    deployment.apps "httpd" created
    service "httpd" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/httpd' 
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc new-app httpd --name httpd1
--> Found image 28666a5 (8 days old) in image stream "openshift/httpd" under tag "2.4-el8" for "httpd"

    Apache httpd 2.4 
    ---------------- 
    Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites.

    Tags: builder, httpd, httpd-24


--> Creating resources ...
    deployment.apps "httpd1" created
    service "httpd1" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/httpd1' 
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc new-app httpd --name httpd2
--> Found image 28666a5 (8 days old) in image stream "openshift/httpd" under tag "2.4-el8" for "httpd"

    Apache httpd 2.4 
    ---------------- 
    Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites.

    Tags: builder, httpd, httpd-24


--> Creating resources ...
    deployment.apps "httpd2" created
    service "httpd2" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/httpd2' 
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc scale --replicas=100 deployment httpd
deployment.apps/httpd scaled
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc scale --replicas=100 deployment httpd1
deployment.apps/httpd1 scaled
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc scale --replicas=100 deployment httpd2
deployment.apps/httpd2 scaled

[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep "Running" |wc -l
300

[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-128-226.us-east-2.compute.internal | wc -l
57
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-136-107.us-east-2.compute.internal | wc -l
58
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-172-151.us-east-2.compute.internal | wc -l
58
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-190-131.us-east-2.compute.internal | wc -l
57
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-211-253.us-east-2.compute.internal | wc -l
70

config before altering the profile:
=====================================
 score:
      enabled:
      - name: NodeResourcesBalancedAllocation
        weight: 1
      - name: ImageLocality
        weight: 1
      - name: InterPodAffinity
        weight: 1
      - name: NodeResourcesLeastAllocated
        weight: 1
      - name: NodeAffinity
        weight: 1
      - name: NodePreferAvoidPods
        weight: 10000
      - name: PodTopologySpread
        weight: 2
      - name: TaintToleration
        weight: 1

config file after the profile has been altered:
==================================================
score:
      enabled:
      - name: ImageLocality
        weight: 1
      - name: InterPodAffinity
        weight: 1
      - name: NodeAffinity
        weight: 1
      - name: NodePreferAvoidPods
        weight: 10000
      - name: PodTopologySpread
        weight: 2
      - name: TaintToleration
        weight: 1
      - name: NodeResourcesMostAllocated
        weight: 5
  schedulerName: default-scheduler

[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc rollout restart deployment httpd httpd1 httpd2
deployment.apps/httpd restarted
deployment.apps/httpd1 restarted
deployment.apps/httpd2 restarted


pod placement after first rollout:
===========================================
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-128-226.us-east-2.compute.internal | wc -l
11
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-136-107.us-east-2.compute.internal | wc -l
103
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-172-151.us-east-2.compute.internal | wc -l
76
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-190-131.us-east-2.compute.internal | wc -l
21
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-211-253.us-east-2.compute.internal | wc -l
89
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$

pod placement after second rollout:
=========================================
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-128-226.us-east-2.compute.internal | wc -l
0
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-136-107.us-east-2.compute.internal | wc -l
93
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-172-151.us-east-2.compute.internal | wc -l
103
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-190-131.us-east-2.compute.internal | wc -l
0
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-211-253.us-east-2.compute.internal | wc -l
104
 

cluster with 7worker nodes:
============================
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc new-app httpd --name httpd
--> Found image 28666a5 (8 days old) in image stream "openshift/httpd" under tag "2.4-el8" for "httpd"

    Apache httpd 2.4 
    ---------------- 
    Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites.

    Tags: builder, httpd, httpd-24


--> Creating resources ...
    deployment.apps "httpd" created
    service "httpd" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/httpd' 
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc new-app httpd --name httpd1
--> Found image 28666a5 (8 days old) in image stream "openshift/httpd" under tag "2.4-el8" for "httpd"

    Apache httpd 2.4 
    ---------------- 
    Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites.

    Tags: builder, httpd, httpd-24


--> Creating resources ...
    deployment.apps "httpd1" created
    service "httpd1" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/httpd1' 
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc new-app httpd --name httpd2
--> Found image 28666a5 (8 days old) in image stream "openshift/httpd" under tag "2.4-el8" for "httpd"

    Apache httpd 2.4 
    ---------------- 
    Apache httpd 2.4 available as container, is a powerful, efficient, and extensible web server. Apache supports a variety of features, many implemented as compiled modules which extend the core functionality. These can range from server-side programming language support to authentication schemes. Virtual hosting allows one Apache installation to serve many different Web sites.

    Tags: builder, httpd, httpd-24


--> Creating resources ...
    deployment.apps "httpd2" created
    service "httpd2" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose service/httpd2' 
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc scale --replicas=100 deployment httpd
deployment.apps/httpd scaled
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc scale --replicas=100 deployment httpd1
deployment.apps/httpd1 scaled
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc scale --replicas=100 deployment httpd2
deployment.apps/httpd2 scaled

[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep "Running" | wc -l
300
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ 

[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-136-244.us-east-2.compute.internal | wc -l
40
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-140-202.us-east-2.compute.internal | wc -l
38
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-159-121.us-east-2.compute.internal | wc -l
36
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-165-196.us-east-2.compute.internal | wc -l
46
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-181-15.us-east-2.compute.internal | wc -l
46
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-199-63.us-east-2.compute.internal | wc -l
48
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-219-222.us-east-2.compute.internal | wc -l
46
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ 

config file before altering the profile:
=======================================
    score:
      enabled:
      - name: NodeResourcesBalancedAllocation
        weight: 1
      - name: ImageLocality
        weight: 1
      - name: InterPodAffinity
        weight: 1
      - name: NodeResourcesLeastAllocated
        weight: 1
      - name: NodeAffinity
        weight: 1
      - name: NodePreferAvoidPods
        weight: 10000
      - name: PodTopologySpread
        weight: 2
      - name: TaintToleration
        weight: 1
  schedulerName: default-scheduler

[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc rollout restart deployment httpd httpd1 httpd2
deployment.apps/httpd restarted
deployment.apps/httpd1 restarted
deployment.apps/httpd2 restarted

pod placement after first rollout:
=======================================
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-136-244.us-east-2.compute.internal | wc -l
115
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-140-202.us-east-2.compute.internal | wc -l
1
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-159-121.us-east-2.compute.internal | wc -l
3
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-165-196.us-east-2.compute.internal | wc -l
73
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-181-15.us-east-2.compute.internal | wc -l
6
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-199-63.us-east-2.compute.internal | wc -l
93
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-219-222.us-east-2.compute.internal | wc -l
9

pod placement after second rollout:
==========================================
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-136-244.us-east-2.compute.internal | wc -l
94
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-140-202.us-east-2.compute.internal | wc -l
0
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-159-121.us-east-2.compute.internal | wc -l
0
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-165-196.us-east-2.compute.internal | wc -l
104
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-181-15.us-east-2.compute.internal | wc -l
0
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-199-63.us-east-2.compute.internal | wc -l
102
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ ./oc get pods -o wide | grep ip-10-0-219-222.us-east-2.compute.internal | wc -l
0
[knarra@knarra openshift-client-linux-4.8.0-0.nightly-2022-03-18-025702]$ 

One observation that was made was after the second rollout the placement of the pods were much better, however after setting the profile to highNodeUtilization i do see the pod placements have changed, so based on that moving the bug to verified state.
Comment 9 errata-xmlrpc 2022-04-11 20:04:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.8.36 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1154