Bug 2032006

Summary: openshift-gitops-application-controller-0 failed to schedule with sufficient node allocatable resource
Product: OpenShift Container Platform Reporter: yliu1
Component: Telco EdgeAssignee: melserng
Telco Edge sub component: ZTP QA Contact: yliu1
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified CC: mcornea
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-12 04:39:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yliu1 2021-12-13 20:46:39 UTC
Description of problem:

openshift-gitops-application-controller-0 stuck at pending with insufficient memory warning. However, the allocatable memory for hub cluster node indicates more than requested memories. 


Version-Release number of selected component (if applicable):
hub version: 4.9.8
ACM version: 2.4.1
gitops version: 1.3.1
cnf-feature-deploy: master (4.10)


How reproducible: 100%


Steps to Reproduce:
1. trigger sno install using ztp image built from latest cnf-feature-deploy contents
2. check gitops apps and pods

Actual results:
2. openshift-gitops-application-controller-0 stuck at pending

// pod resource requests:
    Limits:
      cpu:     16
      memory:  16Gi
    Requests:
      cpu:        8
      memory:     8Gi
// Pod events
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  9m59s  default-scheduler  0/3 nodes are available: 1 Insufficient cpu, 3 Insufficient memory.
  Warning  FailedScheduling  8m57s  default-scheduler  0/3 nodes are available: 1 Insufficient cpu, 3 Insufficient memory.

// node allocatable resources:
  allocatable:
    cpu: 15500m
    ephemeral-storage: "76754504782"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 19364776Ki
    pods: "250"
  capacity:
    cpu: "16"
    ephemeral-storage: 81332Mi
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 20515752Ki
    pods: "250"


Expected results:
openshift-gitops-application-controller-0 should be successfully scheduled.

Additional info:
This issue started to show up after resource requests and limits were increased for this pod. 
It used request 1Gi memory with 2Gi limits, which worked fine.

Comment 2 yliu1 2022-02-01 17:57:48 UTC
Verified on ocp 4.10.0-fc.2 with ztp container image built from master branch.

Comment 5 errata-xmlrpc 2022-03-12 04:39:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056