Bug 1755584 - Memory leak CRIO due to no garbage collection in /run/crio/exits for exited containers [4.1.z]
Summary: Memory leak CRIO due to no garbage collection in /run/crio/exits for exited ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.1.z
Assignee: Mrunal Patel
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On: 1755044
Blocks: 1186913
TreeView+ depends on / blocked
 
Reported: 2019-09-25 18:33 UTC by Derrick Ornelas
Modified: 2019-11-07 06:33 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1755044
Environment:
Last Closed: 2019-11-07 06:32:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github cri-o cri-o pull 2855 0 'None' closed [1.13] sandbox_remove: Ensure that the pod infra container exit file is removed 2020-05-06 12:48:43 UTC
Red Hat Product Errata RHBA-2019:3294 0 None None None 2019-11-07 06:33:05 UTC

Comment 3 Jessica Forrester 2019-10-22 14:43:37 UTC
I don't think this got picked up correctly by the BZ automation. If all fixes related to this are merged, please move the BZ to MODIFIED so the errata sweep will pick it up.

Comment 4 Derrick Ornelas 2019-10-22 16:13:42 UTC
Checking 4.1.z

# yum list cri-o --enablerepo=rhel-7-server-ose-4.1-rpms
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Available Packages
cri-o.x86_64                                        1.13.11-0.10.dev.rhaos4.1.gitbdeb2ca.el7                                        rhel-7-server-ose-4.1-rpms  



$ git clone -q https://github.com/cri-o/cri-o.git && cd cri-o/

$ git checkout -b 1.13.11-0.10.dev.rhaos4.1.gitbdeb2ca.el7 bdeb2ca
Switched to a new branch '1.13.11-0.10.dev.rhaos4.1.gitbdeb2ca.el7'

$ git branch --contains c1333348b80cae27afaa13a5ee0ced89970ef109
* 1.13.11-0.10.dev.rhaos4.1.gitbdeb2ca.el7

$ git branch --contains 6e1cdec42aa9abca5f8b85a280ca9a0fff9c4ad8
* 1.13.11-0.10.dev.rhaos4.1.gitbdeb2ca.el7


It looks like the fixes from https://github.com/cri-o/cri-o/pull/2855 are in there

Comment 6 weiwei jiang 2019-10-29 08:36:14 UTC
Checked with cri-o-1.13.11-0.14.dev.rhaos4.1.git3338d4d.el8.x86_64. Verified.


[root@ip-10-0-154-235 /]# rpm-ostree status                                                                                                                                                                                                   
State: idle                                                                                                                                                                                                                                   
AutomaticUpdates: disabled                                                                                                                                                                                                                    
Deployments:                                                                                                                                                                                                                                  
● pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1a9fb9cbf18134339011c130c571130c7a68040711d7e738710d5f3979f9d76e                                                                                                              
              CustomOrigin: Managed by pivot tool                                                                                                                                                                                             
                   Version: 410.8.20191025.0 (2019-10-25T14:09:05Z)     

➜  ~ oc get nodes -o name | xargs -n 1 -I {} oc debug {} -- chroot /host bash -c 'ls /run/crio/exits | wc -l' 2>/dev/null
6
34
37
5
62
➜  ~ oc get nodes -o wide 
NAME                                         STATUS   ROLES    AGE    VERSION             INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                   KERNEL-VERSION                CONTAINER-RUNTIME
ip-10-0-139-94.us-east-2.compute.internal    Ready    worker   153m   v1.13.4+493dbf621   10.0.139.94    <none>        Red Hat Enterprise Linux CoreOS 410.8.20191025.0 (Ootpa)   4.18.0-80.11.2.el8_0.x86_64   cri-o://1.13.11-0.14.dev.rhaos
4.1.git3338d4d.el8-dev
ip-10-0-141-60.us-east-2.compute.internal    Ready    master   158m   v1.13.4+493dbf621   10.0.141.60    <none>        Red Hat Enterprise Linux CoreOS 410.8.20191025.0 (Ootpa)   4.18.0-80.11.2.el8_0.x86_64   cri-o://1.13.11-0.14.dev.rhaos
4.1.git3338d4d.el8-dev
ip-10-0-151-11.us-east-2.compute.internal    Ready    master   159m   v1.13.4+493dbf621   10.0.151.11    <none>        Red Hat Enterprise Linux CoreOS 410.8.20191025.0 (Ootpa)   4.18.0-80.11.2.el8_0.x86_64   cri-o://1.13.11-0.14.dev.rhaos
4.1.git3338d4d.el8-dev
ip-10-0-154-235.us-east-2.compute.internal   Ready    worker   153m   v1.13.4+493dbf621   10.0.154.235   <none>        Red Hat Enterprise Linux CoreOS 410.8.20191025.0 (Ootpa)   4.18.0-80.11.2.el8_0.x86_64   cri-o://1.13.11-0.14.dev.rhaos
4.1.git3338d4d.el8-dev
ip-10-0-174-97.us-east-2.compute.internal    Ready    master   159m   v1.13.4+493dbf621   10.0.174.97    <none>        Red Hat Enterprise Linux CoreOS 410.8.20191025.0 (Ootpa)   4.18.0-80.11.2.el8_0.x86_64   cri-o://1.13.11-0.14.dev.rhaos
4.1.git3338d4d.el8-dev
➜  ~ for i in {1..10}; do echo ==$i; oc scale dc/h --replicas=10; sleep 10; oc scale dc/h --replicas=1; sleep 10; done 
==1
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==2
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==3
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==4
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==5
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==6
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==7
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==8
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==9
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
==10
deploymentconfig.apps.openshift.io/h scaled
deploymentconfig.apps.openshift.io/h scaled
➜  ~ oc get nodes -o name | xargs -n 1 -I {} oc debug {} -- chroot /host bash -c 'ls /run/crio/exits | wc -l' 2>/dev/null
4
32
35
3
60

Comment 8 errata-xmlrpc 2019-11-07 06:32:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3294


Note You need to log in before you can comment on or make changes to this bug.