Bug 1855003

Summary: cri-o image GC stops on worker nodes when orphaned layers are found
Product: OpenShift Container Platform Reporter: Dan Yocum <dyocum>
Component: NodeAssignee: Peter Hunt <pehunt>
Node sub component: CRI-O QA Contact: Sunil Choudhary <schoudha>
Status: CLOSED WORKSFORME Docs Contact:
Severity: high    
Priority: medium CC: aos-bugs, dornelas, dwalsh, eparis, harpatil, jhonce, jhou, mpatel, mwasher, nagrawal, nalin, nmalik, qiwan, rphillips, tsweeney, umohnani, vlaad, wking, xtian
Version: 4.5Keywords: Reopened
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-22 15:31:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Yocum 2020-07-08 15:39:29 UTC
Description of problem:

When orphaned or "broken" (i.e., partial layer files) are found on a worker node container directory, cri-o garbage collection halts, allowing the FS to fill up and crashes the node.


Version-Release number of selected component (if applicable):

4.4.4

How reproducible:

always

Steps to Reproduce:
1. start to deploy a pod and "yank the power cord" to create a partial layer file on a worker node
2. When rebooted, GC halts when it encounters a partial/broken/orphaned image layer
3.

Actual results:

Fail

Expected results:

Succeed

Additional info:

Comment 9 Harshal Patil 2020-07-17 16:17:39 UTC

*** This bug has been marked as a duplicate of bug 1857224 ***

Comment 20 Tom Sweeney 2020-10-27 20:14:25 UTC
Nalin, any further comments on progress on this one?

Comment 29 Qi Wang 2021-03-18 19:05:21 UTC
Did not get completed this sprint.

Comment 30 Qi Wang 2021-04-09 19:51:20 UTC
Did not get completed this sprint.

Comment 31 Qi Wang 2021-06-07 18:39:48 UTC
container/storage PR https://github.com/containers/storage/pull/898 and https://github.com/containers/storage/pull/926 might fix this bug. Is there a way we can reproduce and test it does work?

Comment 32 Michael Washer 2021-06-20 22:58:27 UTC
The only way that I am I am aware has been used to re-create the issue is by performing the following, during an image pull.
```
echo b >/proc/sysrq-trigger
```