Bug 1936984

Summary: Image Registry pod enters CrashLoopBackoff State for extended periods of time after node reboot
Product: OpenShift Container Platform Reporter: Oleg Bulatov <obulatov>
Component: Image RegistryAssignee: Oleg Bulatov <obulatov>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.5CC: aos-bugs, huebert, jdelft, juzhao, lisowski, openshift-bugzilla-robot, wzheng, xxia
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: sometimes /etc/pki/ca-trust/extracted become unwritable Consequence: the operator cannot put CAs into the pod's trust store Fix: mount emptyDir volume into /etc/pki/ca-trust/extracted Result: ephemeral volume should always be writable by the pod
Story Points: ---
Clone Of: 1893956 Environment:
Last Closed: 2021-04-20 19:27:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1893956    
Bug Blocks:    

Comment 1 Oleg Bulatov 2021-03-19 13:18:46 UTC
*** Bug 1940877 has been marked as a duplicate of this bug. ***

Comment 2 Wenjing Zheng 2021-03-22 07:58:02 UTC
QE verified with open PR with below info:
Volumes:
  registry-storage-keyfile:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  image-registry-private-configuration
    Optional:    false
  registry-tls:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          image-registry-tls
    SecretOptionalName:  <nil>
  ca-trust-extracted:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>


sh-4.4$ mount | grep extracted
/dev/mapper/coreos-luks-root-nocrypt on /etc/pki/ca-trust/extracted type xfs (rw,relatime,seclabel,attr2,inode64,prjquota)
sh-4.4$ ls /etc/pki/ca-trust
README	ca-legacy.conf	extracted  source
sh-4.4$ ls /etc/pki/ca-trust/extracted/
edk2  java  openssl  pem

Comment 4 Wenjing Zheng 2021-03-25 02:05:30 UTC
Verified with 4.6.0-0.nightly-2021-03-24-213203:
$ oc adm release info registry.ci.openshift.org/ocp/release:4.6.0-0.nightly-2021-03-24-213203 --commits | grep registry-operator
  cluster-image-registry-operator                https://github.com/openshift/cluster-image-registry-operator                0924ef6a59f6b1812e2330a74be9eb908dbd3efc

The commit 9f5be0b is contained in the payload.

Comment 10 errata-xmlrpc 2021-04-20 19:27:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.25 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1153