Bug 1512826 - [3.6] Registry console pod meets CrashLoopBackOff
Summary: [3.6] Registry console pod meets CrashLoopBackOff
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Registry Console
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.6.z
Assignee: Peter
QA Contact: Yadan Pei
URL:
Whiteboard:
Depends On:
Blocks: 1512832 1513356
TreeView+ depends on / blocked
 
Reported: 2017-11-14 08:44 UTC by Xingxing Xia
Modified: 2020-02-03 14:58 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1512832 1513356 (view as bug list)
Environment:
Last Closed: 2017-12-14 21:02:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3438 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-15 01:58:11 UTC

Description Xingxing Xia 2017-11-14 08:44:18 UTC
Description of problem
Registry console pod meets CrashLoopBackOff

Version-Release number of selected component (if applicable):
Registry console v3.6, image id 920c98e806f1
(The image 3.6.173.0.63-10 attached in errata advisory 31024 has same id)

How reproducible:
Always

Steps to Reproduce:
1. Install 3.6 OCP env that would deploy registry console
2. Check the registry console pod

Actual results:
2. Check the pod, it shows CrashLoopBackOff
# oc get pod
registry-console-1-qn7hw   0/1       CrashLoopBackOff   2         30s

# oc describe pod registry-console-1-qn7hw
    Image:		brew...:8888/openshift3/registry-console:v3.6
...
  1m		26s		6	kubelet, openshift-14...redhat.com	spec.containers{registry-console}	Warning		BackOff		Back-off restarting failed container
...
  1m		9s		7	kubelet, openshift-14...redhat.com						Warning		FailedSync	Error syncing pod

# oc logs registry-console-1-qn7hw
2017/11/14 08:29:38 Could't link /container/registry-brand to /container/os-release: exit status 1: ln: failed to create symbolic link '/container/os-release': Permission denied
#

Expected results:
2. Pod should be Running

Additional info:

Comment 1 Peter 2017-11-14 19:19:07 UTC
This has been fixed in dist-git and a new container image should be available soon.

Comment 2 Xingxing Xia 2017-11-15 03:11:47 UTC
Hmm, where/when is it available? I don't know what/where is dist-git
Just found the errata advisory 31024 removed registry-console-docker-3.6.173.0.63-10, and added registry-console-docker-v3.6.173.0.63-10 (note the prefix "v"). Compared their image id's, the latter is new one. Tested with the latter, it still has the issue:
# docker images | grep registry-console
brew-...:8888/openshift3/registry-console  v3.6              93c45cdc55d1  12 hours ago  226.1 MB   
brew-...:8888/openshift3/registry-console  v3.6.173.0.63-10  93c45cdc55d1  12 hours ago  226.1 MB

# oc get pod
registry-console-4-dtd4l   0/1       CrashLoopBackOff   2         40s

# oc logs registry-console-4-dtd4l
2017/11/15 02:51:49 Could't link /container/registry-brand to /container/os-release: exit status 1: ln: failed to create symbolic link '/container/os-release': Permission denied

Comment 4 Xingxing Xia 2017-11-16 03:34:12 UTC
Checked tag v3.6 (cockpit 155) in brew registry is updated today, pod with it can be Running, registry console with it works well.
# docker images | grep registry-console
brew-...:8888/openshift3/registry-console v3.6 f45c04cd604a About an hour ago   226.9 MB
The left work is to replace the problematic "v3.6.173.0.63-10" image in advisory 31024. Sam, please help with that, thx

Comment 5 Xingxing Xia 2017-11-22 07:56:03 UTC
Because tag 'v3.6' is like an alias of the real tag that has the highest tag number, what real tag it points to, changes daily
E.g. today I checked the tags in brew registry, now 'v3.6' changes to point to another tag 'v3.6.173.0.78' (whose image id is 4cc98066ed99 instead of the one comment 4)
# docker images | grep registry-console
brew-...:8888/openshift3/registry-console v3.6.173.0.78  4cc98066ed99  28 hours ago 227.9 MB

Like comment 4, I double checked, pod and registry console with this new tag can work normally as well

Thus, Sam, fyi, when you want to update the tag in the advisory 31024, the actual tag's tag number would likey be a bit higher on the day you update, as long as it is a working one. Thx

Comment 6 Xingxing Xia 2017-11-22 08:18:43 UTC
Hmm, a short supplement, not sure what tag number format it should be for the updated image tag when it is attached to advisory. Maybe it should be still like v3.6.173.0.63-xx, that is up to your related workflow. From QE concern, as long as it includes the fix of cockpit 155, pod can running and the registry console function can work

Comment 7 Xingxing Xia 2017-11-30 07:54:00 UTC
Brew's registry-console:v3.6.173.0.63-11 works well in terms of pod and the registry web console
# docker images | grep registry-console
brew.../openshift3/registry-console      v3.6.173.0.63-11      b703612e9160        9 hours ago         227.6 MB
# oc rsh registry-console-3-ww0hk # pod running registry-console:v3.6.173.0.63-11
sh-4.2$ rpm -qa | grep cockpit      
cockpit-ws-138-10.el7_4.x86_64
cockpit-dashboard-151-1.el7.x86_64
cockpit-system-138-10.el7_4.noarch
cockpit-bridge-138-10.el7_4.x86_64
cockpit-kubernetes-155-1.el7.x86_64

Today's advisory/31024 has same tag v3.6.173.0.63-11 and image id [1]. Therefore, bug could move to VERIFIED
[1]: to see the image id in advisory, click the advisory "Builds" tab --> click registry-console-docker-v3.6.173.0.63-11 --> there is 'id': 'sha256:b703612e9160...

Comment 8 Xingxing Xia 2017-11-30 08:25:10 UTC
(In reply to Xingxing Xia from comment #7)
> [1]: to see the image id in advisory, click the advisory "Builds" tab -->
> click registry-console-docker-v3.6.173.0.63-11 --> there is 'id':
> 'sha256:b703612e9160...
Sorry, typo
Should be: click the advisory "Builds" tab
--> SEARCH item registry-console-docker-v3.6.173.0.63-11
--> under the item, click the link after "tar [Docker] (x86_64)": /mnt/redhat/.../registry-console-docker/v3.6.173.0.63/11/...x86_64.tar.gz
--> now come to the page "Information for archive docker-image ...", there is 'id': 'sha256:b703612e9160...

Comment 11 errata-xmlrpc 2017-12-14 21:02:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438


Note You need to log in before you can comment on or make changes to this bug.