Bug 1807471

Summary: nodeca daemon isn't installed if registry is Removed
Product: OpenShift Container Platform Reporter: Oleg Bulatov <obulatov>
Component: ImageStreamsAssignee: Oleg Bulatov <obulatov>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.1.zCC: adam.kaplan, aos-bugs, dkulkarn, eminguez, jokerman, mzali, pweil, ssadhale, wzheng, ykashtan, yprokule
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: nodeca daemon was created only when the registry is managed Consequence: when the registry is removed, nodeca daemon is not created Fix: always create nodeca daemon Result: nodeca daemon is created even if the registry is removed
Story Points: ---
Clone Of:
: 1809176 (view as bug list) Environment:
Last Closed: 2020-07-13 17:21:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1809176    

Description Oleg Bulatov 2020-02-26 12:51:22 UTC
If managementState for the image-registry is Removed, the nodeca daemon doesn't get installed.

As a result, additionalTrustedCA is not applied to the cluster.

Comment 2 Adam Kaplan 2020-02-26 15:56:24 UTC
Work-around - set the registry to `Managed` with `emptyDir` storage to enable the nodeca daemon.

Comment 3 Eduardo Minguez 2020-02-26 15:59:44 UTC
(In reply to Adam Kaplan from comment #2)
> Work-around - set the registry to `Managed` with `emptyDir` storage to
> enable the nodeca daemon.

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'

That worked for me, thanks.

Comment 10 XiuJuan Wang 2020-03-13 10:42:13 UTC
When set image registry to Removed, the node-ca pod can't be running due to "image-registry-certificates" not found
$ oc get co  image-registry 
NAME             VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
image-registry   4.5.0-0.nightly-2020-03-12-233449   True        False         False      53m

$ oc get config.image -o yaml 
apiVersion: v1
items:
- apiVersion: imageregistry.operator.openshift.io/v1
  kind: Config
  metadata:
    creationTimestamp: "2020-03-13T09:46:49Z"
    finalizers:
    - imageregistry.operator.openshift.io/finalizer
    generation: 2
    name: cluster
    resourceVersion: "43408"
    selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
    uid: 2f128c22-5e50-4b92-b438-67368250f2e9
  spec:
    defaultRoute: true
    httpSecret: fc4f238f53cb562dd89a19290f7cbc61f66e4f9c77a6d45e796a297ff8de33cd34f6c8b0f5364209a24e829d3e7c362a0b997a71e4bce6d09ea0f3355550ee17
    logging: 2
    managementState: Removed
    proxy: {}
    replicas: 1
    requests:
      read:
        maxWaitInQueue: 0s
      write:
        maxWaitInQueue: 0s
    rolloutStrategy: RollingUpdate
    storage: {}
  status:
    conditions:
    - lastTransitionTime: "2020-03-13T10:38:59Z"
      message: All registry resources are removed
      reason: Removed
      status: "False"
      type: Progressing
    - lastTransitionTime: "2020-03-13T09:46:50Z"
      message: The registry is removed
      reason: Removed
      status: "True"
      type: Available
    - lastTransitionTime: "2020-03-13T09:46:50Z"
      status: "False"
      type: Degraded
    - lastTransitionTime: "2020-03-13T09:46:50Z"
      message: The registry is removed
      reason: Removed
      status: "True"
      type: Removed
    observedGeneration: 2
    readyReplicas: 0
    storage: {}
    storageManaged: false
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""



$oc get pods
NAME                                               READY   STATUS              RESTARTS   AGE
cluster-image-registry-operator-6cc6df7544-5t7hj   2/2     Running             0          50m
node-ca-8jgmg                                      0/1     ContainerCreating   0          50m
node-ca-ckk6b                                      0/1     ContainerCreating   0          46m
node-ca-cv8xg                                      0/1     ContainerCreating   0          50m
node-ca-fdzw6                                      0/1     ContainerCreating   0          50m
node-ca-nqvtb                                      0/1     ContainerCreating   0          45m

$ oc describe pods node-ca-8jgmg 
=============================
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     
Events:
  Type     Reason       Age                   From                                    Message
  ----     ------       ----                  ----                                    -------
  Normal   Scheduled    52m                   default-scheduler                       Successfully assigned openshift-image-registry/node-ca-8jgmg to wxj-bm3-b49lw-control-plane-2
  Warning  FailedMount  33m (x2 over 39m)     kubelet, wxj-bm3-b49lw-control-plane-2  Unable to attach or mount volumes: unmounted volumes=[serviceca], unattached volumes=[host node-ca-token-s5bqd serviceca]: timed out waiting for the condition
  Warning  FailedMount  31m (x2 over 45m)     kubelet, wxj-bm3-b49lw-control-plane-2  Unable to attach or mount volumes: unmounted volumes=[serviceca], unattached volumes=[node-ca-token-s5bqd serviceca host]: timed out waiting for the condition
  Warning  FailedMount  6m47s (x14 over 50m)  kubelet, wxj-bm3-b49lw-control-plane-2  Unable to attach or mount volumes: unmounted volumes=[serviceca], unattached volumes=[serviceca host node-ca-token-s5bqd]: timed out waiting for the condition
  Warning  FailedMount  102s (x33 over 52m)   kubelet, wxj-bm3-b49lw-control-plane-2  MountVolume.SetUp failed for volume "serviceca" : configmap "image-registry-certificates" not found

$oc get cm  -n openshift-image-registry 
NAME         DATA   AGE
trusted-ca   1      82m

Comment 11 Oleg Bulatov 2020-03-27 14:21:33 UTC
*** Bug 1817975 has been marked as a duplicate of this bug. ***

Comment 13 XiuJuan Wang 2020-04-10 08:38:39 UTC
Test in 4.5.0-0.nightly-2020-04-09-231931 cluster, and pass.
When set image registry to Removed, the node-ca daemon is installed, and pods could be running.

$oc get pods
NAME                                               READY   STATUS    RESTARTS   AGE
cluster-image-registry-operator-845886ff5f-8vbtr   2/2     Running   0          139m
node-ca-5n8xf                                      1/1     Running   0          139m
node-ca-9dgrh                                      1/1     Running   0          139m
node-ca-jx7x8                                      1/1     Running   0          139m
node-ca-qfxrz                                      1/1     Running   1          138m

$ oc get config.image -o yaml  
apiVersion: v1
items:
- apiVersion: imageregistry.operator.openshift.io/v1
  kind: Config
  metadata:
    creationTimestamp: "2020-04-10T05:53:59Z"
    finalizers:
    - imageregistry.operator.openshift.io/finalizer
    generation: 1
    name: cluster
    resourceVersion: "74364"
    selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
    uid: d0066a10-ceba-4d9f-9435-0c9543331049
  spec:
    httpSecret: 1cf7d118baceec502f091760b99398b2c081b050f7f8158ed9d290dea80bee09da8a910246774a7fcc2105d67492c3d774fcc025e1ae12286479325f313feb86
    logging: 2
    managementState: Removed
    proxy: {}
    replicas: 1
    requests:
      read:
        maxWaitInQueue: 0s
      write:
        maxWaitInQueue: 0s
    rolloutStrategy: RollingUpdate
    storage: {}
  status:
    conditions:
    - lastTransitionTime: "2020-04-10T08:30:02Z"
      message: All registry resources are removed
      reason: Removed
      status: "False"
      type: Progressing
    - lastTransitionTime: "2020-04-10T05:53:59Z"
      message: The registry is removed
      reason: Removed
      status: "True"
      type: Available
    - lastTransitionTime: "2020-04-10T05:53:59Z"
      message: The registry is removed
      reason: Removed
      status: "False"
      type: Degraded
    - lastTransitionTime: "2020-04-10T05:53:59Z"
      message: The registry is removed
      reason: Removed
      status: "True"
      type: Removed
    observedGeneration: 1
    readyReplicas: 0
    storage: {}
    storageManaged: false
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 17 errata-xmlrpc 2020-07-13 17:21:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409