Description of problem After apply MachineConfig, the nodes get restart one by one. When the single master node restarts, all aide-ds pods re-initiate instead of relevant aide-ds pod to node. Version-Release -Cluster version 4.6.0-0.nightly-2020-07-25-091217 How reproducible: always Steps: 1.install the FIO git clone git:openshift/file-integrity-operator.git oc create -f file-integrity-operator/deploy/ns.yaml oc project openshift-file-integrity for l in `ls -1 file-integrity-operator/deploy/crds/*crd.yaml`; do oc create -f $l; done oc create -f file-integrity-operator/deploy/ 2. create a FileIntegrity $ oc create -f - << EOF apiVersion: fileintegrity.openshift.io/v1alpha1 kind: FileIntegrity metadata: name: example-fileintegrity6 namespace: openshift-file-integrity spec: config: name: myconf namespace: openshift-file-integrity key: aide-conf gracePeriod: 20 debug: true nodeSelector: node-role.kubernetes.io/worker: "" tolerations: - key: "key1" value: "value1" operator: "Equal" effect: "NoSchedule" EOF 3. when aid-pods are running, create a machineconfig: oc create -f - <<EOF > apiVersion: machineconfiguration.openshift.io/v1 > kind: MachineConfig > metadata: > creationTimestamp: "2020-06-22T11:02:48Z" > generation: 1 > labels: > machineconfiguration.openshift.io/role: master > name: 50-testfileintegrity1 > spec: > config: > ignition: > config: {} > security: > tls: {} > timeouts: {} > version: 2.2.0 > networkd: {} > passwd: {} > storage: > files: > - contents: > source: data:,file-integrity-operator-was-here > verification: {} > filesystem: root > mode: 420 > path: /etc/fi-test-file > systemd: {} > fips: false > kernelArguments: null > kernelType: "" > osImageURL: "" > EOF machineconfig.machineconfiguration.openshift.io/50-testfileintegrity1 created Actual result: After apply MachineConfig, the nodes get restart one by one. When the single master node restarts, all aide-ds pods re-initiate instead of relevant aide-ds pod to node. $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-1e225db3d4bedf0f67e9ad6751c3246a False True False 3 1 1 0 8h worker rendered-worker-c7725b2664b8c0d0ffda4ff9957a7416 True False False 3 3 3 0 8h $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-137-158.us-east-2.compute.internal Ready master 8h v1.18.3+a34fde4 ip-10-0-137-196.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-175-183.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-178-77.us-east-2.compute.internal Ready master 8h v1.18.3+a34fde4 ip-10-0-196-136.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-198-162.us-east-2.compute.internal Ready,SchedulingDisabled master 8h v1.18.3+a34fde4 $ oc get pod NAME READY STATUS RESTARTS AGE aide-ds-example-fileintegrity-6vjss 2/2 Terminating 0 4m24s aide-ds-example-fileintegrity-7j68d 2/2 Terminating 0 4m20s aide-ds-example-fileintegrity-g6xgb 2/2 Terminating 0 4m43s aide-ds-example-fileintegrity-mm682 2/2 Terminating 0 3m30s aide-ds-example-fileintegrity-qwqr4 2/2 Terminating 0 4m49s aide-ds-example-fileintegrity-rs87g 0/2 Terminating 0 4m41s file-integrity-operator-7467f6fd47-plrfx 1/1 Running 0 7h50m ip-10-0-137-158.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h49m ip-10-0-137-196.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 45m ip-10-0-175-183.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h49m ip-10-0-178-77.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 23s ip-10-0-196-136.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h49m ip-10-0-198-162.us-east-2.compute.internal-addholdoff 0/1 Completed 0 17s $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-1e225db3d4bedf0f67e9ad6751c3246a False True False 3 1 1 0 8h worker rendered-worker-c7725b2664b8c0d0ffda4ff9957a7416 True False False 3 3 3 0 8h $ oc get pod NAME READY STATUS RESTARTS AGE aide-ds-example-fileintegrity-5kps2 0/2 Init:0/1 0 6s aide-ds-example-fileintegrity-bbppr 0/2 Init:0/1 0 41s aide-ds-example-fileintegrity-flm97 2/2 Running 0 41s aide-ds-example-fileintegrity-k5gtx 2/2 Running 0 38s aide-ds-example-fileintegrity-s2p2l 0/2 Init:0/1 0 35s aide-ds-example-fileintegrity-z9678 0/2 Init:0/1 0 6s aide-reinit-ds-example-fileintegrity-5tzv7 0/1 Pending 0 6s aide-reinit-ds-example-fileintegrity-d6hfg 0/1 Init:0/1 0 6s aide-reinit-ds-example-fileintegrity-fbnx4 0/1 PodInitializing 0 5s aide-reinit-ds-example-fileintegrity-k7tn2 0/1 PodInitializing 0 5s aide-reinit-ds-example-fileintegrity-kzvqn 0/1 PodInitializing 0 6s aide-reinit-ds-example-fileintegrity-phtnv 0/1 PodInitializing 0 6s file-integrity-operator-7467f6fd47-plrfx 1/1 Running 0 7h51m ip-10-0-137-158.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h50m ip-10-0-137-196.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 46m ip-10-0-175-183.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h50m ip-10-0-178-77.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 78s ip-10-0-196-136.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h50m ip-10-0-198-162.us-east-2.compute.internal-addholdoff 0/1 Completed 0 72s $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-137-158.us-east-2.compute.internal Ready master 8h v1.18.3+a34fde4 ip-10-0-137-196.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-175-183.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-178-77.us-east-2.compute.internal Ready master 8h v1.18.3+a34fde4 ip-10-0-196-136.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-198-162.us-east-2.compute.internal Ready,SchedulingDisabled master 8h v1.18.3+a34fde4 $ oc get pod NAME READY STATUS RESTARTS AGE aide-ds-example-fileintegrity-5kps2 2/2 Running 0 39s aide-ds-example-fileintegrity-bbppr 2/2 Running 0 74s aide-ds-example-fileintegrity-flm97 2/2 Running 0 74s aide-ds-example-fileintegrity-k5gtx 2/2 Running 0 71s aide-ds-example-fileintegrity-s2p2l 0/2 Init:0/1 0 68s aide-ds-example-fileintegrity-z9678 2/2 Running 0 39s aide-reinit-ds-example-fileintegrity-5tzv7 0/1 Init:0/1 0 39s aide-reinit-ds-example-fileintegrity-d6hfg 1/1 Running 0 39s aide-reinit-ds-example-fileintegrity-fbnx4 1/1 Running 0 38s aide-reinit-ds-example-fileintegrity-k7tn2 1/1 Running 0 38s aide-reinit-ds-example-fileintegrity-kzvqn 1/1 Running 0 39s aide-reinit-ds-example-fileintegrity-phtnv 1/1 Running 0 39s file-integrity-operator-7467f6fd47-plrfx 1/1 Running 0 7h51m ip-10-0-137-158.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h51m ip-10-0-137-196.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 46m ip-10-0-175-183.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h51m ip-10-0-178-77.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 111s ip-10-0-196-136.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h51m ip-10-0-198-162.us-east-2.compute.internal-rmholdoff 0/1 ContainerCreating 0 2s $ oc get pod NAME READY STATUS RESTARTS AGE aide-ds-example-fileintegrity-5kps2 2/2 Running 0 58s aide-ds-example-fileintegrity-bbppr 2/2 Running 0 93s aide-ds-example-fileintegrity-flm97 2/2 Running 0 93s aide-ds-example-fileintegrity-k5gtx 2/2 Running 0 90s aide-ds-example-fileintegrity-s2p2l 0/2 Init:0/1 0 87s aide-ds-example-fileintegrity-z9678 2/2 Running 0 58s aide-reinit-ds-example-fileintegrity-5tzv7 1/1 Running 0 58s aide-reinit-ds-example-fileintegrity-d6hfg 1/1 Running 0 58s aide-reinit-ds-example-fileintegrity-fbnx4 1/1 Running 0 57s aide-reinit-ds-example-fileintegrity-k7tn2 1/1 Running 0 57s aide-reinit-ds-example-fileintegrity-kzvqn 1/1 Running 0 58s aide-reinit-ds-example-fileintegrity-phtnv 1/1 Running 0 58s file-integrity-operator-7467f6fd47-shxqd 0/1 ContainerCreating 0 14s ip-10-0-137-158.us-east-2.compute.internal-addholdoff 0/1 Completed 0 15s ip-10-0-137-196.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 47m ip-10-0-175-183.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h51m ip-10-0-178-77.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 2m10s ip-10-0-196-136.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 7h51m ip-10-0-198-162.us-east-2.compute.internal-rmholdoff 0/1 Completed 0 21s $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-137-158.us-east-2.compute.internal Ready,SchedulingDisabled master 8h v1.18.3+a34fde4 ip-10-0-137-196.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-175-183.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-178-77.us-east-2.compute.internal Ready master 8h v1.18.3+a34fde4 ip-10-0-196-136.us-east-2.compute.internal Ready worker 8h v1.18.3+a34fde4 ip-10-0-198-162.us-east-2.compute.internal Ready master 8h v1.18.3+a34fde4 $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-1e225db3d4bedf0f67e9ad6751c3246a False True False 3 2 2 0 8h worker rendered-worker-c7725b2664b8c0d0ffda4ff9957a7416 True False False 3 3 3 0 8h Expected Result: After apply MachineConfig, the only relevant aide-ds pod to node should re-initiate instead of all aide-ds pods. Additional Info:
I wasn't able to reproduce this with your example. the FIO pods restarted (cause they're scheduled in the master nodes), but the file-integrity daemonsets stayed intact since the change didn't affect them (the MC was meant for the master nodes while the fileintegrity object was meant for the worker nodes). Could you re-verify this? I didn't notice any re-init workload being scheduled either.
(In reply to Juan Antonio Osorio from comment #2) > I wasn't able to reproduce this with your example. the FIO pods restarted > (cause they're scheduled in the master nodes), but the file-integrity > daemonsets stayed intact since the change didn't affect them (the MC was > meant for the master nodes while the fileintegrity object was meant for the > worker nodes). Could you re-verify this? I didn't notice any re-init > workload being scheduled either. Yes, I could reproduce it every time. Below test result is based on latest payload 4.6.0-0.nightly-2020-08-18-030245. The key point in the bug in that: when one node rebooting(like below from master mcp change from 3 0 0 > 3 1 1), all aide-ds pods will reinit. That means if you have 10 nodes in one cluster, all aide-ds pods will reinit 10 times. $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-cc13710aecf8e96814b37ab3f62edd96 False True False 3 0 0 0 5h15m worker rendered-worker-f02bdb3fdd8bb7313d88fa0301778826 True False False 3 3 3 0 5h15m $ oc get pod NAME READY STATUS RESTARTS AGE aide-ds-example-fileintegrity-7wjmc 1/1 Running 0 4m17s aide-ds-example-fileintegrity-82q82 1/1 Running 0 4m17s aide-ds-example-fileintegrity-fjl2z 1/1 Running 0 4m17s aide-ds-example-fileintegrity-kdr6p 1/1 Running 0 4m17s aide-ds-example-fileintegrity-q5ccw 0/1 ContainerCreating 0 4m17s aide-ds-example-fileintegrity-tknmn 1/1 Running 0 4m17s aide-reinit-ds-example-fileintegrity-4bhbh 0/1 Init:0/1 0 12s aide-reinit-ds-example-fileintegrity-5qsjh 0/1 Init:0/1 0 38s aide-reinit-ds-example-fileintegrity-f7ld7 0/1 Init:0/1 0 38s aide-reinit-ds-example-fileintegrity-kgh2b 0/1 Init:0/1 0 38s aide-reinit-ds-example-fileintegrity-l7lnd 0/1 Init:0/1 0 38s aide-reinit-ds-example-fileintegrity-plnhn 0/1 Init:0/1 0 38s file-integrity-operator-65db875847-qnpfz 1/1 Running 0 16m xiyuan08181-8wcp8-master-0-addholdoff 0/1 ContainerCreating 0 2m2s xiyuan08181-8wcp8-master-1-rmholdoff 0/1 Completed 0 15m xiyuan08181-8wcp8-master-2-rmholdoff 0/1 Completed 0 15m xiyuan08181-8wcp8-worker-h4nw8-rmholdoff 0/1 Completed 0 15m xiyuan08181-8wcp8-worker-kxcts-rmholdoff 0/1 Completed 0 15m xiyuan08181-8wcp8-worker-s7v7h-rmholdoff 0/1 Completed 0 15m $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-cc13710aecf8e96814b37ab3f62edd96 False True False 3 1 1 0 5h16m worker rendered-worker-f02bdb3fdd8bb7313d88fa0301778826 True False False 3 3 3 0 5h16m
I think I have an idea of what is going on -- the issue seems to be related to operator restarts. 0) the nodeController is Watching for Node objects 1) the MC is deployed to master nodes. This triggers a reboot of master nodes and restarts of the operator which is running on one of them 2) for the node that is being updated, the nodeController notices that and marks the FI object as held off 3) the operator pod is restarted as the node it is running on is being rebooted and starts again 4) on startup, the nodeController receives a Reconcile object per node - keep in mind that the FI object now has the holdoff annotation - but, the first Reconcile loop is not for the node that is being updated, but already was updated - the nodeController sees that the node is done updating and the FI object is held off, so it removes the holdoff annotation and marks the FI object for reinit - then the Reconcile for the node being updated arrives and marks the FI object as held off again 5) when the next master node reboots, the operator gets restarted again and we go back to 3) again all in all, this can trigger several unneeded reinits of the FI depending on the order of the watches. I don't think the issue is too dangerous, maybe a bit irritating and confusing. I can think of several ways of solving the issue: i) before removing the holdoff annotation for a FI object, we could check if any other nodes that FI references with its nodeSelector are still being updated and only remove the holdoff if no others are still updating. This could be a quick fix. ii) instead of watching for nodes, watch for machineConfigPools and remove or add the holdoffs based on when a pool is updated. The disadvantage here is that the holdoff time might be quite long, but I'm not sure it matters too much whether we have one long pause or several shorter ones with reinits between them. Another disadvantage is that we tie the operator even more to OpenShift APIs, but I'm not sure if we care that much. The advantage is that we would have fewer synchronization points during the update. If we want to do anything in the short term, then we could do i) otherwise I think ii) is a more systematic option.
Turns out Matt is already working on a fix for this bug. I'm switching the ownership back to him, hopefully comment #4 makes it clearer that we don't need to fix this for 4.6 (QE: please holler if you disagree, of course)
Hi Matt, From the below details, I could see that after applied MachineConfig on master nodes: o The aide-ds pod re-initiate after each master node gets restarted and aide db created o However, the fileintegritynodestatus object reports the status Failed for one of master node even though the integritylog logs are same for all master nodes. This result is not expected. o One of worker node re-init ds pod is sticking around and it has re-initialise aide db twice. Verified On: 4.9.0-x86_64 + file-integrity-operator.v0.1.20 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1762715 $ oc project openshift-file-integrity Now using project "openshift-file-integrity" on server "https://api.jwei-isc-1019-1.qe.devcluster.openshift.com:6443". $ oc get csv NAME DISPLAY VERSION REPLACES PHASE elasticsearch-operator.5.2.2-33 OpenShift Elasticsearch Operator 5.2.2-33 Succeeded file-integrity-operator.v0.1.20 File Integrity Operator 0.1.20 Succeeded $ oc get pods NAME READY STATUS RESTARTS AGE file-integrity-operator-f5c454df9-xmcgz 1/1 Running 1 (4m11s ago) 4m43s $ oc apply -f - <<EOF > apiVersion: fileintegrity.openshift.io/v1alpha1 > kind: FileIntegrity > metadata: > name: example-fileintegrity > namespace: openshift-file-integrity > spec: > debug: false > config: > gracePeriod: 15 > EOF fileintegrity.fileintegrity.openshift.io/example-fileintegrity created $ oc get pods NAME READY STATUS RESTARTS AGE aide-example-fileintegrity-529vg 1/1 Running 0 50s aide-example-fileintegrity-c5z7j 1/1 Running 0 50s aide-example-fileintegrity-dp2fz 1/1 Running 0 50s aide-example-fileintegrity-gqmwj 1/1 Running 0 50s aide-example-fileintegrity-m79fq 1/1 Running 0 50s aide-example-fileintegrity-nf4zq 1/1 Running 0 50s file-integrity-operator-f5c454df9-xmcgz 1/1 Running 1 (5m32s ago) 6m4s $ oc get fileintegritynodestatus -w NAME NODE STATUS example-fileintegrity-ip-10-0-213-202.us-east-2.compute.internal ip-10-0-213-202.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-147-39.us-east-2.compute.internal ip-10-0-147-39.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-173-91.us-east-2.compute.internal ip-10-0-173-91.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-164-203.us-east-2.compute.internal ip-10-0-164-203.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-223-150.us-east-2.compute.internal ip-10-0-223-150.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-135-75.us-east-2.compute.internal ip-10-0-135-75.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-213-202.us-east-2.compute.internal ip-10-0-213-202.us-east-2.compute.internal Succeeded $ oc create -f - <<EOF > apiVersion: machineconfiguration.openshift.io/v1 > kind: MachineConfig > metadata: > creationTimestamp: "2020-06-22T11:02:48Z" > generation: 1 > labels: > machineconfiguration.openshift.io/role: master > name: 50-testfileintegrity1 > spec: > config: > ignition: > config: {} > security: > tls: {} > timeouts: {} > version: 2.2.0 > networkd: {} > passwd: {} > storage: > files: > - contents: > source: data:,file-integrity-operator-was-here > verification: {} > filesystem: root > mode: 420 > path: /etc/fi-test-file > systemd: {} > fips: false > kernelArguments: null > kernelType: "" > osImageURL: "" > EOF machineconfig.machineconfiguration.openshift.io/50-testfileintegrity1 created $ oc get mc 50-testfileintegrity1 NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 50-testfileintegrity1 2.2.0 14m $ oc get nodes -w NAME STATUS ROLES AGE VERSION ip-10-0-135-75.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-147-39.us-east-2.compute.internal Ready,SchedulingDisabled master 9h v1.22.0-rc.0+894a78b ip-10-0-164-203.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-173-91.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-213-202.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-223-150.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b $ oc get pods NAME READY STATUS RESTARTS AGE aide-example-fileintegrity-529vg 1/1 Running 0 5m49s aide-example-fileintegrity-c5z7j 1/1 Running 0 5m49s aide-example-fileintegrity-dp2fz 1/1 Running 0 5m49s aide-example-fileintegrity-gqmwj 1/1 Running 0 5m49s aide-example-fileintegrity-m79fq 1/1 Running 0 5m49s aide-example-fileintegrity-nf4zq 1/1 Running 0 5m49s aide-ini0bd9916b43277aa585731634cc4491b57697a541-94mcr 1/1 Running 0 24s file-integrity-operator-f5c454df9-xmcgz 1/1 Running 1 (10m ago) 11m $ oc get mcp -w NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-410d51b495cc9b7a0cc1070b5a5a879a False True False 3 0 0 0 9h worker rendered-worker-78aa65c6a4a32c6e19488560fd20e3c4 True False False 3 3 3 0 9h $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-135-75.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-147-39.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-164-203.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-173-91.us-east-2.compute.internal Ready,SchedulingDisabled master 9h v1.22.0-rc.0+894a78b ip-10-0-213-202.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-223-150.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-529vg 1/1 Running 0 12m 10.128.2.17 ip-10-0-135-75.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-c5z7j 1/1 Running 0 12m 10.128.0.5 ip-10-0-173-91.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-dp2fz 1/1 Running 0 12m 10.131.0.16 ip-10-0-223-150.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-gqmwj 1/1 Running 0 12m 10.130.0.12 ip-10-0-213-202.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-m79fq 1/1 Running 1 12m 10.129.0.32 ip-10-0-147-39.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-nf4zq 1/1 Running 0 12m 10.129.2.145 ip-10-0-164-203.us-east-2.compute.internal <none> <none> aide-ini0bd9916b43277aa585731634cc4491b57697a541-m2hh8 1/1 Running 0 97s 10.130.0.40 ip-10-0-213-202.us-east-2.compute.internal <none> <none> aide-inie276dbdddd487f55b1e5848763fa7fd7ba872c7f-2rwx8 1/1 Running 0 108s 10.129.0.36 ip-10-0-147-39.us-east-2.compute.internal <none> <none> file-integrity-operator-f5c454df9-xmcgz 1/1 Running 1 (16m ago) 17m 10.130.0.11 ip-10-0-213-202.us-east-2.compute.internal <none> <none> $ oc get nodes -w NAME STATUS ROLES AGE VERSION ip-10-0-135-75.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-147-39.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-164-203.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-173-91.us-east-2.compute.internal NotReady,SchedulingDisabled master 9h v1.22.0-rc.0+894a78b ip-10-0-213-202.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-223-150.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b $ oc get mcp -w NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-410d51b495cc9b7a0cc1070b5a5a879a False True False 3 2 2 0 9h worker rendered-worker-78aa65c6a4a32c6e19488560fd20e3c4 True False False 3 3 3 0 9h $ oc get nodes -w NAME STATUS ROLES AGE VERSION ip-10-0-135-75.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-147-39.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-164-203.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b ip-10-0-173-91.us-east-2.compute.internal Ready master 9h v1.22.0-rc.0+894a78b ip-10-0-213-202.us-east-2.compute.internal Ready,SchedulingDisabled master 9h v1.22.0-rc.0+894a78b ip-10-0-223-150.us-east-2.compute.internal Ready worker 9h v1.22.0-rc.0+894a78b $ oc get pods NAME READY STATUS RESTARTS AGE aide-example-fileintegrity-529vg 1/1 Running 0 20m aide-example-fileintegrity-c5z7j 1/1 Running 1 20m aide-example-fileintegrity-dp2fz 1/1 Running 0 20m aide-example-fileintegrity-gqmwj 1/1 Running 0 20m aide-example-fileintegrity-m79fq 1/1 Running 1 20m aide-example-fileintegrity-nf4zq 1/1 Running 0 20m aide-ini430fec4737cc764879e7b4bc0a691d6ebbd01616-8l8qw 1/1 Running 0 2m11s aide-inie276dbdddd487f55b1e5848763fa7fd7ba872c7f-2rwx8 1/1 Running 0 10m file-integrity-operator-f5c454df9-2845x 1/1 Running 0 2m3s $ oc get mcp -w NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-9b93831cfce247ead63e57b3a8069706 True False False 3 3 3 0 9h worker rendered-worker-78aa65c6a4a32c6e19488560fd20e3c4 True False False 3 3 3 0 9h $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-529vg 1/1 Running 0 26m 10.128.2.17 ip-10-0-135-75.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-c5z7j 1/1 Running 1 26m 10.128.0.5 ip-10-0-173-91.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-dp2fz 1/1 Running 0 26m 10.131.0.16 ip-10-0-223-150.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-gqmwj 1/1 Running 1 26m 10.130.0.12 ip-10-0-213-202.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-m79fq 1/1 Running 1 26m 10.129.0.32 ip-10-0-147-39.us-east-2.compute.internal <none> <none> aide-example-fileintegrity-nf4zq 1/1 Running 0 26m 10.129.2.145 ip-10-0-164-203.us-east-2.compute.internal <none> <none> aide-inid2ac098e022a163f44125a01fcd92839bcb92736-tsq66 1/1 Running 0 28s 10.131.0.20 ip-10-0-223-150.us-east-2.compute.internal <none> <none> file-integrity-operator-f5c454df9-2845x 1/1 Running 0 7m44s 10.128.0.15 ip-10-0-173-91.us-east-2.compute.internal <none> <none> $ oc debug node/ip-10-0-147-39.us-east-2.compute.internal -- chroot /host ls -ltr /etc/kubernetes Starting pod/ip-10-0-147-39us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` total 5768 -rw-r--r--. 1 root root 9191 Oct 19 01:40 kubeconfig drwxr-xr-x. 3 root root 19 Oct 19 06:16 cni drwxr-xr-x. 24 root root 4096 Oct 19 06:20 static-pod-resources drwxr-xr-x. 3 root root 20 Oct 19 06:20 kubelet-plugins drwxr-xr-x. 2 root root 129 Oct 19 06:20 manifests -rw-r--r--. 1 root root 109 Oct 19 11:24 apiserver-url.env -rw-r--r--. 1 root root 5875 Oct 19 11:24 kubelet-ca.crt -rw-r--r--. 1 root root 1123 Oct 19 11:24 ca.crt -rw-r--r--. 1 root root 0 Oct 19 11:24 cloud.conf -rw-r--r--. 1 root root 1076 Oct 19 11:24 kubelet.conf -rw-------. 1 root root 1952444 Oct 19 11:30 aide.db.gz.backup-20211019T11_30_34 -rw-------. 1 root root 1519 Oct 19 11:30 aide.log.backup-20211019T11_30_34 -rw-------. 1 root root 1952539 Oct 19 11:31 aide.db.gz.new -rw-------. 1 root root 1952539 Oct 19 11:31 aide.db.gz -rw-------. 1 root root 651 Oct 19 11:50 aide.log -rw-------. 1 root root 0 Oct 19 11:50 aide.log.new Removing debug pod ... $ oc debug node/ip-10-0-223-150.us-east-2.compute.internal -- chroot /host ls -ltr /etc/kubernetes Starting pod/ip-10-0-223-150us-east-2computeinternal-debug ... To use host binaries, run `chroot /host` total 7668 -rw-r--r--. 1 root root 6052 Oct 19 01:49 kubeconfig -rw-r--r--. 1 root root 5875 Oct 19 06:19 kubelet-ca.crt -rw-r--r--. 1 root root 1123 Oct 19 06:19 ca.crt -rw-r--r--. 1 root root 1076 Oct 19 06:19 kubelet.conf -rw-r--r--. 1 root root 0 Oct 19 06:19 cloud.conf drwxr-xr-x. 3 root root 19 Oct 19 06:19 cni drwxr-xr-x. 3 root root 24 Oct 19 06:20 static-pod-resources drwxr-xr-x. 3 root root 20 Oct 19 06:20 kubelet-plugins drwxr-xr-x. 2 root root 6 Oct 19 06:20 manifests -rw-------. 1 root root 1952262 Oct 19 11:39 aide.db.gz.backup-20211019T11_39_13 -rw-------. 1 root root 651 Oct 19 11:39 aide.log.backup-20211019T11_39_13 -rw-------. 1 root root 1952262 Oct 19 11:45 aide.db.gz.backup-20211019T11_45_05 -rw-------. 1 root root 651 Oct 19 11:45 aide.log.backup-20211019T11_45_05 -rw-------. 1 root root 1952262 Oct 19 11:45 aide.db.gz.new -rw-------. 1 root root 1952262 Oct 19 11:45 aide.db.gz -rw-------. 1 root root 651 Oct 19 11:51 aide.log -rw-------. 1 root root 0 Oct 19 11:51 aide.log.new Removing debug pod ... $ oc get cm NAME DATA AGE aide-example-fileintegrity-ip-10-0-147-39.us-east-2.compute.internal-failed 1 25m aide-example-fileintegrity-ip-10-0-173-91.us-east-2.compute.internal-failed 1 16m aide-example-fileintegrity-ip-10-0-213-202.us-east-2.compute.internal-failed 1 10m aide-pause 1 37m aide-reinit 1 37m example-fileintegrity 1 37m file-integrity-operator-lock 0 18m kube-root-ca.crt 1 43m openshift-service-ca.crt 1 43m $ oc get pods NAME READY STATUS RESTARTS AGE aide-example-fileintegrity-529vg 1/1 Running 0 37m aide-example-fileintegrity-c5z7j 1/1 Running 1 37m aide-example-fileintegrity-dp2fz 1/1 Running 0 37m aide-example-fileintegrity-gqmwj 1/1 Running 1 37m aide-example-fileintegrity-m79fq 1/1 Running 1 37m aide-example-fileintegrity-nf4zq 1/1 Running 0 37m aide-inid2ac098e022a163f44125a01fcd92839bcb92736-tsq66 1/1 Running 0 12m file-integrity-operator-f5c454df9-2845x 1/1 Running 0 19m $ oc get fileintegritynodestatus NAME NODE STATUS example-fileintegrity-ip-10-0-135-75.us-east-2.compute.internal ip-10-0-135-75.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-147-39.us-east-2.compute.internal ip-10-0-147-39.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-164-203.us-east-2.compute.internal ip-10-0-164-203.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-173-91.us-east-2.compute.internal ip-10-0-173-91.us-east-2.compute.internal Succeeded example-fileintegrity-ip-10-0-213-202.us-east-2.compute.internal ip-10-0-213-202.us-east-2.compute.internal Failed example-fileintegrity-ip-10-0-223-150.us-east-2.compute.internal ip-10-0-223-150.us-east-2.compute.internal Succeeded $ oc extract cm/aide-example-fileintegrity-ip-10-0-213-202.us-east-2.compute.internal-failed --confirm integritylog $ cat integritylog Start timestamp: 2021-10-19 11:57:50 +0000 (AIDE 0.16) AIDE found differences between database and filesystem!! Summary: Total number of entries: 35162 Added entries: 4 Removed entries: 0 Changed entries: 1 --------------------------------------------------- Added entries: --------------------------------------------------- f++++++++++++++++: /hostroot/etc/fi-test-file d++++++++++++++++: /hostroot/etc/machine-config-daemon/noorig d++++++++++++++++: /hostroot/etc/machine-config-daemon/noorig/etc f++++++++++++++++: /hostroot/etc/machine-config-daemon/noorig/etc/fi-test-file.mcdnoorig --------------------------------------------------- Changed entries: --------------------------------------------------- d ... n ... : /hostroot/etc/machine-config-daemon --------------------------------------------------- Detailed information about changes: --------------------------------------------------- Directory: /hostroot/etc/machine-config-daemon Linkcount: 3 | 4 --------------------------------------------------- The attributes of the (uncompressed) database(s): --------------------------------------------------- /hostroot/etc/kubernetes/aide.db.gz SHA1 : /j+Fq30m1yE6sBK34tRvSDrmorA= SHA256 : uhpKA6P+Hse2V6b/2vnlgjvYRrcz7s9v m9DnWxZqACY= SHA512 : D+NTl/k0cucEOdfKfMxZ3ocuu0ZC20XD gLPwnNHwJGJ4jsm0IG7klLSajoycE1kj cMlzrAXDh4KHgvzklP6JPg== End timestamp: 2021-10-19 11:58:38 +0000 (run time: 0m 48s) $ oc extract cm/aide-example-fileintegrity-ip-10-0-173-91.us-east-2.compute.internal-failed --confirm integritylog $ cat integritylog Start timestamp: 2021-10-19 11:37:17 +0000 (AIDE 0.16) AIDE found differences between database and filesystem!! Summary: Total number of entries: 35162 Added entries: 4 Removed entries: 0 Changed entries: 1 --------------------------------------------------- Added entries: --------------------------------------------------- f++++++++++++++++: /hostroot/etc/fi-test-file d++++++++++++++++: /hostroot/etc/machine-config-daemon/noorig d++++++++++++++++: /hostroot/etc/machine-config-daemon/noorig/etc f++++++++++++++++: /hostroot/etc/machine-config-daemon/noorig/etc/fi-test-file.mcdnoorig --------------------------------------------------- Changed entries: --------------------------------------------------- d ... n ... : /hostroot/etc/machine-config-daemon --------------------------------------------------- Detailed information about changes: --------------------------------------------------- Directory: /hostroot/etc/machine-config-daemon Linkcount: 3 | 4 --------------------------------------------------- The attributes of the (uncompressed) database(s): --------------------------------------------------- /hostroot/etc/kubernetes/aide.db.gz SHA1 : +Xv9wqkzinUmqbqKcR33CPuYoZA= SHA256 : LGXpVAPFR+pmNnAjLh9JD9GsS10Utv+t oU/P5iwdL9Y= SHA512 : z8bZHj06QeSfvDrx05ST7UFHvMukKm0K eCV1PHakxwAQY/h6npsGHzwnT3QSQJux kaXRFI7FJEQazJzsLMNqOw== End timestamp: 2021-10-19 11:38:44 +0000 (run time: 1m 27s)
Thanks, the issues I see left are still the ones described in https://issues.redhat.com/browse/CMP-1097 . We can consider this one verified since the re-inits were happening separately.
I see no fileintegritynodestatuses failur after applied MachineConfig on master nodes, suppose we can close the bug as verfied.
Correct comment23 Get same results as @Prashant o The aide-ds pod re-initiate after each master node gets restarted and aide db created o However, the fileintegritynodestatus object reports the status Failed for one of master node even though the integritylog logs are same for all master nodes. This result is not expected. hongyli@hongyli-mac fio % oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-64d7143d92fc8a3dd2cc17ea0a350883 True False False 3 3 3 0 4h37m worker rendered-worker-f0b57f0ec6d200e9e3325892da349c6d True False False 3 3 3 0 4h37m hongyli@hongyli-mac fio % oc get fileintegritynodestatus NAME NODE STATUS example-fileintegrity-hongyli-azure-oct20-gbwlj-master-0 hongyli-azure-oct20-gbwlj-master-0 Succeeded example-fileintegrity-hongyli-azure-oct20-gbwlj-master-1 hongyli-azure-oct20-gbwlj-master-1 Succeeded example-fileintegrity-hongyli-azure-oct20-gbwlj-master-2 hongyli-azure-oct20-gbwlj-master-2 Failed example-fileintegrity-hongyli-azure-oct20-gbwlj-worker-centralus1-hzl5l hongyli-azure-oct20-gbwlj-worker-centralus1-hzl5l Succeeded example-fileintegrity-hongyli-azure-oct20-gbwlj-worker-centralus2-66knh hongyli-azure-oct20-gbwlj-worker-centralus2-66knh Succeeded example-fileintegrity-hongyli-azure-oct20-gbwlj-worker-centralus3-q58pb hongyli-azure-oct20-gbwlj-worker-centralus3-q58pb Succeeded
[Bug_Verification] Looks good. After apply MachineConfig, the aide-ds pod re-initiate one by one after relevant master node restarted. Also once the all master nodes are getting restarted, the the fileintegritynodestatus object reports the status succeeded. Marking this bug as verified if we observe behaviour mentioned in comments 21,23 then we will open a another bug to track this issue. Verified On: 4.9.0-x86_64 + file-integrity-operator.v0.1.20 https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1762715 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0 True False 3h34m Cluster version is 4.9.0 $ oc get csv NAME DISPLAY VERSION REPLACES PHASE elasticsearch-operator.5.2.2-34 OpenShift Elasticsearch Operator 5.2.2-34 Succeeded file-integrity-operator.v0.1.20 File Integrity Operator 0.1.20 Succeeded $ oc get pods NAME READY STATUS RESTARTS AGE file-integrity-operator-f5c454df9-t6j2c 1/1 Running 1 (38m ago) 39m $ oc create -f - << EOF > apiVersion: fileintegrity.openshift.io/v1alpha1 > kind: FileIntegrity > metadata: > name: example-fileintegrity > namespace: openshift-file-integrity > spec: > debug: false > config: > gracePeriod: 15 > EOF fileintegrity.fileintegrity.openshift.io/example-fileintegrity created $ oc get pods NAME READY STATUS RESTARTS AGE aide-example-fileintegrity-5srhg 1/1 Running 0 3m24s aide-example-fileintegrity-9spv5 1/1 Running 0 3m24s aide-example-fileintegrity-btqnn 1/1 Running 0 3m24s aide-example-fileintegrity-r8ljd 1/1 Running 0 3m24s aide-example-fileintegrity-rw66h 1/1 Running 0 3m24s aide-example-fileintegrity-tt5l9 1/1 Running 0 3m24s file-integrity-operator-f5c454df9-t6j2c 1/1 Running 1 (43m ago) 43m $ oc get fileintegritynodestatus NAME NODE STATUS example-fileintegrity-fileintegrity1820-7wzbd-master-0 fileintegrity1820-7wzbd-master-0 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-1 fileintegrity1820-7wzbd-master-1 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-2 fileintegrity1820-7wzbd-master-2 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus1-lq69j fileintegrity1820-7wzbd-worker-centralus1-lq69j Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus2-zw7nd fileintegrity1820-7wzbd-worker-centralus2-zw7nd Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus3-6qc5z fileintegrity1820-7wzbd-worker-centralus3-6qc5z Succeeded $ oc create -f - <<EOF > apiVersion: machineconfiguration.openshift.io/v1 > kind: MachineConfig > metadata: > creationTimestamp: "2020-06-22T11:02:48Z" > generation: 1 > labels: > machineconfiguration.openshift.io/role: master > name: 50-testfileintegrity1 > spec: > config: > ignition: > config: {} > security: > tls: {} > timeouts: {} > version: 2.2.0 > networkd: {} > passwd: {} > storage: > files: > - contents: > source: data:,file-integrity-operator-was-here > verification: {} > filesystem: root > mode: 420 > path: /etc/fi-test-file > systemd: {} > fips: false > kernelArguments: null > kernelType: "" > osImageURL: "" > EOF machineconfig.machineconfiguration.openshift.io/50-testfileintegrity1 created $ oc get mc 50-testfileintegrity1 NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 50-testfileintegrity1 2.2.0 25s $ oc get mcp -w NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-f17f6acfafc4067aae7fb098656416c2 False True False 3 0 0 0 4h5m worker rendered-worker-155d8762d1680bede3b058cd5c20511d True False False 3 3 3 0 4h5m $ oc get nodes -w NAME STATUS ROLES AGE VERSION fileintegrity1820-7wzbd-master-0 Ready master 4h9m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-1 Ready,SchedulingDisabled master 4h9m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-2 Ready master 4h9m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-worker-centralus1-lq69j Ready worker 3h54m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-worker-centralus2-zw7nd Ready worker 3h55m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-worker-centralus3-6qc5z Ready worker 3h38m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-2 Ready master 4h9m v1.22.0-rc.0+894a78b $ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 5m5s 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 0 5m5s 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 5m5s 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 5m5s 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 0 5m5s 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 0 5m5s 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masswbz5 0/1 Init:0/1 0 4s <none> fileintegrity1820-7wzbd-master-0 <none> <none> file-integrity-operator-f5c454df9-5q7nm 1/1 Running 0 52s 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> $ oc get mcp -w NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-f17f6acfafc4067aae7fb098656416c2 False True False 3 0 0 0 4h9m worker rendered-worker-155d8762d1680bede3b058cd5c20511d True False False 3 3 3 0 4h9m $ oc get nodes -w NAME STATUS ROLES AGE VERSION fileintegrity1820-7wzbd-master-0 Ready master 4h13m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-1 Ready,SchedulingDisabled master 4h13m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-2 Ready master 4h13m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-worker-centralus1-lq69j Ready worker 3h58m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-worker-centralus2-zw7nd Ready worker 3h59m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-worker-centralus3-6qc5z Ready worker 3h43m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-1 Ready,SchedulingDisabled master 4h13m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-1 Ready master 4h14m v1.22.0-rc.0+894a78b fileintegrity1820-7wzbd-master-1 Ready master 4h14m v1.22.0-rc.0+894a78b $ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 8m59s 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 8m59s 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 8m59s 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 8m59s 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 0 8m59s 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 0 8m59s 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-mas6gzlg 0/1 Init:0/1 0 5s <none> fileintegrity1820-7wzbd-master-1 <none> <none> file-integrity-operator-f5c454df9-5q7nm 1/1 Running 0 4m46s 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> $ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 9m24s 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 9m24s 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 9m24s 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 9m24s 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 0 9m24s 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 0 9m24s 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-mas6gzlg 0/1 PodInitializing 0 30s 10.129.0.60 fileintegrity1820-7wzbd-master-1 <none> <none> file-integrity-operator-f5c454df9-5q7nm 1/1 Running 0 5m11s 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> $ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 14m 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 14m 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 14m 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 14m 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 0/1 ContainerCreating 1 14m <none> fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 0 14m 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 0/1 Init:0/1 0 3s <none> fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini84d4bf5622fdb1c54a43fa33eba77ab3ad71895d-bhjwv 0/1 Init:0/1 0 2s <none> fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-iniec4cef28694fecc143c024304ff0c15b5f19e3cd-2tmpd 0/1 Init:0/1 0 10s <none> fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> file-integrity-operator-f5c454df9-5q7nm 1/1 Running 0 10m 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> $ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 15m 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 15m 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 15m 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 15m 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 1 15m 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 0 15m 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 1/1 Running 0 42s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> aide-iniec4cef28694fecc143c024304ff0c15b5f19e3cd-2tmpd 1/1 Running 0 49s 10.131.0.47 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> file-integrity-operator-f5c454df9-5q7nm 1/1 Terminating 0 11m 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> file-integrity-operator-f5c454df9-kmpfq 0/1 ContainerCreating 0 10s <none> fileintegrity1820-7wzbd-master-2 <none> <none> $ oc get pods -o wide -w NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 16m 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 16m 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 16m 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 16m 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 1 16m 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 0 16m 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 1/1 Running 0 71s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> aide-iniec4cef28694fecc143c024304ff0c15b5f19e3cd-2tmpd 1/1 Running 0 78s 10.131.0.47 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> file-integrity-operator-f5c454df9-5q7nm 1/1 Terminating 0 11m 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> file-integrity-operator-f5c454df9-kmpfq 1/1 Running 0 39s 10.130.0.80 fileintegrity1820-7wzbd-master-2 <none> <none> file-integrity-operator-f5c454df9-5q7nm 0/1 Terminating 0 12m 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> file-integrity-operator-f5c454df9-5q7nm 0/1 Terminating 0 12m 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> file-integrity-operator-f5c454df9-5q7nm 0/1 Terminating 0 12m 10.128.0.75 fileintegrity1820-7wzbd-master-0 <none> <none> $ oc get fileintegritynodestatus -w NAME NODE STATUS example-fileintegrity-fileintegrity1820-7wzbd-master-0 fileintegrity1820-7wzbd-master-0 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-1 fileintegrity1820-7wzbd-master-1 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-2 fileintegrity1820-7wzbd-master-2 Failed example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus1-lq69j fileintegrity1820-7wzbd-worker-centralus1-lq69j Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus2-zw7nd fileintegrity1820-7wzbd-worker-centralus2-zw7nd Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus3-6qc5z fileintegrity1820-7wzbd-worker-centralus3-6qc5z Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus3-6qc5z fileintegrity1820-7wzbd-worker-centralus3-6qc5z Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-1 fileintegrity1820-7wzbd-master-1 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-2 fileintegrity1820-7wzbd-master-2 Failed example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus1-lq69j fileintegrity1820-7wzbd-worker-centralus1-lq69j Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus2-zw7nd fileintegrity1820-7wzbd-worker-centralus2-zw7nd Succeeded $ oc get pods -o wide -w NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 21m 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 21m 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 21m 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 21m 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 1 21m 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 0/1 ContainerCreating 1 21m <none> fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 0/1 PodInitializing 1 7m3s <none> fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-maslfm5z 0/1 PodInitializing 0 14s 10.130.0.6 fileintegrity1820-7wzbd-master-2 <none> <none> aide-ini08a93fedeff9810606d484ee905960bb3a41f69f-8bt5l 1/1 Running 0 5m7s 10.128.2.27 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> file-integrity-operator-f5c454df9-kmpfq 1/1 Running 0 6m31s 10.130.0.80 fileintegrity1820-7wzbd-master-2 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-maslfm5z 1/1 Running 0 21s 10.130.0.6 fileintegrity1820-7wzbd-master-2 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-maslfm5z 1/1 Terminating 0 22s 10.130.0.6 fileintegrity1820-7wzbd-master-2 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-maslfm5z 0/1 Terminating 0 25s 10.130.0.6 fileintegrity1820-7wzbd-master-2 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-maslfm5z 0/1 Terminating 0 25s 10.130.0.6 fileintegrity1820-7wzbd-master-2 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-maslfm5z 0/1 Terminating 0 25s 10.130.0.6 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 1 22m 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> $ oc get pods -o wide -w NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 22m 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 22m 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 22m 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 22m 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 1 22m 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 1 22m 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 0/1 PodInitializing 1 7m31s <none> fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini08a93fedeff9810606d484ee905960bb3a41f69f-8bt5l 1/1 Running 0 5m35s 10.128.2.27 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> file-integrity-operator-f5c454df9-kmpfq 1/1 Running 0 6m59s 10.130.0.80 fileintegrity1820-7wzbd-master-2 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 0/1 PodInitializing 1 7m33s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 1/1 Running 1 7m44s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 1/1 Terminating 1 7m44s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 0/1 Terminating 1 7m48s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 0/1 Terminating 1 7m48s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini-example-fileintegrity-fileintegrity1820-7wzbd-masfz66f 0/1 Terminating 1 7m48s 10.128.0.80 fileintegrity1820-7wzbd-master-0 <none> <none> $ oc get pods -o wide -w NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 22m 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 22m 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 22m 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 22m 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 1 22m 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 1 22m 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini08a93fedeff9810606d484ee905960bb3a41f69f-8bt5l 1/1 Running 0 5m59s 10.128.2.27 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> file-integrity-operator-f5c454df9-kmpfq 1/1 Running 0 7m23s 10.130.0.80 fileintegrity1820-7wzbd-master-2 <none> <none> $ oc get fileintegritynodestatus -w NAME NODE STATUS example-fileintegrity-fileintegrity1820-7wzbd-master-0 fileintegrity1820-7wzbd-master-0 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-1 fileintegrity1820-7wzbd-master-1 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-2 fileintegrity1820-7wzbd-master-2 Failed example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus1-lq69j fileintegrity1820-7wzbd-worker-centralus1-lq69j Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus2-zw7nd fileintegrity1820-7wzbd-worker-centralus2-zw7nd Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus3-6qc5z fileintegrity1820-7wzbd-worker-centralus3-6qc5z Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-2 fileintegrity1820-7wzbd-master-2 Failed $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-4f8ef3dbf995a7c914568b3a25c0ef02 True False False 3 3 3 0 4h32m worker rendered-worker-155d8762d1680bede3b058cd5c20511d True False False 3 3 3 0 4h32m $ oc get fileintegritynodestatus -w NAME NODE STATUS example-fileintegrity-fileintegrity1820-7wzbd-master-0 fileintegrity1820-7wzbd-master-0 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-1 fileintegrity1820-7wzbd-master-1 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-2 fileintegrity1820-7wzbd-master-2 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus1-lq69j fileintegrity1820-7wzbd-worker-centralus1-lq69j Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus2-zw7nd fileintegrity1820-7wzbd-worker-centralus2-zw7nd Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus3-6qc5z fileintegrity1820-7wzbd-worker-centralus3-6qc5z Succeeded $ oc get fileintegritynodestatus NAME NODE STATUS example-fileintegrity-fileintegrity1820-7wzbd-master-0 fileintegrity1820-7wzbd-master-0 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-1 fileintegrity1820-7wzbd-master-1 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-master-2 fileintegrity1820-7wzbd-master-2 Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus1-lq69j fileintegrity1820-7wzbd-worker-centralus1-lq69j Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus2-zw7nd fileintegrity1820-7wzbd-worker-centralus2-zw7nd Succeeded example-fileintegrity-fileintegrity1820-7wzbd-worker-centralus3-6qc5z fileintegrity1820-7wzbd-worker-centralus3-6qc5z Succeeded $ oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES aide-example-fileintegrity-5srhg 1/1 Running 0 31m 10.131.0.43 fileintegrity1820-7wzbd-worker-centralus2-zw7nd <none> <none> aide-example-fileintegrity-9spv5 1/1 Running 1 31m 10.129.0.56 fileintegrity1820-7wzbd-master-1 <none> <none> aide-example-fileintegrity-btqnn 1/1 Running 0 31m 10.128.2.25 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> aide-example-fileintegrity-r8ljd 1/1 Running 0 31m 10.129.2.94 fileintegrity1820-7wzbd-worker-centralus3-6qc5z <none> <none> aide-example-fileintegrity-rw66h 1/1 Running 1 31m 10.130.0.57 fileintegrity1820-7wzbd-master-2 <none> <none> aide-example-fileintegrity-tt5l9 1/1 Running 1 31m 10.128.0.74 fileintegrity1820-7wzbd-master-0 <none> <none> aide-ini08a93fedeff9810606d484ee905960bb3a41f69f-8bt5l 1/1 Running 0 14m 10.128.2.27 fileintegrity1820-7wzbd-worker-centralus1-lq69j <none> <none> file-integrity-operator-f5c454df9-kmpfq 1/1 Running 0 16m 10.130.0.80 fileintegrity1820-7wzbd-master-2 <none> <none>
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (File Integrity Operator version 0.1.21 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4631