Bug 2002519 - After upgrade file integrity operator from v0.1.16 > file-integrity-operator.v0.1.18, there are two aide pods for each node
Summary: After upgrade file integrity operator from v0.1.16 > file-integrity-operator....
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: File Integrity Operator
Version: 4.9
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.10.0
Assignee: Matt Rogers
QA Contact: xiyuan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-09 04:21 UTC by xiyuan
Modified: 2021-11-15 11:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-15 11:11:40 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift file-integrity-operator pull 200 0 None open Bug 2002519: Delete legacy daemonSets after upgrade 2021-09-21 13:50:46 UTC
Red Hat Product Errata RHBA-2021:4631 0 None None None 2021-11-15 11:11:45 UTC

Description xiyuan 2021-09-09 04:21:34 UTC 
Description of problem:
After upgrade file integrity operator from v0.1.16 > file-integrity-operator.v0.1.18, there are two aide pods for each node
$ oc get pod
NAME                                      READY   STATUS    RESTARTS      AGE
aide-ds-example-fileintegrity-6nf9k       1/1     Running   0             13m
aide-ds-example-fileintegrity-6s7hp       1/1     Running   0             13m
aide-ds-example-fileintegrity-7wlph       1/1     Running   0             13m
aide-ds-example-fileintegrity-dmshb       1/1     Running   0             13m
aide-ds-example-fileintegrity-ng8w5       1/1     Running   0             13m
aide-ds-example-fileintegrity-rqhx4       1/1     Running   0             13m
aide-ds-example-fileintegrity-rxkl7       1/1     Running   0             13m
aide-example-fileintegrity-42tlw          1/1     Running   0             10m
aide-example-fileintegrity-9qpj4          1/1     Running   0             10m
aide-example-fileintegrity-gc8rx          1/1     Running   0             10m
aide-example-fileintegrity-j52ml          1/1     Running   0             10m
aide-example-fileintegrity-m8qpd          1/1     Running   0             10m
aide-example-fileintegrity-nx557          1/1     Running   0             10m
aide-example-fileintegrity-x7ggc          1/1     Running   0             10m
file-integrity-operator-bbb79b476-kjnxf   1/1     Running   1 (11m ago)   11m

Version-Release number of selected component (if applicable):
4.9.0-0.nightly-2021-09-07-201519

How reproducible:
Always

Steps to Reproduce:
1. Install file-integrity-operator.v0.1.16
2. Create Fileintegrity:
$ oc apply -f - <<EOF
apiVersion: fileintegrity.openshift.io/v1alpha1
kind: FileIntegrity
metadata:
  name: example-fileintegrity
  namespace: openshift-file-integrity
spec:
  # Change to debug: true to enable more verbose logging from the logcollector
  # container in the aide pods
  debug: false
  config: 
    gracePeriod: 15
EOF
3. Upgrade operator to file-integrity-operator.v0.1.18, and check the pods after operator upgrade done
$ oc get ip
NAME            CSV                               APPROVAL    APPROVED
install-8rzt5   file-integrity-operator.v0.1.16   Automatic   true
install-fg7ch   file-integrity-operator.v0.1.18   Automatic   true
$ oc get csv
NAME                              DISPLAY                   VERSION   REPLACES                          PHASE
file-integrity-operator.v0.1.18   File Integrity Operator   0.1.18    file-integrity-operator.v0.1.16   Succeeded
kiali-operator.v1.4.2             Kiali Operator            1.4.2                                       Pending

Actual Results:
After operator upgrade, there should be only one aide pod for each node


Expected Results:
After operator upgrade, there should be only one aide pod for each node

Additional info:
Logs will be attached later.
Comment 2 Matt Rogers 2021-09-21 13:49:32 UTC 
Fixing with https://github.com/openshift/file-integrity-operator/pull/200
Comment 5 Prashant Dhamdhere 2021-10-19 06:44:18 UTC 
[Bug_Verification]

Looks good now. After upgrade the file integrity operator from v0.1.16 > v0.1.20,
there is only one aide pod get generated for each node. The aide-ds-example-* pods
are getting removed during update.


Verified On:

4.9.0-x86_64 + file-integrity-operator.v0.1.20
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1762715


$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0     True        False         91m     Cluster version is 4.9.0

$ oc project openshift-file-integrity
Now using project "openshift-file-integrity" on server "https://api.pdhamdhe-19.qe.devcluster.openshift.com:6443".

$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
elasticsearch-operator.5.2.2-33   OpenShift Elasticsearch Operator   5.2.2-33              Succeeded
file-integrity-operator.v0.1.16   File Integrity Operator            0.1.16                Succeeded

$ oc get pods
NAME                                       READY   STATUS    RESTARTS   AGE
file-integrity-operator-758c5cb4b7-wdfgx   1/1     Running   0          104s


$ oc create -f - <<EOF  
> apiVersion: operators.coreos.com/v1alpha1
> kind: CatalogSource
> metadata:
>   name: file-integrity-operator
>   namespace: openshift-marketplace
> spec:
>   displayName: openshift-file-integrity-operator
>   publisher: Red Hat
>   sourceType: grpc
>   image: quay.io/openshift-qe-optional-operators/file-integrity-operator-index-0.1:latest
> EOF
catalogsource.operators.coreos.com/file-integrity-operator created

$ oc get pods -nopenshift-marketplace
NAME                                                              READY   STATUS      RESTARTS   AGE
3a96b092adcbc63283408026e6146d64f3fbecf0375b5102042d2a--1-j2rr9   0/1     Completed   0          44m
8023db5d693fe57d2d938248d4780c8607ca1611aa8b73d9df6495--1-922rw   0/1     Completed   0          133m
certified-operators-bvsn5                                         1/1     Running     0          156m
community-operators-gfnhp                                         1/1     Running     0          156m
f6c9dcab52c9c936c7d0fb0b62840d609ce63ea2210c4473e68fab--1-tz5ww   0/1     Completed   0          133m
file-integrity-operator-fhnrx                                     1/1     Running     0          41m
marketplace-operator-6dc6dd9896-qqdx5                             1/1     Running     0          158m
qe-app-registry-wv6s9                                             1/1     Running     0          133m
redhat-marketplace-xtm8q                                          1/1     Running     0          156m
redhat-operators-2brvg                                            1/1     Running     0          156m

$ oc create -f - <<EOF
> apiVersion: fileintegrity.openshift.io/v1alpha1
> kind: FileIntegrity
> metadata:
>   name: example-fileintegrity
>   namespace: openshift-file-integrity
> spec:
>   debug: false
>   config: 
>     gracePeriod: 15
> EOF
fileintegrity.fileintegrity.openshift.io/example-fileintegrity created


$ oc get pods
NAME                                       READY   STATUS    RESTARTS   AGE
aide-ds-example-fileintegrity-8rrk6        1/1     Running   0          34s
aide-ds-example-fileintegrity-d8qt8        1/1     Running   0          34s
aide-ds-example-fileintegrity-f79tm        1/1     Running   0          34s
aide-ds-example-fileintegrity-hzjnv        1/1     Running   0          34s
aide-ds-example-fileintegrity-l2jhn        1/1     Running   0          34s
aide-ds-example-fileintegrity-n8mf9        1/1     Running   0          34s
file-integrity-operator-758c5cb4b7-wdfgx   1/1     Running   0          4m55s

$ oc get fileintegritynodestatus
NAME                                                               NODE                                         STATUS
example-fileintegrity-ip-10-0-130-127.us-east-2.compute.internal   ip-10-0-130-127.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-153-16.us-east-2.compute.internal    ip-10-0-153-16.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-177-237.us-east-2.compute.internal   ip-10-0-177-237.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-184-30.us-east-2.compute.internal    ip-10-0-184-30.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-216-27.us-east-2.compute.internal    ip-10-0-216-27.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-219-91.us-east-2.compute.internal    ip-10-0-219-91.us-east-2.compute.internal    Succeeded

$ oc get packagemanifest file-integrity-operator -ojsonpath={.status.channels[0].currentCSV}
file-integrity-operator.v0.1.20

$ oc patch subscriptions file-integrity-operator -p '{"spec":{"source":"file-integrity-operator"}}' --type='merge'
subscription.operators.coreos.com/file-integrity-operator patched

$ oc get subscriptions file-integrity-operator -ojsonpath={.spec} | jq -r
{
  "channel": "release-0.1",
  "name": "file-integrity-operator",
  "source": "file-integrity-operator",
  "sourceNamespace": "openshift-marketplace"
}

$ oc get csv -w
NAME                              DISPLAY                            VERSION    REPLACES   PHASE
elasticsearch-operator.5.2.2-33   OpenShift Elasticsearch Operator   5.2.2-33              Succeeded
file-integrity-operator.v0.1.16   File Integrity Operator            0.1.16                Succeeded

$ oc get ip
NAME            CSV                               APPROVAL    APPROVED
install-2z8sb   file-integrity-operator.v0.1.16   Automatic   true
install-l6bdr   file-integrity-operator.v0.1.20   Automatic   true

$ oc get csv -w
NAME                              DISPLAY                            VERSION    REPLACES                          PHASE
elasticsearch-operator.5.2.2-33   OpenShift Elasticsearch Operator   5.2.2-33                                     Succeeded
file-integrity-operator.v0.1.16   File Integrity Operator            0.1.16                                       Replacing
file-integrity-operator.v0.1.20   File Integrity Operator            0.1.20     file-integrity-operator.v0.1.16   Installing
file-integrity-operator.v0.1.20   File Integrity Operator            0.1.20     file-integrity-operator.v0.1.16   Succeeded
file-integrity-operator.v0.1.16   File Integrity Operator            0.1.16                                       Deleting
file-integrity-operator.v0.1.16   File Integrity Operator            0.1.16                                       Deleting

$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES                          PHASE
elasticsearch-operator.5.2.2-33   OpenShift Elasticsearch Operator   5.2.2-33                                     Succeeded
file-integrity-operator.v0.1.20   File Integrity Operator            0.1.20     file-integrity-operator.v0.1.16   Installing

$ oc get pods -w
NAME                                      READY   STATUS    RESTARTS     AGE
aide-ds-example-fileintegrity-8rrk6       1/1     Running   0            40m
aide-ds-example-fileintegrity-d8qt8       1/1     Running   0            40m
aide-ds-example-fileintegrity-f79tm       1/1     Running   0            40m
aide-ds-example-fileintegrity-hzjnv       1/1     Running   0            40m
aide-ds-example-fileintegrity-l2jhn       1/1     Running   0            40m
aide-ds-example-fileintegrity-n8mf9       1/1     Running   0            40m
file-integrity-operator-f5c454df9-dbld8   1/1     Running   1 (8s ago)   33s
aide-ini-example-fileintegrity-kf828      0/1     Pending   0            0s
aide-ini-example-fileintegrity-kf828      0/1     Pending   0            0s

]$ oc get csv
NAME                              DISPLAY                            VERSION    REPLACES                          PHASE
elasticsearch-operator.5.2.2-33   OpenShift Elasticsearch Operator   5.2.2-33                                     Succeeded
file-integrity-operator.v0.1.20   File Integrity Operator            0.1.20     file-integrity-operator.v0.1.16   Succeeded

$ oc get pods -w
NAME                                      READY   STATUS              RESTARTS      AGE
aide-ds-example-fileintegrity-hzjnv       1/1     Terminating         0             41m
aide-ds-example-fileintegrity-l2jhn       1/1     Terminating         0             41m
aide-ds-example-fileintegrity-n8mf9       1/1     Terminating         0             41m
aide-example-fileintegrity-fjtmq          1/1     Running             0             17s
aide-example-fileintegrity-gdjxd          0/1     ContainerCreating   0             17s
aide-example-fileintegrity-jmwmf          0/1     ContainerCreating   0             17s
aide-example-fileintegrity-l7vpf          1/1     Running             0             17s
aide-example-fileintegrity-mnxzn          0/1     ContainerCreating   0             17s
aide-example-fileintegrity-vnmt2          1/1     Running             0             17s
aide-ini-example-fileintegrity-5599x      0/1     Init:0/1            0             18s
aide-ini-example-fileintegrity-dclmk      0/1     Init:0/1            0             18s
aide-ini-example-fileintegrity-fnrpr      0/1     Init:0/1            0             18s
aide-ini-example-fileintegrity-kf828      0/1     Init:0/1            0             18s
aide-ini-example-fileintegrity-v8lld      1/1     Running             0             18s
aide-ini-example-fileintegrity-zc2zx      0/1     Init:0/1            0             18s
file-integrity-operator-f5c454df9-dbld8   1/1     Running             1 (33s ago)   58s
aide-ini-example-fileintegrity-fnrpr      0/1     PodInitializing     0             19s
aide-ds-example-fileintegrity-l2jhn       0/1     Terminating         0             41m
aide-ds-example-fileintegrity-l2jhn       0/1     Terminating         0             41m
aide-ds-example-fileintegrity-l2jhn       0/1     Terminating         0             41m
aide-example-fileintegrity-jmwmf          1/1     Running             0             18s
aide-ini-example-fileintegrity-kf828      0/1     Init:0/1            0             19s
aide-ini-example-fileintegrity-kf828      0/1     PodInitializing     0             20s
aide-ini-example-fileintegrity-fnrpr      1/1     Running             0             21s


$ oc get pods 
NAME                                      READY   STATUS    RESTARTS        AGE
aide-example-fileintegrity-fjtmq          1/1     Running   0               3m43s
aide-example-fileintegrity-gdjxd          1/1     Running   0               3m43s
aide-example-fileintegrity-jmwmf          1/1     Running   0               3m43s
aide-example-fileintegrity-l7vpf          1/1     Running   0               3m43s
aide-example-fileintegrity-mnxzn          1/1     Running   0               3m43s
aide-example-fileintegrity-vnmt2          1/1     Running   0               3m43s
aide-ini-example-fileintegrity-5599x      1/1     Running   0               3m44s
aide-ini-example-fileintegrity-dclmk      1/1     Running   0               3m44s
aide-ini-example-fileintegrity-fnrpr      1/1     Running   0               3m44s
aide-ini-example-fileintegrity-kf828      1/1     Running   0               3m44s
aide-ini-example-fileintegrity-v8lld      1/1     Running   0               3m44s
aide-ini-example-fileintegrity-zc2zx      1/1     Running   0               3m44s
file-integrity-operator-f5c454df9-dbld8   1/1     Running   1 (3m59s ago)   4m24s

$ oc describe pod file-integrity-operator-f5c454df9-dbld8 |grep -A1 "RELATED_IMAGE_OPERATOR"
      RELATED_IMAGE_OPERATOR:   registry.redhat.io/compliance/openshift-file-integrity-rhel8-operator@sha256:3a1d27c689a1283edbd809097e963d734a294a31226d6b984b86b9eb1226e77e
      OPERATOR_CONDITION_NAME:  file-integrity-operator.v0.1.20


$ oc get daemonset
NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aide-example-fileintegrity       6         6         6       6            6           <none>          104m
aide-ini-example-fileintegrity   6         6         6       6            6           <none>          104m
Comment 7 errata-xmlrpc 2021-11-15 11:11:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (File Integrity Operator version 0.1.21 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4631
Note You need to log in before you can comment on or make changes to this bug.