Bug 1996638 - Helm operator manager container restart when CR is creating&deleting
Summary: Helm operator manager container restart when CR is creating&deleting
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Operator SDK
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.11.0
Assignee: Bryce Palmer
QA Contact: Fan Jia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-23 11:20 UTC by Fan Jia
Modified: 2022-08-10 10:37 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
N/A
Clone Of:
Environment:
Last Closed: 2022-08-10 10:36:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:37:07 UTC

Comment 3 Jesus M. Rodriguez 2022-06-06 18:51:40 UTC
@bpalmer

Comment 4 Jesus M. Rodriguez 2022-06-06 20:38:04 UTC
The expected and actual results from the original comment are flipped. We expect it to work and the actual results is a failure.

Comment 5 Bryce Palmer 2022-06-06 21:37:22 UTC
When taking a look at this and testing with the latest ocp-release-operator-sdk, following the steps results in a controller-manager log output that contains the logged error.

Output of `kubectl logs nginx-controller-manager-6fd9f84dbc-x2g8c`

```
{"level":"error","ts":1654551120.0068138,"logger":"controller.nginx-controller","msg":"Reconciler error","name":"nginx-sample","namespace":"default","error":"Operation cannot be fulfilled on nginxes.helmdemo.example.com \"nginx-sample\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\toperator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

```


Pod does not restart, seen by output of `kubectl get pods`:

```
NAME                                                              READY   STATUS      RESTARTS   AGE
3a5a1bcea941fcf2724f8c6000df5fab6f6ba6972c7b9d70018f8b9888mms9j   0/1     Completed   0          4m15s
docker-io-bpalmer-nginx-bundle-v4-9                               1/1     Running     0          4m24s
nginx-controller-manager-6fd9f84dbc-x2g8c                         2/2     Running     0          4m3s
nginx-sample-685f77dcbc-84mf6                                     1/1     Running     0          3m37s
nginx-sample-685f77dcbc-tbsld                                     1/1     Running     0          3m37s

```

Comment 7 Bryce Palmer 2022-06-22 18:48:02 UTC
I tested this using the latest commit on the master branch of https://github.com/openshift/ocp-release-operator-sdk and OpenShift 4.11.0-fc.0 using the following steps:

1. `operator-sdk version`:
```
operator-sdk version: "v1.22.0-ocp", commit: "8e224f3704e7c9fdd84aac3a8f76364c68249372", kubernetes version: "v1.24.1", go version: "go1.18.3", GOOS: "linux", GOARCH: "amd64"

```

2. `mkdir ocp-latest-nginx && cd ocp-latest-nginx`

3. `operator-sdk init --plugins=helm --domain=example.com --group=helmdemo --version=v1 --kind=Nginx`

4. edit *config/samples/helmdemo_v1_nginx.yaml*
```
apiVersion: helmdemo.example.com/v1
kind: Nginx
metadata:
  name: nginx-sample
...
spec:
...
  replicaCount: 2
  service:
    port: 8080

```

5. edit `Dockerfile`
```
FROM quay.io/operator-framework/helm-operator:v1.22
```
(Used the upstream image for verification that it works as I am not sure the downstream image has been updated to use OSDK v1.22.0 at this time, but the v4.11 should be updated soon as the downstream repo has been synced by: https://github.com/openshift/ocp-release-operator-sdk/pull/264 )

6. generate the bundle image:
- `make docker-build docker-push IMG=docker.io/bpalmer/ocp-latest-nginx-operator:v4.11`

- edit Makefile by setting the IMG to be docker.io/bpalmer/ocp-latest-nginx-operator:v4.11
```
IMG ?= docker.io/bpalmer/ocp-latest-nginx-operator:v4.11
```

- `make bundle`

- `make bundle-build bundle-push BUNDLE_IMG=docker.io/bpalmer/ocp-latest-nginx-bundle:v4.11`

7. `operator-sdk run bundle docker.io/bpalmer/ocp-latest-nginx-bundle:v4.11`

8. Create CR with `kubectl apply -f config/samples/helmdemo_v1_nginx.yaml`

After letting the cluster run for a few minutes after creating the CR we can see that there are no restarts on the controller-manager:
```
NAME                                                              READY   STATUS      RESTARTS   AGE
c384cf1887a53bd1be9ca55a1476ddf47beace8603af733b862348edccgzmd9   0/1     Completed   0          4m34s
docker-io-bpalmer-ocp-latest-nginx-bundle-v4-11                   1/1     Running     0          4m39s
nginx-sample-5f9c6d457-hhn8k                                      1/1     Running     0          3m7s
nginx-sample-5f9c6d457-k2tt7                                      1/1     Running     0          3m7s
ocp-latest-nginx-controller-manager-7cbd84bc8c-shbfm              2/2     Running     0          4m18s

```

The results from my testing lead me to believe that this problem should be resolved with the latest ocp-release-operator-sdk version (v1.22.0).

Comment 8 Fan Jia 2022-06-23 07:19:47 UTC
test env: 
operator-sdk version: "v1.22.0-ocp", commit: "7cce8230bda4895aacaee6103fe9c1dadc87b77d", kubernetes version: "v1.24.1", go version: "go1.18.3", GOOS: "linux", GOARCH: "amd64"

test pass.

Comment 10 errata-xmlrpc 2022-08-10 10:36:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069


Note You need to log in before you can comment on or make changes to this bug.