Bug 1992255 - csi-snapshot-controller needs to handle API server downtime gracefully in SNO
Summary: csi-snapshot-controller needs to handle API server downtime gracefully in SNO
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.9
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: ---
: 4.9.0
Assignee: Fabio Bertinatto
QA Contact: Rohit Patil
URL:
Whiteboard: chaos
Depends On: 1986215
Blocks: 1984730
TreeView+ depends on / blocked
 
Reported: 2021-08-10 20:11 UTC by Fabio Bertinatto
Modified: 2021-10-18 17:45 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1986215
Environment:
Last Closed: 2021-10-18 17:45:44 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-csi-snapshot-controller-operator pull 99 0 None None None 2021-08-17 13:55:58 UTC
Github openshift csi-external-snapshotter pull 56 0 None None None 2021-08-17 13:23:01 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:45:58 UTC

Comment 1 Fabio Bertinatto 2021-08-10 20:18:17 UTC
The upstream PR [1] that addresses this issue has been merged.

Once upstream tags a new release for the csi-snapshot-controller, we'll pull the changes to our downstream fork.

[1] https://github.com/kubernetes-csi/external-snapshotter/pull/575

Comment 3 Rohit Patil 2021-08-30 11:57:55 UTC
Hi All, 

Verified on build:  4.9.0-0.nightly-2021-08-29-010334. There are no restarts observed on the csi-snapshot-controller.

Observation. As there are no restart happened with csi-snapshot-controller, getting logs for this pod with non existed container throwing message as 
"Error from server (BadRequest): previous terminated container"

Execution steps
1. Created sno cluster
rohitpatil@ropatil-mac Downloads % oc get nodes
NAME                                         STATUS   ROLES           AGE   VERSION
ip-10-0-129-189.us-west-1.compute.internal   Ready    master,worker   59m   v1.22.0-rc.0+b708912

2. Observed the logs
rohitpatil@ropatil-mac Downloads % oc get pods -n openshift-cluster-storage-operator                                         
NAME                                                READY   STATUS    RESTARTS      AGE
cluster-storage-operator-7566bdd45c-scfws           1/1     Running   1 (76m ago)   79m
csi-snapshot-controller-85b5595f65-65w6h            1/1     Running   0             77m
csi-snapshot-controller-operator-7f6c5fb9b4-2w5b9   1/1     Running   0             79m
csi-snapshot-webhook-696b489f7b-z95sq               1/1     Running   0             77m

3. Triggered the oc path command
rohitpatil@ropatil-mac Downloads % oc patch kubeapiserver/cluster --type merge -p '{"spec":{"forceRedeploymentReason":"ITERATION1"}}'
kubeapiserver.operator.openshift.io/cluster patched

4. Observed the logs
NAME                                                READY   STATUS    RESTARTS      AGE
cluster-storage-operator-7566bdd45c-scfws           1/1     Running   1 (67m ago)   70m
csi-snapshot-controller-85b5595f65-65w6h            1/1     Running   0             68m
csi-snapshot-controller-operator-7f6c5fb9b4-2w5b9   1/1     Running   0             69m
csi-snapshot-webhook-696b489f7b-z95sq               1/1     Running   0             68m

5. Error Message:
rohitpatil@ropatil-mac Downloads % oc logs -p csi-snapshot-controller-85b5595f65-65w6h -n openshift-cluster-storage-operator
Error from server (BadRequest): previous terminated container "snapshot-controller" in pod "csi-snapshot-controller-85b5595f65-65w6h" not found

Comment 6 errata-xmlrpc 2021-10-18 17:45:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.