Bug 1999133 - removing and recreating static pod manifest leaves pod in error state
Summary: removing and recreating static pod manifest leaves pod in error state
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.9
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.10.0
Assignee: Ryan Phillips
QA Contact: MinLi
URL:
Whiteboard:
: 2001078 (view as bug list)
Depends On:
Blocks: 1981999 2005108
TreeView+ depends on / blocked
 
Reported: 2021-08-30 13:32 UTC by Ryan Phillips
Modified: 2023-09-15 01:14 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:37:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 938 0 None open Bug 1999133: kubelet: Handle UID reuse in pod worker 2021-09-14 21:21:51 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:38:16 UTC

Description Ryan Phillips 2021-08-30 13:32:44 UTC
Description of problem:
Removing and recreating a static pod can leave it in an error state.

Upstream issue: https://github.com/kubernetes/kubernetes/issues/104648

Comment 5 Peter Hunt 2021-09-09 15:32:20 UTC
*** Bug 2001078 has been marked as a duplicate of this bug. ***

Comment 14 MinLi 2021-10-14 08:24:27 UTC
test on 4.10.0-0.nightly-2021-10-13-081040, I think the issue is fixed, but it's better not to show "RESTARTS 1 (26s ago)" in output of "oc get pod"(Please see my following steps)
@Ryan, what do you think? 

create a static pod in /etc/kubernetes/manifests
sh-4.4# echo '
> kind: Pod
> apiVersion: v1
> metadata:
>   name: mypod
> spec:
>   terminationGracePeriodSeconds: 1
>   containers:
>   - name: busybox
>     image: busybox
>     command: ["sh", "-c", "echo $RANDOM; sleep 100000"]
> ' > mypod.yaml

$ oc get pod 
NAME                                               READY   STATUS    RESTARTS   AGE
ip-10-0-189-142us-east-2computeinternal-debug      1/1     Running   0          2m7s
mypod-ip-10-0-189-142.us-east-2.compute.internal   1/1     Running   0          17s

remove and recreate the static pod
sh-4.4# rm mypod.yaml 
sh-4.4# echo '
> kind: Pod
> apiVersion: v1
> metadata:
>   name: mypod
> spec:
>   terminationGracePeriodSeconds: 1
>   containers:
>   - name: busybox
>     image: busybox
>     command: ["sh", "-c", "echo $RANDOM; sleep 100000"]
> ' > mypod.yaml

$ oc get pod -o wide 
NAME                                               READY   STATUS    RESTARTS      AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
ip-10-0-189-142us-east-2computeinternal-debug      1/1     Running   0             6m42s   10.0.189.142   ip-10-0-189-142.us-east-2.compute.internal   <none>           <none>
mypod-ip-10-0-189-142.us-east-2.compute.internal   1/1     Running   1 (26s ago)   6s      10.128.2.109   ip-10-0-189-142.us-east-2.compute.internal   <none>           <none>
// here, the pod recreate, but it show AGE 6s,  RESTARTS 1 (26s ago), it's not reasonable.

Comment 15 Ryan Phillips 2021-10-14 16:36:27 UTC
Looks like it worked fine...

Comment 16 MinLi 2021-10-15 03:57:34 UTC
Hi, Ryan
I mean the pod's AGE is 6s, but RESTARTS "26s ago", that mean the pod restart before it is created. It's not reasonable in term of time.
If kubelet take the removal of the same static pod manifest as a restart, it should not recalculate pod's AGE.

$ oc get pod -o wide 
NAME                                               READY   STATUS    RESTARTS      AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
ip-10-0-189-142us-east-2computeinternal-debug      1/1     Running   0             6m42s   10.0.189.142   ip-10-0-189-142.us-east-2.compute.internal   <none>           <none>
mypod-ip-10-0-189-142.us-east-2.compute.internal   1/1     Running   1 (26s ago)   6s      10.128.2.109   ip-10-0-189-142.us-east-2.compute.internal   <none>           <none>
// here, the pod recreate, but it show AGE 6s,  RESTARTS 1 (26s ago), it's not reasonable.

Comment 17 MinLi 2021-10-18 02:19:07 UTC
confirm with Ryan, this is a static pod and it has different rules behind it. 
the test here is to make sure the pod restarts, verified!

Comment 20 errata-xmlrpc 2022-03-12 04:37:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056

Comment 21 Red Hat Bugzilla 2023-09-15 01:14:25 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.