Bug 1456156
| Summary: | Pod is killed unexpectedly as DeadlineExceeded when activeDeadlineSeconds: 341547784951046144 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Weihua Meng <wmeng> |
| Component: | Node | Assignee: | Derek Carr <decarr> |
| Status: | CLOSED ERRATA | QA Contact: | Weihua Meng <wmeng> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.6.0 | CC: | aos-bugs, eparis, jokerman, mmccomas, smunilla |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-10 05:26:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I opened the following upstream pull requests: https://github.com/kubernetes/kubernetes/pull/46642 has the kubelet detect overflows that result with the provided value and make it think the active deadline was actually negative. https://github.com/kubernetes/kubernetes/pull/46640 which restricts validation of activeDeadlineSeconds to max uint32 it's possible this will be considered backwards incompatible since it tightens validation, but i think its safe to do given the value space. ORIGIN PR: https://github.com/openshift/origin/pull/14424 restricts validation of activeDeadlineSeconds to max uint32 correction, to fix cross build errors, it will be max int32. Verified on openshift v3.6.94 Failed. # oc get pod NAME READY STATUS RESTARTS AGE hello-pod1 0/1 DeadlineExceeded 0 54s tested on commit: 6473593c449ab84cfdf57361e41eae5b78e766ee $ oc create -f pod.yaml The Pod "hello-pod1" is invalid: spec.activeDeadlineSeconds: Invalid value: 341547784951046144: must be between 1 and 2147483647, inclusive @wmeng can you include the exact version you are running and the pod definition you created if you still see the problem? @epairs the exact version mentioned in comment 5, Verified on openshift v3.6.94 Failed. and pod definition in repo steps of description. and I tried again in latest OCP, openshift v3.6.96, still has this problem. Can you have a try? Could you please indicate which OCP version fixed this bug? openshift v3.6.114
$ cat ~/scripts/samples/kube/active-deadline-too-big.yaml
apiVersion: v1
kind: Pod
metadata:
name: burstable
spec:
activeDeadlineSeconds: 341547784951046144
containers:
- image: busybox
name: busybox
command:
- sleep
- "3600"
resources:
requests:
memory: 1500Mi
cpu: 300m
restartPolicy: Always
terminationGracePeriodSeconds: 0
$ oc create -f ~/scripts/samples/kube/active-deadline-too-big.yaml
The Pod "burstable" is invalid: spec.activeDeadlineSeconds: Invalid value: 341547784951046144: must be between 1 and 2147483647, inclusive
Verified on openshift v3.6.114 kubernetes v1.6.1+5115d708d7 Fixed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716 |
Description of problem: Pod is killed unexpectedly as DeadlineExceeded when activeDeadlineSeconds: 341547784951046144 This is wired that we have value overflow check, this yaml file passed check but not work properly. $ oc create -f pod.yaml Error from server (BadRequest): error when creating "pod.yaml": Pod in version "v1" cannot be handled as a Pod: [pos 139]: json: overflow integer Version-Release number of selected component (if applicable): oc v3.6.85 kubernetes v1.6.1+5115d708d7 How reproducible: Always Steps to Reproduce: 1. create pod with activeDeadlineSeconds: 341547784951046144 $ oc create -f pod.yaml pod created. cat pod.yaml apiVersion: v1 kind: Pod metadata: name: hello-pod1 spec: activeDeadlineSeconds: 341547784951046144 containers: - image: "docker.io/ocpqe/hello-pod" name: hello-pod 2. check pod $ oc get pod $ oc describe pod Actual results: NAME READY STATUS RESTARTS AGE hello-pod1 0/1 DeadlineExceeded 0 4s <---snip---> Status: Failed Reason: DeadlineExceeded Message: Pod was active on the node longer than the specified deadline <---snip---> Expected results: pod should be running for specified time, should not be killed too soon. Additional info: If invalid value set for activeDeadlineSeconds, we can get error $ oc create -f pod.yaml Error from server (BadRequest): error when creating "pod.yaml": Pod in version "v1" cannot be handled as a Pod: [pos 139]: json: overflow integer OCP 3.5 also has the same issue.