Bug 1983205 - StatefulSet fails to deploy with error Readiness Probe exec failed open /dev/tty failure no such address when .spec.tty is set to true [OCP 4.6.34]
Summary: StatefulSet fails to deploy with error Readiness Probe exec failed open /dev/...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.6
Hardware: All
OS: All
high
high
Target Milestone: ---
: 4.9.0
Assignee: Peter Hunt
QA Contact: pmali
URL:
Whiteboard:
Depends On:
Blocks: 1985539
TreeView+ depends on / blocked
 
Reported: 2021-07-16 20:20 UTC by Hradayesh Shukla
Modified: 2023-09-15 01:11 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-18 17:39:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github cri-o cri-o pull 5107 0 None closed oci: properly handle tty on execsync 2021-07-23 20:16:07 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:40:15 UTC

Description Hradayesh Shukla 2021-07-16 20:20:18 UTC
Description of problem:
StatefulSet fails to deploy with error Readiness Probe exec failed open /dev/tty failure no such address when .spec.tty is set to true []


Version-Release number of selected component (if applicable):
OCP 4.6.34


How reproducible:
100%


Steps to Reproduce:
1. Create a statefulset with ReadinessProbe exec and specifiy .spec.tty to true and try to deploy it. 

2. The pods will fail due to failed ReadinessProbe. 

3. If the .spec.tty is set to false, the pod is scheduled successfully and it works without any issue. 
 

Actual results:
- We see failed readiness probe with following error message: 
~~~
  Warning  Unhealthy       77m                    kubelet            Readiness probe failed: time="2021-07-02T15:29:49Z" level=error msg="exec failed: open /dev/tty: no such device or address"
  Warning  Unhealthy       77m                    kubelet            Readiness probe failed: time="2021-07-02T15:29:59Z" level=error msg="exec failed: open /dev/tty: no such device or address"
~~~


Expected results:
- Stateful set should deploy properly. 


Additional info:
- I feel this could be related to this issue reported here [0], but I am not sure.   

[0] [rootless+"privileged"] creating device nodes caused: open /..<snipped>../dev/tty: no such device or address #2450  https://github.com/opencontainers/runc/issues/2450

Comment 2 molnars 2021-07-17 11:56:37 UTC
we are seeing the same behavior in OCP 4.6.36 as well, but not in 4.6.20.


(In reply to Hradayesh Shukla from comment #0)
> Description of problem:
> StatefulSet fails to deploy with error Readiness Probe exec failed open
> /dev/tty failure no such address when .spec.tty is set to true []
> 
> 
> Version-Release number of selected component (if applicable):
> OCP 4.6.34
> 
> 
> How reproducible:
> 100%
> 
> 
> Steps to Reproduce:
> 1. Create a statefulset with ReadinessProbe exec and specifiy .spec.tty to
> true and try to deploy it. 
> 
> 2. The pods will fail due to failed ReadinessProbe. 
> 
> 3. If the .spec.tty is set to false, the pod is scheduled successfully and
> it works without any issue. 
>  
> 
> Actual results:
> - We see failed readiness probe with following error message: 
> ~~~
>   Warning  Unhealthy       77m                    kubelet           
> Readiness probe failed: time="2021-07-02T15:29:49Z" level=error msg="exec
> failed: open /dev/tty: no such device or address"
>   Warning  Unhealthy       77m                    kubelet           
> Readiness probe failed: time="2021-07-02T15:29:59Z" level=error msg="exec
> failed: open /dev/tty: no such device or address"
> ~~~
> 
> 
> Expected results:
> - Stateful set should deploy properly. 
> 
> 
> Additional info:
> - I feel this could be related to this issue reported here [0], but I am not
> sure.   
> 
> [0] [rootless+"privileged"] creating device nodes caused: open
> /..<snipped>../dev/tty: no such device or address #2450 
> https://github.com/opencontainers/runc/issues/2450

Comment 10 Peter Hunt 2021-07-20 16:40:57 UTC
weird I'm not able to reproduce on 4.6.34 with the reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=1983205#c5

is there anything I may be missing? Both with and without tty run no problem

Comment 12 Peter Hunt 2021-07-20 20:54:14 UTC
I have possibly found a fix to the issue in the attached PR

Comment 13 Peter Hunt 2021-07-20 20:54:46 UTC
setting target for that PR. I will backport to appropriate branches if it proves to work.

Comment 29 dlandis@us.ibm.com 2021-08-15 15:44:08 UTC
Hi -- are there any plans yet to backport this fix, in particular to the 4.6.x versions?

Comment 34 mg4924.kim 2021-09-29 08:31:00 UTC
Hi.

I've got this issue as well.

In 4.6.43 there was no issue. But in 4.6.45 it occurred.

When will it be fixed? Why is it occured again..?

Comment 39 errata-xmlrpc 2021-10-18 17:39:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

Comment 40 Red Hat Bugzilla 2023-09-15 01:11:38 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.