Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1591145

Summary: [free-stg]Met Readiness probe failed if pod is scheduled to node ip-172-31-70-31.us-east-2.compute.internal and ip-172-31-71-90.us-east-2.compute.internal
Product: OpenShift Container Platform Reporter: Wenjing Zheng <wzheng>
Component: ContainersAssignee: Mrunal Patel <mpatel>
Status: CLOSED DUPLICATE QA Contact: Xiaoli Tian <xtian>
Severity: medium Docs Contact:
Priority: medium    
Version: unspecifiedCC: aos-bugs, bparees, dwalsh, jokerman, jupierce, mmccomas, mmuzila, wzheng
Target Milestone: ---Keywords: OnlineStarter
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-29 18:36:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Describe showing old warning, but pod Ready none

Description Wenjing Zheng 2018-06-14 07:33:43 UTC
Description of problem:
Met below error when pod is scheduled to node ip-172-31-70-31.us-east-2.compute.internal:
Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)

Version-Release number of selected component (if applicable):
v3.10.0-0.66.0

How reproducible:
always

Steps to Reproduce:
1.oc new-app --template=mysql-persistent (pod should be scheduled to node ip-172-31-70-31.us-east-2.compute.internal)
2.oc describe pods mysql-1-8gczb
3.

Actual results:
Met below error:
Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)

Expected results:
Should no such warning.

Additional info:
1. Although there is such warning, database can still be connected to;
2. There were two OCP bugs about this issue for crio and docker node:
https://bugzilla.redhat.com/show_bug.cgi?id=1549029
https://bugzilla.redhat.com/show_bug.cgi?id=1590762

Comment 1 Justin Pierce 2018-06-14 16:38:42 UTC
Tried to reproduce but found that the readiness warning was temporary (during early initialization of the database). Eventually, I see the 'Ready' condition go True for the pod. Please see attachment -- readiness warning event reproduced, but it is aging (8 minutes at the time of the describe).

Comment 2 Justin Pierce 2018-06-14 16:39:07 UTC
Created attachment 1451463 [details]
Describe showing old warning, but pod Ready

Comment 3 Justin Pierce 2018-06-14 16:39:54 UTC
Can you confirm whether this condition persists for a long period of time in you test?

Comment 4 Wenjing Zheng 2018-06-15 03:21:58 UTC
The warning still exist after 27 minutes:
Events:
  Type     Reason                  Age   From                                                 Message
  ----     ------                  ----  ----                                                 -------
  Normal   Scheduled               29m   default-scheduler                                    Successfully assigned mysql-1-4szzz to ip-172-31-71-90.us-east-2.compute.internal
  Normal   SuccessfulAttachVolume  29m   attachdetach-controller                              AttachVolume.Attach succeeded for volume "pvc-1542e8eb-7047-11e8-9e61-02306c0cdc4b"
  Normal   Pulling                 28m   kubelet, ip-172-31-71-90.us-east-2.compute.internal  pulling image "docker-registry.default.svc:5000/openshift/mysql@sha256:01c59852b0c720322598a5370ed93038e04af101b1e3ee55a1aab77bf600a7af"
  Normal   Pulled                  28m   kubelet, ip-172-31-71-90.us-east-2.compute.internal  Successfully pulled image "docker-registry.default.svc:5000/openshift/mysql@sha256:01c59852b0c720322598a5370ed93038e04af101b1e3ee55a1aab77bf600a7af"
  Normal   Created                 28m   kubelet, ip-172-31-71-90.us-east-2.compute.internal  Created container
  Normal   Started                 28m   kubelet, ip-172-31-71-90.us-east-2.compute.internal  Started container
  Warning  Unhealthy               28m   kubelet, ip-172-31-71-90.us-east-2.compute.internal  Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)

Comment 5 Wenjing Zheng 2018-06-15 06:22:43 UTC
I check again after hours, the warning disappears. My project is wzheng234.

Comment 8 Ben Parees 2018-06-15 14:21:35 UTC
Actually i'm not clear on why this bug was opened... if we think the issue is due to underlying implementation details in crio and docker (for which bugs exist), why was a bug opened for the image as well?

Comment 9 Wenjing Zheng 2018-06-19 02:57:48 UTC
I don't think this is an image bug, change to container like below similar bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1549029
https://bugzilla.redhat.com/show_bug.cgi?id=1590762

Comment 11 Mrunal Patel 2019-03-29 18:36:49 UTC

*** This bug has been marked as a duplicate of bug 1622476 ***