Bug 1552670
| Summary: | oc exec -i process hangs waiting for stdin | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Rich Megginson <rmeggins> |
| Component: | oc | Assignee: | Maciej Szulik <maszulik> |
| Status: | CLOSED ERRATA | QA Contact: | Xingxing Xia <xxia> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.10.0 | CC: | aos-bugs, jhonce, jmontleo, jokerman, mmccomas, mpatel, pportant |
| Target Milestone: | --- | ||
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause:
Connection was not properly closed.
Consequence:
oc exec was hanging.
Fix:
Implement proper closing method.
Result:
oc exec -i does not hang.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-03-13 22:27:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1550644 | ||
|
Description
Rich Megginson
2018-03-07 14:10:12 UTC
Juan, FWIW I think BZ#1550644 may be related to this rather than the crio issue it was marked as a duplicate of. Last night I was duplicating the issue in that BZ with cat doing 'echo a | kubectl exec <pod> cat, after finding https://github.com/kubernetes/kubernetes/issues/13394. More often than not it would return to a prompt, but I still got a number of hangs. I also saw the tar / cat processes in the containers hang on read(0, when I attached strace yesterday. I also noticed that even though oc rsync is working 100% of the time for me oc rsync --strategy=tar fails at a similar rate to oc cp and exec, with the tar process in the container hanging on read(0, Wondering if this is related to the docker bump to 1.13, since there have been no CLI/exec related changes in the past several weeks. We saw other behavior changes related to exec + stderr handling (https://github.com/openshift/origin/issues/18726). opened debugging PR in https://github.com/openshift/origin/pull/18881 Jason, > FWIW I think BZ#1550644 may be related to this rather than the crio issue it was marked as a duplicate of. I agree. Will update https://bugzilla.redhat.com/show_bug.cgi?id=1550644. CC'ing container team. Related upstream issue: https://github.com/moby/moby/issues/36516 I rebuilt openshift and oc binaries with https://github.com/openshift/origin/pull/18883 After updating the server binary and restarting services I had no freezes with a few runs of 'for i in $(seq 1 500); do echo $i | oc exec -i docker-registry-1-p95mw cat; done' Waiting PR lands in OCP and then will check. First, tried to confirm way to reproduce issue: In on hand old version OCP 3.9.1 with docker 1.12,try repeating command [1], exec command not hangs. Later upgrade docker 1.12 to 1.13, restart docker/master/node, try command again, it hangs often enough. Second, install new version OCP 3.9.4 with docker 1.13, try same command, exec didn't hang. So moving to VERIFIED [1] the tried steps: $ oc new-app mysql-ephemeral # wait pod running $ for i in $(seq 1 500) do echo "$i testing ... `date '+%H:%M:%S'`" echo $i | oc exec -i mysql-1-b9nb7 cat # comment 7's way echo "$i ended `date '+%H:%M:%S'`" # check timestamp difference echo "-----------------" done It gets output: ... ----------------- 109 testing ... 17:18:00 109 ended 17:18:22 ... Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0403 |