Bug 1356062 - oc attach didn't returning output of remote container
Summary: oc attach didn't returning output of remote container
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Andy Goldstein
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On: 1317784
Blocks: 1486196
TreeView+ depends on / blocked
 
Reported: 2016-07-13 10:56 UTC by Yadan Pei
Modified: 2017-08-29 08:06 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1486196 (view as bug list)
Environment:
Last Closed: 2016-09-23 19:14:58 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC

Description Yadan Pei 2016-07-13 10:56:50 UTC
Description of problem:
'oc attach' to a running pod hangs out and didn't return output of remote container

Version-Release number of selected component (if applicable):
oc v3.3.0.4
kubernetes v1.3.0+57fb9ac
features: Basic-Auth GSSAPI Kerberos SPNEGO


How reproducible:
Always

Steps to Reproduce:
1. Create a test pod
# oc run mypod --image=aosqe/hello-openshift --generator=run-pod/v1
2.Check pod status
# oc get pods
NAME                        READY     STATUS      RESTARTS   AGE
mypod                       1/1       Running     0          20s
3. Attach to the pod container and print logs
# oc attach mypod --loglevel=6
I0713 18:43:04.143179   10365 loader.go:327] Config loaded from file /home/yapei/.kube/config
I0713 18:43:05.805266   10365 round_trippers.go:318] GET https://master:8443/api 200 OK in 1661 milliseconds
I0713 18:43:06.215094   10365 round_trippers.go:318] GET https://master:8443/apis 200 OK in 409 milliseconds
I0713 18:43:06.558606   10365 round_trippers.go:318] GET https://master:8443/api 200 OK in 343 milliseconds
I0713 18:43:06.902311   10365 round_trippers.go:318] GET https://master:8443/apis 200 OK in 343 milliseconds
I0713 18:43:07.341622   10365 round_trippers.go:318] GET https://master:8443/api/v1/namespaces/yapei/pods/mypod 200 OK in 438 milliseconds
I0713 18:43:07.342506   10365 attach.go:238] defaulting container name to mypod
I0713 18:43:08.933794   10365 round_trippers.go:318] POST https://master:8443/api/v1/namespaces/yapei/pods/mypod/attach?container=mypod&stderr=true&stdin=true&stdout=true 101 Switching Protocols in 1591 milliseconds
I0713 18:43:08.933820   10365 round_trippers.go:318] POST https://master:8443/api/v1/namespaces/yapei/pods/mypod/attach?container=mypod&stderr=true&stdin=true&stdout=true 101 Switching Protocols in 1591 milliseconds



Actual results:
3. 'oc attach' hangs and didn't return any output from container

Expected results:
3. Should get correct output from remote container

Additional info:

Comment 1 Andy Goldstein 2016-07-13 19:51:54 UTC
This will be fixed when bug 1317784 is fixed. Additionally, we probably aren't going to be able to make this work with 100% of the images out there. For example, the hello-openshift image doesn't even work with `docker attach` using the example you have above. If you want to test attach once bug 1317784 is fixed, please use some other image like centos:centos7 instead of hello-openshift.

I'm also removing the regression keyword because it isn't a regression.

Comment 2 Yadan Pei 2016-07-14 00:52:59 UTC
That's ok, thanks for your info

Comment 3 Yadan Pei 2016-08-05 09:11:32 UTC
Checked against v3.3.0.14

I tried image centos:centos7, but the pod can't be started

# oc run testpod --image=centos:centos7 --generator=run-pod/v1
pod "testpod" created

# oc get pods
NAME        READY     STATUS      RESTARTS   AGE
testpod     0/1       Completed   4          1m

# oc get pods
NAME        READY     STATUS             RESTARTS   AGE
testpod     0/1       CrashLoopBackOff   4          2m


Here is verification steps with another image(i'm sure this image docker.io/deshuai/hello-pod:latest supports docker attach, because in v3.2.1.12 "oc attach hello-pod" return correct strings "serving on 8888,serving on 8080"):

1. Download test file 
# curl -k -O https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/hello-pod.yaml

2. Create pod
# oc create -f hello-pod.yaml
pod "hello-pod" created

3. Check pod is running
# oc get pods
NAME        READY     STATUS    RESTARTS   AGE
hello-pod   1/1       Running   0          2m

4. Check attach
# oc attach hello-pod
If you don't see a command prompt, try pressing enter.


Still couldn't get any output

Comment 4 Andy Goldstein 2016-08-05 18:21:17 UTC
"oc run testpod --image=centos:centos7 --generator=run-pod/v1" isn't a valid test case for this bz because:

1) You are using the default command for the image (bash), AND
2) You are not attaching stdin (-i)

Therefore, the container exits immediately. This is the same as 'docker run centos:centos7' - you will see it exit immediately.

You alternatively could do:

(in one terminal)
1. oc run -it testpod --image=centos:centos7 --generator=run-pod/v1
2. yum -y install less
3. less /etc/services

(in another terminal)
1. oc attach -it testpod (you should see the output from the "less" command)


As for hello-pod, that does not work with either 'oc attach' or 'docker attach', so I don't think it's a valid test image.

Comment 5 Weihua Meng 2016-08-06 18:31:04 UTC
Works fine with those steps:
(in one terminal)
1. oc run -it testpod --image=centos:centos7 --generator=run-pod/v1

(in another terminal)
1. oc attach -it testpod

openshift v3.3.0.16
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

Comment 6 Yadan Pei 2016-08-08 02:46:56 UTC
Thanks wmeng, Andy

Move to VERIFIED

Comment 7 Andy Goldstein 2016-09-21 13:10:23 UTC
It turns out there are two parts to this issue:

1) If you run a container with a TTY, it used to be that when you detached and reattached, you would have to press a key (e.g. enter) to get the screen to refresh and show you what was there. We fixed that with bug 1317784.

2) In 3.2, we used fsouza/go-dockerclient to talk to docker, and that had the option to ask docker to display the container's logs immediately after attaching. In 3.3, we switched to docker's engine-api, and the logs option is no longer available. This means that if you now attach to a container like hello-openshift and you want to see what was written to the terminal prior to attaching, you won't be able to any more (but you could get this data via 'oc logs').

I have opened a PR to docker (https://github.com/docker/docker/pull/26718) to add a logs option to the client API. Assuming it gets merged, we'd then need to update Kubernetes to use it, and then Origin.

Derek, how do you want to handle this bz? Should treat it as a regression (because of 2) and move it to UpcomingRelease and target 3.5? Or something else?

Comment 8 Derek Carr 2016-09-23 19:12:25 UTC
It's not going to be fixed until docker accepts your PR, so let's wait and see what happens there first.

Comment 9 Li Zhe 2016-11-21 06:26:34 UTC
Reproduced on ded-int-aws
OpenShift Master:
    v3.3.1.3
Kubernetes Master:
    v1.3.0+52492b4

Comment 10 Xingxing Xia 2017-05-26 08:14:31 UTC
FYI, searched this bug and did a regression check. Results are below.
3.4 (v3.4.1.27) still reproduces.
3.5 (v3.5.5.18), 3.6 (v3.6.84) fixed the bug (BTW, comment 7's PR was merged):
$ oc run mypod --image=aosqe/hello-openshift --generator=run-pod/v1
$ oc attach mypod # after pod running
If you don't see a command prompt, try pressing enter.
serving on 8081
serving on 8888

Comment 11 Andy Goldstein 2017-05-30 14:11:56 UTC
Comment 7's PR was backported in https://github.com/openshift/origin/pull/12648 (3.5+).


Note You need to log in before you can comment on or make changes to this bug.