Maciej marked he always sees this.  To compare, I was not able to reproduce with one of my env's.  Some particulars on my env listed below, followed by my initial theory:


[ec2-user@ip-172-18-1-53 ~]$ oc get builds
NAME            TYPE      FROM          STATUS     STARTED         DURATION
blast-image-1   Source    Git@c4df774   Complete   2 minutes ago   1m51s
[ec2-user@ip-172-18-1-53 ~]$ 


The ec2 instance, which I spun up yesterday, is at these versions:

[ec2-user@ip-172-18-1-53 ~]$ openshift version
openshift v1.3.0-alpha.3+d99ddc1
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git
[ec2-user@ip-172-18-1-53 ~]$ 

And docker is at these versions:

[ec2-user@ip-172-18-1-53 ~]$ docker version
Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.10.x86_64
 Go version:      go1.6.2
 Git commit:      2a93377-unsupported
 Built:           Fri Jul 29 13:45:25 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.10.x86_64
 Go version:      go1.6.2
 Git commit:      2a93377-unsupported
 Built:           Fri Jul 29 13:45:25 2016
 OS/Arch:         linux/amd64
[ec2-user@ip-172-18-1-53 ~]$ 

Any difference of significance in the versions Maciej?

Initial theory:  On why the check Maciej referenced did not catch it?  Something in the path must be wrappering or replacing the io.EOF with a different error whose String() method produces "EOF".

My scan of the docker repos I had locally turned up nothing conclusive along those lines, but I probably don't have the complete set of relevant code cloned.

Though on the hackish side, my only initial thought would be to check the strings produced by the error in addition to seeing if it is actually io.EOF.  Same rationale probably would be needed for io.ErrClosedPipe.

Comments?

you should both list the version of of your origin-sti-builder image as well.  that is part of the build log output at level 5 (it's one of the first things printed).

I would expect gabe's would match his origin version since it was a fresh env, but Maciej may have an old image in his env.

[ec2-user@ip-172-18-1-53 ~]$ docker images | grep builder
openshift/origin-sti-builder                 d99ddc1             bc147a577334        46 hours ago        483.6 MB
openshift/origin-sti-builder                 latest              bc147a577334        46 hours ago        483.6 MB
openshift/origin-docker-builder              d99ddc1             a1924bd973fe        46 hours ago        483.6 MB
openshift/origin-docker-builder              latest              a1924bd973fe        46 hours ago        483.6 MB
docker.io/openshift/origin-sti-builder       v1.3.0-alpha.3      c2fa186d844a        2 days ago          483.5 MB
[ec2-user@ip-172-18-1-53 ~]$ 


d99ddc1 does match what is present in the openshift version.

Both master and builder version from the log is:
Master version "v3.2.1.9-1-g2265530", Builder version "v3.2.1.9-1-g2265530"
This is the hack day environment, as a reminder.

Both you guys should have access to my pyconpl project where I just kicked a fresh build and saw the faulty line.

So we discussed this at scrum a bit and the biggest hurdle to debug is the difficulty involved in updating the builder image in the dedicated env to provide additional debug.

We'll minimally change that V(2) debug print to %#+v, so we can get the type for future reference.

I also continued my code investigation for *other* EOF's.

I found several instances of godeps in origin/s2i that provide their own version of EOF.

However, the "text/scanner" package has its own version of EOF.  This is my leading suspect, as docker has a fileutils.go file with regexp type functionality.  The corresponding fileutils package is used by go-dockerclient's tar code, as well as docker's archive code.

Both paths are exercised by current s2i, including the sti.go:Execute method which included the goroutine Maciej highlighted earlier

Looks enough like a duck for me ... going to create a pull to check for text/scanner.EOF, as well as improve the glog.

Spoke too soon on the text/scanner theory ... its actually a rune.  Back to just augmenting the print, and seeing if we relent on checking for the error string (which I see a fair amount of in other godeps, though it is still hacky).

https://github.com/openshift/source-to-image/pull/569

Commits pushed to master at https://github.com/openshift/source-to-image

https://github.com/openshift/source-to-image/commit/dcd3d9bd00774b9d75cd4f11b6a77519826869e7
s2i: unexpected error entry in s2i build logs

Bug 1364965
Bugzilla link https://bugzilla.redhat.com/show_bug.cgi?id=1364965
Detailed explanation when gathering build output, the intent is to
ignore expect "errors" like EOF, but an EOF error showed up in the logs;
we could not decipher actual type besides io.EOF that arrived, but we augmented
to log to print the type; next time it arrives, we can add that type to the check list

https://github.com/openshift/source-to-image/commit/df7de73a185a50013fbc0558134dabe9e6bc895a
Merge pull request #569 from gabemontero/bug1364965

Merged by openshift-bot

Note, we know need to godep bump origin with the s2i commit noted in #Comment 8 to then get the sufficient debug described.

PR https://github.com/openshift/origin/pull/10425 and associated commit https://github.com/openshift/origin/commit/293df28f5c55132abf5d702b1b6c120d3243b5f7 bumped s2i in origin to the point where it contains the s2i change noted in Comment #7 and Comment #8.

Moving over to QA to try this change out (ideally in the same dedicated env where this showed up).

Reproduce with Online stg: Master version "v3.2.1.13-5-gddf7d17" Builder image version "v3.2.1.13-1-gc2a90e1"
 1.Create an application
 $ oc new-app https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json
 2.Check build logs
 ...
I0817 02:57:29.466456       1 docker.go:689] Invoking postExecution function
E0817 02:57:29.466463       1 sti.go:571] Error reading docker stdout, EOF
I0817 02:57:29.466556 1 environment.go:60] Setting 1 environment variables provided by environment file in sources
 ...


Test with devenv-rhel7_4845 (ami-db3fa5cc)
openshift v1.3.0-alpha.3+0eb9cf4
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

openshift/origin-sti-builder                 0eb9cf4             ccd809fa7f50        7 hours ago         484.8 MB
openshift/origin-sti-builder                 latest              ccd809fa7f50        7 hours ago         484.8 MB

Cannot get EOF error in buildlogs:
 ...
I0817 07:19:26.349147       1 docker.go:834] Invoking PostExecute function
I0817 07:19:26.349216       1 postexecutorstep.go:59] Skipping step: store previous image
I0817 07:19:26.349225       1 postexecutorstep.go:106] Executing step: commit image
I0817 07:19:26.349305       1 sti.go:583] 
I0817 07:19:26.357686  
 ...

When I test with ose
openshift v3.3.0.19
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

openshift3/ose-sti-builder       v3.3.0.19           199f41e0a5f0        4 days ago          595 MB

I also cannot get an EOF error in buildlogs

this pr https://github.com/openshift/origin/pull/10425 is not merged in ose v3.3.0.19

For comment12 , I confuse why cannot get EOF error in ose v3.3.0.19 without fixed pr.
But can reproduce successfully in ose v3.2.1.

@DongboYan - the PR only dropped in additional debug, not an actual fix.  The new log will tell us the type of error with "EOF" as the string, but which is not io.EOF.

Per #Comment 13 I have a suspicion on what it might be, but we need a reproduction with the additional debug.

This has been merged into ose and is in OSE v3.3.0.22 or newer.

I can't tell if this is fixed or not.
If it is not fixed, please changed this to Assigned, and not Modified.
If it is fix, please let me know what the pull request is/was.

Marking low severity to get it off the blocker list.  It's just a log message.

we've significantly refactored a lot of the s2i stdin/out handling, i'm going to put this back to ON_QA and see if it's still an issue at this point.  We haven't heard any other reports of this issue.

cannot reproduce this issue with recent ose versions

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0884