Bug 1308712 - Tests fail with docker 1.9
Tests fail with docker 1.9
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Routing (Show other bugs)
3.x
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Maru Newby
zhaozhanqi
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-15 15:54 EST by Dan McPherson
Modified: 2016-05-12 13:09 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 13:09:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Test log (1.18 MB, text/plain)
2016-02-15 15:54 EST, Dan McPherson
no flags Details
Test log 2 (1.37 MB, text/plain)
2016-02-23 09:43 EST, Dan McPherson
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Origin (Github) 7574 None None None 2016-02-23 16:53 EST

  None (edit)
Description Dan McPherson 2016-02-15 15:54:44 EST
Created attachment 1127428 [details]
Test log

Description of problem:

Running [test] with docker 1.9 fails consistently



Version-Release number of selected component (if applicable):


How reproducible:

100% excluding random flakes before getting to this point


Steps to Reproduce:
vagrant origin-init --stage os --os rhel7 --instance-type m4.large <yourname>-dev
vagrant up --provider aws
vagrant build-origin-base
vagrant clone-upstream-repos --clean
vagrant checkout-repos
vagrant build-origin-base-images
vagrant install-origin-assets-base
vagrant install-origin
vagrant build-origin-base-images
vagrant build-origin --images
vagrant build-sti --binary-only
vagrant test-origin --all > /tmp/out.txt
echo $?

Actual results:

sent 91399 bytes  received 210 bytes  61072.67 bytes/sec
total size is 90650  speedup is 0.99
[INFO] Back to 'default' project with 'admin' user...
Now using project "default" from context named "default/172-18-8-6:8443/system:admin" on server "https://172.18.8.6:8443".
[INFO] Waiting for command to finish: '[[ "$(oc get endpoints router --output-version=v1beta3 --template="{{ if .subsets }}{{ len .subsets }}{{ else }}0{{ end }}" || echo "0")" != "0" ]]'...
[INFO] Success running command: '[[ "$(oc get endpoints router --output-version=v1beta3 --template="{{ if .subsets }}{{ len .subsets }}{{ else }}0{{ end }}" || echo "0")" != "0" ]]' after 0 seconds
[INFO] Validating privileged pod exec
Login successful.

Using project "default".
Now using project "default" from context named "default/172-18-8-6:8443/system:admin" on server "https://172.18.8.6:8443".
[INFO] Validating routed app response...
[INFO] Response is invalid: 
[INFO] Stacked plot for log subset "iops" written to /tmp/openshift/test-end-to-end-docker//logs/iops.pdf
[INFO] Stacked plot for log subset "paging" written to /tmp/openshift/test-end-to-end-docker//logs/paging.pdf
[INFO] Stacked plot for log subset "cpu" written to /tmp/openshift/test-end-to-end-docker//logs/cpu.pdf
[INFO] Stacked plot for log subset "queue" written to /tmp/openshift/test-end-to-end-docker//logs/queue.pdf
[INFO] Stacked plot for log subset "memory" written to /tmp/openshift/test-end-to-end-docker//logs/memory.pdf

[FAIL] !!!!! Test Failed !!!!



Expected results:
SUCCESS



Additional info:
Comment 1 Maru Newby 2016-02-16 13:22:47 EST
Failing tests are: 

TestImageStreamImportRedHatRegistry
TestRegistryClientConnectPulpRegistry
Comment 2 Maru Newby 2016-02-16 13:25:52 EST
(In reply to Maru Newby from comment #1)
> Failing tests are: 
> 
> TestImageStreamImportRedHatRegistry
> TestRegistryClientConnectPulpRegistry

I see those tests were passing in the attached test log.  Re-running to see if failure is consistent.
Comment 3 Dan McPherson 2016-02-16 13:29:58 EST
I missed a step 1) in the recreate steps.  You first need to pull in:

https://github.com/openshift/vagrant-openshift/pull/396

and rebuild vagrant-openshift so it will use docker-1.9
Comment 4 Maru Newby 2016-02-17 09:01:41 EST
I ran again with vagrant-openshift 3.0.3 and saw the missing image failures.  Those issues should be resolved now, re-running to see if a clean result is possible.
Comment 5 Maru Newby 2016-02-23 01:37:53 EST
New error suggests a problem with the test environment unrelated to docker 1.9:

Running hack/../hack/../test/end-to-end/core.sh:62: executing 'oc create -f examples/image-streams/image-streams-centos7.json --namespace=openshift' expecting success...
FAILURE after 0.660s: hack/../hack/../test/end-to-end/core.sh:62: executing 'oc create -f examples/image-streams/image-streams-centos7.json --namespace=openshift' expecting success: the command returned the wrong error code
There was no output from the command.
Standard error from the command:
Error from server: imagestreams "ruby" already exists
Error from server: imagestreams "nodejs" already exists
Error from server: imagestreams "perl" already exists
Error from server: imagestreams "php" already exists
Error from server: imagestreams "python" already exists
Error from server: imagestreams "wildfly" already exists
Error from server: imagestreams "mysql" already exists
Error from server: imagestreams "postgresql" already exists
Error from server: imagestreams "mongodb" already exists
Error from server: imagestreams "jenkins" already exists
!!! Error in hack/../hack/../test/end-to-end/../../hack/cmd_util.sh:195
	'return 1' exited with status 1
Call stack:
	1: hack/../hack/../test/end-to-end/../../hack/cmd_util.sh:195 os::cmd::expect_success(...)
	2: hack/../hack/../test/end-to-end/core.sh:62 main(...)
Exiting with status 1
Comment 6 Maru Newby 2016-02-23 01:53:43 EST
I can't reproduce the problem by running hack/test-end-to-end.sh locally, is there something in the reproduction steps that would be creating the imagestreams in question before test/end-to-end/core.sh is invoked?
Comment 7 Dan McPherson 2016-02-23 07:49:29 EST
Not sure but that error looks familiar.  It's probably just a different test flake.  The error I got was received 3 or 4 times in a row.  I'll run it now and see if anything has changed.
Comment 8 Dan McPherson 2016-02-23 09:41:07 EST
Still getting the same error:


There was no output from the command.
Standard error from the command:
W0223 09:25:26.734739     324 cmd.go:213] -p POD is DEPRECATED and will be removed in a future version. Use exec POD instead.
STDIN is not a terminal
[INFO] Validating routed app response...
[INFO] Response is invalid: 
[INFO] Stacked plot for log subset "iops" written to /tmp/openshift/test-end-to-end-docker//logs/iops.pdf
[INFO] Stacked plot for log subset "paging" written to /tmp/openshift/test-end-to-end-docker//logs/paging.pdf
[INFO] Stacked plot for log subset "cpu" written to /tmp/openshift/test-end-to-end-docker//logs/cpu.pdf
[INFO] Stacked plot for log subset "queue" written to /tmp/openshift/test-end-to-end-docker//logs/queue.pdf
[INFO] Stacked plot for log subset "memory" written to /tmp/openshift/test-end-to-end-docker//logs/memory.pdf

[FAIL] !!!!! Test Failed !!!!

[INFO] Dumping container logs to /tmp/openshift/test-end-to-end-docker//logs
[INFO] Dumping all resources to /tmp/openshift/test-end-to-end-docker//logs/export_all.json
[INFO] Dumping etcd contents to /tmp/openshift/test-end-to-end-docker//artifacts/etcd_dump.json

[INFO] remove the openshift container
origin
origin
[INFO] Stopping k8s docker containers
e72514700216
c6cd07f5a398
e1d796eedba5
c502f30abc12
a8dc360a32b0
77627ef9b638
41fd8ef24e43
b8c8034c834e
bc5b3a40344f
fbe7b2d21f43
[INFO] Removing k8s docker containers
d521d265a0f1
a322357b2c83
bd3d27b47494
716bfbe8bb98
d6733fc8c0c8
38026548c0ff
42d8e141666d
e72514700216
6e6b78715ea2
c6cd07f5a398
e1d796eedba5
c502f30abc12
182b1e787f9c
b9b6f17f7863
d75d7985c9cb
c1532a96240a
c304758b5b38
f8c56b5c1fa7
a8dc360a32b0
77627ef9b638
2d65d6784fe3
b4e23f440bb8
d6992ab774f1
3a7704763668
5dddd5d239d2
81fb958e15a7
dc003edfe3d3
bfd16f47122e
dde40b4f0e1d
65fb9d4482cb
2d3c317b0ed6
1d8f753e9c94
a9cc16804360
1d983358e82c
f6d11da91b92
2d16f3644d5a
f1e5a23575ee
3a70a24046da
c45a880e2e1c
4345a1c76d30
ee0c021f829a
1f012b75146d
e7a63f49c79e
04e9dc72186f
969f2eb09f44
b863032996eb
7ba0f9293494
3eb7755d5dc0
22d5c1f13afb
9a123175ed4d
3835b32ccb4d
d6ec39ccfe04
5857158d5894
40ac9ee84d9e
f1296a331416
30aff2582134
35ff3df4674a
701fe7aad8db
dd4bd293787e
41fd8ef24e43
b8c8034c834e
bc5b3a40344f
fbe7b2d21f43
53a89ca51cc2
c02e7da6a85e
285e452f21c4
32bf9e0c2e07
[INFO] Exiting
hack/../hack/test-end-to-end-docker.sh took 631 seconds
make[1]: Leaving directory `/data/src/github.com/openshift/origin'


Stderr from the command:

go tool pprof: signal: broken pipe
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    87  100    87    0     0    560      0 --:--:-- --:--:-- --:--:--   557
100    87  100    87    0     0    560      0 --:--:-- --:--:-- --:--:--   557
/data/src/github.com/openshift/origin/hack/lib/log.sh: line 30:  6258 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"
The connection to the server 127.0.0.1:12000 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:12000 was refused - did you specify the right host or port?
/data/src/github.com/openshift/origin/hack/util.sh: line 526:  6328 Terminated              openshift start etcd --config=${MASTER_CONFIG_DIR}/master-config.yaml &>${LOG_DIR}/etcd.log
info: password for stats user admin has been set to LxqlMjL6YQ
The import completed successfully.

Command "build-logs" is deprecated, use "oc logs build/<build-name>" instead.
!!! Error in hack/../hack/../test/end-to-end/core.sh:433
	'return 1' exited with status 1
Call stack:
	1: hack/../hack/../test/end-to-end/core.sh:433 main(...)
Exiting with status 1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1306k    0 1306k    0     0  7860k      0 --:--:-- --:--:-- --:--:-- 7871k
make[1]: *** [test-end-to-end] Error 1
make: *** [test] Error 2
Comment 9 Dan McPherson 2016-02-23 09:43 EST
Created attachment 1129773 [details]
Test log 2
Comment 10 Maru Newby 2016-02-24 13:06:12 EST
Origin PR has merged.

Note You need to log in before you can comment on or make changes to this bug.