Description of problem: 1) Failure: test_user_can_create_key(SettingsFlowsTest) [/root/openshift-test/site/test/integration/settings_flows_test.rb:34]: Expected at least 1 element matching "td", found 0. Ex: https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5317/consoleFull Almost always works on retries.
Can't recreate - https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5322/consoleFull
This error hasn't happened in the last 5 runs: https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5329/consoleFull https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5328/consoleFull https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5327/consoleFull https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5326/consoleFull https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5325/consoleFull
There were plenty of other failures in those runs. Want to fix those instead :)
Hmmm, seeing this one intermittently: test_user_can_create_domain(SettingsFlowsTest) [/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.4/test/support/base.rb:63]: Could not find 'form#new_domain' in the response
Looks like a timeout from here when showing the settings page: 2013-11-14 15:35:56.448 [ERROR] Unhandled exception reference #4cd319f5fa2f880c357f43bf3dfbced6: The thread #<Thread:0x00000007f76d20 dead> did not complete within 30 seconds. /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:35:in `block in join' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:31:in `map' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:31:in `join' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:44:in `join!' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/settings_controller.rb:13:in `show'
2013-11-14 15:35:56.217 [DEBUG] OpenShift API (58078.7ms) get https://localhost:443/broker/rest/user/keys.json [ code: 200 ] (pid :23442) 2013-11-14 15:35:56.435 [DEBUG] OpenShift API (58341.1ms) get https://localhost:443/broker/rest/user/authorizations.json [ code: 200 ] (pid:23442)
Site reports keys and authorizations taking 58 seconds for the broker to respond. On the broker side, the /keys and /authorizations calls appear to be blocked by an app create, which took 71 seconds. The broker log reports the /keys and /authorizations calls taking < 1 second each. Broker log: 2013-11-14 15:35:54.228 [DEBUG] DEBUG: MCollective Response Time (execute_direct: update-cluster): 9.327626258s (Request ID: 250f91a835f4c41de0015a530ee0dbde) (pid:19473) 2013-11-14 15:35:54.304 [DEBUG] Execute UpdateAppConfigOp (pid:19473) 2013-11-14 15:35:56.381 [DEBUG] DEBUG: Output of parallel execute: [{:tag=>{"op_id"=>"5285342a747321f7b300003c"}, :gear=>"528533e5747321f7b300001a", :job=>{:cartridge=>"openshift-origin-node", :action=>"authorized-ssh-ke y-add", :args=>{"--with-app-uuid"=>"528533e5747321f7b300001a", "--with-app-name"=>"scalable814122", "--with-container-uuid"=>"528533e5747321f7b300001a", "--with-container-name"=>"scalable814122", "--with-namespace"=>"d814122", "--with-req uest-id"=>"250f91a835f4c41de0015a530ee0dbde", "--with-ssh-key"=>"AAAAB3NzaC1yc2EAAAABIwAAAQEA9j5VbRgSr8oFXJPTeBl1m1gqdUihrkuzDrAl1IgMqYNPkfcGoxuYk9t2YQoCc+WyIZMSbVvg2IXxbXj7mBuuUWvzksewDcZBBdpufMWUTlfHahkxfECmSiENZMQjJIsPrXFRy3sbLXkW3DBMH R/EdF5jrjm991n3Pr9I5YoQywtj3XlCMuRyfYqcHYy7fSqRRtLO2jfVI0V3HT/W2/+y8DlKuGo7rc+yLchWEuTaOxHBEgZn0ccFQlYiW4+a9UM1I6GsjXc0ifOF2E4Z1UExR4jKF25tySigXWMKkkqwK8EqGmEzY24+u6vm2Fm5W1Rc+hlSRjIlpTYYKyuqiiVwzw==", "--with-ssh-key-type"=>"ssh-rsa", "- -with-ssh-key-comment"=>"application-528533e5747321f7b300001a"}}, :result_stdout=>"", :result_stderr=>"", :result_exit_code=>0, :result_addtl_params=>nil}, {:tag=>{"op_id"=>"5285342a747321f7b300003c"}, :gear=>"528533e5747321f7b300001a", : job=>{:cartridge=>"openshift-origin-node", :action=>"authorized-ssh-key-add", :args=>{"--with-app-uuid"=>"528533e5747321f7b300001a", "--with-app-name"=>"scalable814122", "--with-container-uuid"=>"528533e5747321f7b300001a", "--with-contain er-name"=>"scalable814122", "--with-namespace"=>"d814122", "--with-request-id"=>"250f91a835f4c41de0015a530ee0dbde", "--with-ssh-key"=>"AAAAB3NzaC1yc2EAAAADAQABAAABAQDcefMJ5FWEWEN+hScUi7a/Xkv7QMgTzqSBulwyGysYtqMvf8ddw2ZR9EMsm7qV3/EY6dWriUg 3pQT8BBPQKtydL7FCw6tipb14xq7uSm7Rfs/53DKmpSYPxT0j+f5KuMqyB+BgXsjoiYQEdCp/Aq/qu3lG1sfcDBEncnDysrWUWdemFU+SB8vzFsv17YPGOEGvy4slOdeV74tOLwiteqh3KYJtziXIK/Q6X4VRkWusGNQwDcxxPbZrMcsxpGFR6UuVi6cpulzpwijmSPQW6ZkH4t+b4CldsikdYiVixBAO7JZPgBPXfiUKR D8WFa+mLtJPM3ftpY9iFCBB/hptYIXH", "--with-ssh-key-type"=>"ssh-rsa", "--with-ssh-key-comment"=>"528533e0747321f7b3000014-default"}}, :result_stdout=>"", :result_stderr=>"", :result_exit_code=>0, :result_addtl_params=>nil}], exitcode: 0, fr om: ip-10-76-249-39 (Request ID: 250f91a835f4c41de0015a530ee0dbde) (pid:19473) 2013-11-14 15:35:56.451 [DEBUG] DEBUG: MCollective Response Time (execute_parallel): 1680ms (Request ID: 250f91a835f4c41de0015a530ee0dbde) (pid:19473) 2013-11-14 15:35:56.122 [DEBUG] SUCCESS ACTION=ADD_APPLICATION USER_ID=528533e0747321f7b3000014 LOGIN=test_user_814122 APP_UUID=528533e5747321f7b300001a DOMAIN=d814122 Application scalable814122 was created. (pid:19473) 2013-11-14 15:35:56.135 [INFO ] Completed 201 Created in 71408ms (Views: 10.1ms) (pid:19473) 2013-11-14 15:35:56.177 [INFO ] Started GET "/broker/rest/user/keys.json" for 127.0.0.1 at 2013-11-14 15:35:56 -0500 (pid:19473) 2013-11-14 15:35:56.186 [INFO ] Processing by KeysController#index as JSON (pid:19473) 2013-11-14 15:35:56.187 [DEBUG] API version 1.6 (pid:19473) 2013-11-14 15:35:56.203 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b IP=127.0.0.1 SCOPES=session Authenticated (pid:19473) 2013-11-14 15:35:56.204 [DEBUG] SUCCESS ACTION=LIST_KEY USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b DOMAIN= Found 0 ssh keys (pid:19473) 2013-11-14 15:35:56.206 [INFO ] Completed 200 OK in 19ms (Views: 0.5ms) (pid:19473) 2013-11-14 15:35:56.220 [INFO ] Started GET "/broker/rest/user/authorizations.json" for 127.0.0.1 at 2013-11-14 15:35:56 -0500 (pid:19473) 2013-11-14 15:35:56.222 [INFO ] Processing by AuthorizationsController#index as JSON (pid:19473) 2013-11-14 15:35:56.223 [DEBUG] API version 1.6 (pid:19473) 2013-11-14 15:35:56.238 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b IP=127.0.0.1 SCOPES=session Authenticated (pid:19473) 2013-11-14 15:35:56.248 [DEBUG] SUCCESS ACTION=LIST_AUTHORIZATION USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b DOMAIN= IP=127.0.0.1 List authorizations (pid:19473) 2013-11-14 15:35:56.251 [INFO ] Completed 200 OK in 28ms (Views: 0.9ms) (pid:19473)
Both errors seem to come from broker connections being tied up with long running app create tasks while the settings_flows_test is running: I can sometimes recreate the failure from my local machine by running both of these at the same time: cd ~/git/Openshift/rhc RHC_SERVER=... RHC_TARGET=rhel RHC_DEBUG=true TEST_INSECURE=1 TEST_RANDOM_USER=1 bundle exec rspec features/core_feature.rb -l 150 cd ~/git/Openshift/li/site bundle exec ruby test/integration/settings_flows_test.rb
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/72290f8e50f2e456f2af08ce5b68d87b95e9d04f Fix bug 1029678: move domain fetch out of async block
test_cartridge_scale_parameters_can_be_changed is the only test case I see failing consistently. 1029240 is open specifically for that test. *** This bug has been marked as a duplicate of bug 1029240 ***