Bug 1029678 - Frequent error from site test case
Frequent error from site test case
Status: CLOSED DUPLICATE of bug 1029240
Product: OpenShift Online
Classification: Red Hat
Component: Website (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Jordan Liggitt
libra bugs
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-12 16:52 EST by Dan McPherson
Modified: 2015-05-14 21:31 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-16 15:26:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan McPherson 2013-11-12 16:52:57 EST
Description of problem:

1) Failure:
test_user_can_create_key(SettingsFlowsTest) [/root/openshift-test/site/test/integration/settings_flows_test.rb:34]:
Expected at least 1 element matching "td", found 0.


Ex:

https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5317/consoleFull


Almost always works on retries.
Comment 3 Dan McPherson 2013-11-13 13:25:00 EST
There were plenty of other failures in those runs.  Want to fix those instead :)
Comment 4 Jordan Liggitt 2013-11-13 13:39:17 EST
Hmmm, seeing this one intermittently:

test_user_can_create_domain(SettingsFlowsTest) [/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.4/test/support/base.rb:63]:
Could not find 'form#new_domain' in the response
Comment 5 Jordan Liggitt 2013-11-14 15:43:11 EST
Looks like a timeout from here when showing the settings page:

2013-11-14 15:35:56.448 [ERROR] Unhandled exception reference #4cd319f5fa2f880c357f43bf3dfbced6: The thread #<Thread:0x00000007f76d20 dead> did not complete 
within 30 seconds.
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:35:in `block in join'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:31:in `map'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:31:in `join'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/async_aware.rb:44:in `join!'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.5/app/controllers/settings_controller.rb:13:in `show'
Comment 6 Jordan Liggitt 2013-11-14 15:44:21 EST
2013-11-14 15:35:56.217 [DEBUG] OpenShift API (58078.7ms) get https://localhost:443/broker/rest/user/keys.json [ code: 200 ] (pid
:23442)
2013-11-14 15:35:56.435 [DEBUG] OpenShift API (58341.1ms) get https://localhost:443/broker/rest/user/authorizations.json [ code: 200
 ] (pid:23442)
Comment 7 Jordan Liggitt 2013-11-14 15:49:03 EST
Site reports keys and authorizations taking 58 seconds for the broker to respond.

On the broker side, the /keys and /authorizations calls appear to be blocked by an app create, which took 71 seconds. The broker log reports the /keys and /authorizations calls taking < 1 second each.


Broker log:
2013-11-14 15:35:54.228 [DEBUG] DEBUG: MCollective Response Time (execute_direct: update-cluster): 9.327626258s  (Request ID: 250f91a835f4c41de0015a530ee0dbde) (pid:19473)
2013-11-14 15:35:54.304 [DEBUG] Execute UpdateAppConfigOp (pid:19473)
2013-11-14 15:35:56.381 [DEBUG] DEBUG: Output of parallel execute: [{:tag=>{"op_id"=>"5285342a747321f7b300003c"}, :gear=>"528533e5747321f7b300001a", :job=>{:cartridge=>"openshift-origin-node", :action=>"authorized-ssh-ke
y-add", :args=>{"--with-app-uuid"=>"528533e5747321f7b300001a", "--with-app-name"=>"scalable814122", "--with-container-uuid"=>"528533e5747321f7b300001a", "--with-container-name"=>"scalable814122", "--with-namespace"=>"d814122", "--with-req
uest-id"=>"250f91a835f4c41de0015a530ee0dbde", "--with-ssh-key"=>"AAAAB3NzaC1yc2EAAAABIwAAAQEA9j5VbRgSr8oFXJPTeBl1m1gqdUihrkuzDrAl1IgMqYNPkfcGoxuYk9t2YQoCc+WyIZMSbVvg2IXxbXj7mBuuUWvzksewDcZBBdpufMWUTlfHahkxfECmSiENZMQjJIsPrXFRy3sbLXkW3DBMH
R/EdF5jrjm991n3Pr9I5YoQywtj3XlCMuRyfYqcHYy7fSqRRtLO2jfVI0V3HT/W2/+y8DlKuGo7rc+yLchWEuTaOxHBEgZn0ccFQlYiW4+a9UM1I6GsjXc0ifOF2E4Z1UExR4jKF25tySigXWMKkkqwK8EqGmEzY24+u6vm2Fm5W1Rc+hlSRjIlpTYYKyuqiiVwzw==", "--with-ssh-key-type"=>"ssh-rsa", "-
-with-ssh-key-comment"=>"application-528533e5747321f7b300001a"}}, :result_stdout=>"", :result_stderr=>"", :result_exit_code=>0, :result_addtl_params=>nil}, {:tag=>{"op_id"=>"5285342a747321f7b300003c"}, :gear=>"528533e5747321f7b300001a", :
job=>{:cartridge=>"openshift-origin-node", :action=>"authorized-ssh-key-add", :args=>{"--with-app-uuid"=>"528533e5747321f7b300001a", "--with-app-name"=>"scalable814122", "--with-container-uuid"=>"528533e5747321f7b300001a", "--with-contain
er-name"=>"scalable814122", "--with-namespace"=>"d814122", "--with-request-id"=>"250f91a835f4c41de0015a530ee0dbde", "--with-ssh-key"=>"AAAAB3NzaC1yc2EAAAADAQABAAABAQDcefMJ5FWEWEN+hScUi7a/Xkv7QMgTzqSBulwyGysYtqMvf8ddw2ZR9EMsm7qV3/EY6dWriUg
3pQT8BBPQKtydL7FCw6tipb14xq7uSm7Rfs/53DKmpSYPxT0j+f5KuMqyB+BgXsjoiYQEdCp/Aq/qu3lG1sfcDBEncnDysrWUWdemFU+SB8vzFsv17YPGOEGvy4slOdeV74tOLwiteqh3KYJtziXIK/Q6X4VRkWusGNQwDcxxPbZrMcsxpGFR6UuVi6cpulzpwijmSPQW6ZkH4t+b4CldsikdYiVixBAO7JZPgBPXfiUKR
D8WFa+mLtJPM3ftpY9iFCBB/hptYIXH", "--with-ssh-key-type"=>"ssh-rsa", "--with-ssh-key-comment"=>"528533e0747321f7b3000014-default"}}, :result_stdout=>"", :result_stderr=>"", :result_exit_code=>0, :result_addtl_params=>nil}], exitcode: 0, fr
om: ip-10-76-249-39  (Request ID: 250f91a835f4c41de0015a530ee0dbde) (pid:19473)
2013-11-14 15:35:56.451 [DEBUG] DEBUG: MCollective Response Time (execute_parallel): 1680ms  (Request ID: 250f91a835f4c41de0015a530ee0dbde) (pid:19473)
2013-11-14 15:35:56.122 [DEBUG] SUCCESS ACTION=ADD_APPLICATION USER_ID=528533e0747321f7b3000014 LOGIN=test_user_814122 APP_UUID=528533e5747321f7b300001a DOMAIN=d814122 Application scalable814122 was created. (pid:19473)
2013-11-14 15:35:56.135 [INFO ] Completed 201 Created in 71408ms (Views: 10.1ms) (pid:19473)
2013-11-14 15:35:56.177 [INFO ] Started GET "/broker/rest/user/keys.json" for 127.0.0.1 at 2013-11-14 15:35:56 -0500 (pid:19473)
2013-11-14 15:35:56.186 [INFO ] Processing by KeysController#index as JSON (pid:19473)
2013-11-14 15:35:56.187 [DEBUG] API version 1.6 (pid:19473)
2013-11-14 15:35:56.203 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b IP=127.0.0.1 SCOPES=session Authenticated (pid:19473)
2013-11-14 15:35:56.204 [DEBUG] SUCCESS ACTION=LIST_KEY USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b DOMAIN= Found 0 ssh keys (pid:19473)
2013-11-14 15:35:56.206 [INFO ] Completed 200 OK in 19ms (Views: 0.5ms) (pid:19473)
2013-11-14 15:35:56.220 [INFO ] Started GET "/broker/rest/user/authorizations.json" for 127.0.0.1 at 2013-11-14 15:35:56 -0500 (pid:19473)
2013-11-14 15:35:56.222 [INFO ] Processing by AuthorizationsController#index as JSON (pid:19473)
2013-11-14 15:35:56.223 [DEBUG] API version 1.6 (pid:19473)
2013-11-14 15:35:56.238 [DEBUG] SUCCESS ACTION=AUTHENTICATE USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b IP=127.0.0.1 SCOPES=session Authenticated (pid:19473)
2013-11-14 15:35:56.248 [DEBUG] SUCCESS ACTION=LIST_AUTHORIZATION USER_ID=528533f27473219d74000003 LOGIN=138446129788c46b DOMAIN= IP=127.0.0.1 List authorizations (pid:19473)
2013-11-14 15:35:56.251 [INFO ] Completed 200 OK in 28ms (Views: 0.9ms) (pid:19473)
Comment 8 Jordan Liggitt 2013-11-15 13:31:32 EST
Both errors seem to come from broker connections being tied up with long running app create tasks while the settings_flows_test is running:

I can sometimes recreate the failure from my local machine by running both of these at the same time:

cd ~/git/Openshift/rhc
RHC_SERVER=... RHC_TARGET=rhel RHC_DEBUG=true TEST_INSECURE=1 TEST_RANDOM_USER=1 bundle exec rspec features/core_feature.rb -l 150

cd ~/git/Openshift/li/site
bundle exec ruby test/integration/settings_flows_test.rb
Comment 9 openshift-github-bot 2013-12-18 10:34:32 EST
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/72290f8e50f2e456f2af08ce5b68d87b95e9d04f
Fix bug 1029678: move domain fetch out of async block
Comment 10 Jordan Liggitt 2014-01-16 15:26:38 EST
test_cartridge_scale_parameters_can_be_changed is the only test case I see failing consistently. 1029240 is open specifically for that test.

*** This bug has been marked as a duplicate of bug 1029240 ***

Note You need to log in before you can comment on or make changes to this bug.