Bug 1029240

Summary: Frequent errors from site test cases
Product: OpenShift Online Reporter: Dan McPherson <dmcphers>
Component: WebsiteAssignee: Fabiano Franz <ffranz>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.xCC: agoldste, yujzhang
Target Milestone: ---Keywords: UpcomingRelease
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-26 19:08:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dan McPherson 2013-11-12 00:05:08 UTC 
From:

https://ci.dev.openshift.redhat.com/jenkins/job/test_pull_requests/5259/console

  1) Error:
test_cartridge_scale_parameters_can_be_changed(RestApiCartridgeTest):
RestApi::ServerUnavailable: Failed.  Response code = 503.  Response message = .
    /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.2/app/models/rest_api/base.rb:353:in `rescue in save_with_exception_handling'
    /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.2/app/models/rest_api/base.rb:335:in `save_with_exception_handling'
    /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.2/app/models/rest_api/base.rb:362:in `save_with_change_tracking'
    /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.17.2/test/integration/rest_api/cartridge_test.rb:47:in `block in <class:RestApiCartridgeTest>'

18 tests, 170 assertions, 0 failures, 1 errors, 0 skips
Coverage report generated for /opt/rh/ruby193/root/usr/share/gems/gems/rake-0.9.2.2/lib/rake/rake_test_loader.rb test/functional/cartridge_types_controller_test.rb /opt/rh/ruby193/root/usr/share/gems/gems/ci_reporter-1.7.2/lib/ci/reporter/rake/minitest_loader.rb, /opt/rh/ruby193/root/usr/share/gems/gems/rake-0.9.2.2/lib/rake/rake_test_loader.rb test/functional/cartridges_controller_test.rb /opt/rh/ruby193/root/usr/share/gems/gems/ci_reporter-1.7.2/lib/ci/reporter/rake/minitest_loader.rb, /opt/rh/ruby193/root/usr/share/gems/gems/rake-0.9.2.2/lib/rake/rake_test_loader.rb test/integration/rest_api/membership_test.rb test/integration/rest_api/cartridge_test.rb test/integration/rest_api/cartridge_type_test.rb /opt/rh/ruby193/root/usr/share/gems/gems/ci_reporter-1.7.2/lib/ci/reporter/rake/minitest_loader.rb, /opt/rh/ruby193/root/usr/share/gems/gems/rake-0.9.2.2/lib/rake/rake_test_loader.rb test/integration/status_app_test.rb test/integration/rescue_from_test.rb test/integration/settings_flows_test.rb test/integration/singular_resources_test.rb test/integration/quickstarts_test.rb test/integration/assets_test.rb test/integration/static_pages_test.rb test/unit/helpers/home_helper_test.rb test/unit/helpers/model_helper_test.rb test/unit/helpers/country_helper_test.rb test/unit/helpers/application_helper_test.rb test/unit/helpers/account_helper_test.rb test/unit/helpers/express_domain_helper_test.rb test/unit/helpers/product_helper_test.rb test/unit/community_api_test.rb test/unit/async_aware_test.rb test/unit/download_test.rb test/unit/configuration_test.rb test/unit/aria_test.rb test/unit/log_subscriber_test.rb test/unit/tweet_test.rb test/unit/rest_api_test.rb test/unit/cost_aware_test.rb test/unit/streamline_test.rb test/unit/capability_aware_test.rb test/unit/bootstrap_form_builder_test.rb test/unit/overrides_test.rb test/unit/active_model_compliance_test.rb test/unit/web_user_test.rb test/unit/cloud_access_test.rb test/unit/faq_item_test.rb test/unit/filter_hash_test.rb /opt/rh/ruby193/root/usr/share/gems/gems/ci_reporter-1.7.2/lib/ci/reporter/rake/minitest_loader.rb to /root/openshift-test/site/test/coverage/. 5191 / 7562 LOC (68.65%) covered.
rake aborted!
Command failed with status (1): [/opt/rh/ruby193/root/usr/bin/ruby -I"lib:t...]



How reproducible:

60%


And 99% of the time this works fine on retry.  Decent chance this is a problem below the site.  If so please provide the context of the issue.
Comment 1 Andy Goldstein 2013-11-12 17:48:00 UTC 
app uuid 5280fda8873c793f3c000094

At 10:55:58, it looks like a scale up to 3 was started (PUT "/broker/rest/application/5280fda8873c793f3c000094/cartridge/ruby-1.8.json) (pid 5632).

At 10:57:19, update-cluster was invoked via MCollective (pid 5632).

At 10:57:20, another request came in (also PUT "/broker/rest/application/5280fda8873c793f3c000094/cartridge/ruby-1.8.json) (pid 28292) setting the scale min/max to 2

At 10:58:10, the 2nd scale request failed - Unable to perform action. Another operation is already running. (pid:28292)

At 10:58:37, update-cluster for pid 5632 finished
Comment 2 Jordan Liggitt 2014-01-16 20:26:38 UTC 
*** Bug 1029678 has been marked as a duplicate of this bug. ***
Comment 3 Fabiano Franz 2014-01-23 16:41:20 UTC 
Fixed in https://github.com/openshift/li/pull/2300
Comment 4 openshift-github-bot 2014-01-23 18:27:55 UTC 
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/6f7208d3448261bf741fcc5a7fb7833c74ed3b0f
Bug 1029240 - moved REST API cartridge test to avoid race conditions when scaling with_scalable_app in different suites
Comment 5 Yujie Zhang 2014-01-24 05:24:28 UTC 
Tested on devenv_4269 and ran the cartridge_test.rb, pass without error, result is as following:

[root@ip-10-138-38-81 rest_api]# ruby cartridge_test.rb 

*** Mocha deprecation warning: Change `require 'mocha'` to `require 'mocha/setup'`.

Run options: --seed 14072

# Running tests:

..

Finished tests in 147.932346s, 0.0135 tests/s, 0.1082 assertions/s.

2 tests, 16 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for cartridge_test.rb to /var/www/openshift/site/test/integration/rest_api/test/coverage/. 3346 / 7651 LOC (43.73%) covered.

So verify this bug, thanks.