Bug 981109

Summary: Creating apps in INT always fail with 504 error
Product: OpenShift Online Reporter: Zhe Wang <zhewang>
Component: ContainersAssignee: Krishna Raman <kraman>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: admiller, kraman, mfisher, xtian
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-22 15:20:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Zhe Wang 2013-07-04 03:38:13 UTC 
Description of problem:
It always fails to create an apps with either REST API or RHC in INT with 504 error.

Version-Release number of selected component (if applicable):
INT(devenv_3448)
rhc-1.11.2

How reproducible:
always

Steps to Reproduce:
1. create an app in INT with RHC
rhc app create phptst php-5.3

2. create an app in INT with REST API
curl -k --user zhewang+1:redhat https://int.openshift.redhat.com/broker/rest/domains/zwint/applications -d name=phptst3 -d cartridges=php-5.3  -XPOST | python -m json.tool

3. try other rhc commands and REST APIs, e.g.,

rhc cartridge list

curl -k --user zhewang+1:redhat https://int.openshift.redhat.com/broker/rest/domains/zwint/ XPOST | python -m json.tool

Actual results:
Creating apps in Step 1 and 2 fail with 504 error:

Creating application 'phptst' ... Server returned an unexpected error code: 504

However, INT server responds normally in Step 3.

Expected results:
Server should respond normally without any errors

Additional info:
Comment 1 Adam Miller 2013-07-05 13:15:25 UTC 
This is the last 100 lines from /var/log/mcollective.log on ex-sml-node1.int.rhcloud.com ... I suspect this to be part of the issue:


I, [2013-07-03T22:38:13.958915 #14420]  INFO -- : openshift.rb:151:in `execute_action' Executing action [app-state-show] using method oo_app_state_show with args [{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"e2e9213a4e6c7d7ab0230857377a0803"}]
I, [2013-07-03T22:38:13.959603 #14420]  INFO -- : openshift.rb:166:in `execute_action' Finished executing action [app-state-show] (0)
I, [2013-07-03T22:38:13.959758 #14420]  INFO -- : openshift.rb:198:in `execute_parallel_action' execute_parallel_action call - [{:tag=>"", :gear=>"51d3664a03ef64060600000f", :job=>{:cartridge=>"openshift-origin-node", :action=>"app-state-show", :args=>{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"e2e9213a4e6c7d7ab0230857377a0803"}}, :result_stdout=>"started", :result_stderr=>"", :result_exit_code=>0}]
I, [2013-07-03T22:39:14.016028 #14420]  INFO -- : openshift.rb:181:in `execute_parallel_action' execute_parallel_action call / action: execute_parallel, agent=openshift, data={"ex-sml-node1.int.rhcloud.com"=>
  [{:tag=>"",
    :gear=>"51d3664a03ef64060600000f",
    :job=>
     {:cartridge=>"openshift-origin-node",
      :action=>"app-state-show",
      :args=>
       {"--with-app-uuid"=>"51d3664a03ef64060600000f",
        "--with-app-name"=>"nagiosapp",
        "--with-container-uuid"=>"51d3664a03ef64060600000f",
        "--with-container-name"=>"nagiosapp",
        "--with-namespace"=>"nagiosmonitor",
        "--with-uid"=>4728,
        "--with-request-id"=>"0853a468524097ccf71c34595ea74ae4"}},
    :result_stdout=>"",
    :result_stderr=>"",
    :result_exit_code=>""}],
 :process_results=>true}

I, [2013-07-03T22:39:14.016511 #14420]  INFO -- : openshift.rb:151:in `execute_action' Executing action [app-state-show] using method oo_app_state_show with args [{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"0853a468524097ccf71c34595ea74ae4"}]
I, [2013-07-03T22:39:14.024835 #14420]  INFO -- : openshift.rb:166:in `execute_action' Finished executing action [app-state-show] (0)
I, [2013-07-03T22:39:14.025047 #14420]  INFO -- : openshift.rb:198:in `execute_parallel_action' execute_parallel_action call - [{:tag=>"", :gear=>"51d3664a03ef64060600000f", :job=>{:cartridge=>"openshift-origin-node", :action=>"app-state-show", :args=>{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"0853a468524097ccf71c34595ea74ae4"}}, :result_stdout=>"started", :result_stderr=>"", :result_exit_code=>0}]
I, [2013-07-03T22:39:18.235713 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:18.270397 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:18.270974 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:19.296047 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:19.331938 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:19.332381 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:20.250690 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:20.264132 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:20.264613 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:21.244428 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:21.276572 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:21.277039 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:22.217334 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:22.236455 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:22.236965 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:23.167040 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:23.193845 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:23.194669 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:24.117519 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:24.176152 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:24.176605 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:25.095928 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:25.111928 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:25.112364 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:26.075342 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:26.087851 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:26.107710 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:27.227803 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:27.246358 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:27.246802 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:28.187100 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:28.210078 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:28.214915 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:29.152668 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:29.279458 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:29.280038 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:30.228979 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:30.244945 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:30.245411 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:31.181024 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:31.194894 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:31.195343 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:32.201860 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:32.240827 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:32.241233 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:33.125948 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:33.152999 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:33.153418 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:34.222911 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:34.286761 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:34.287306 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:35.176785 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:35.196295 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:35.204034 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:36.206649 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:36.224306 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:36.231050 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:37.231090 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:37.269422 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:37.269882 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:38.328009 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:38.340238 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:38.340708 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
I, [2013-07-03T22:39:39.345382 #14420]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
W, [2013-07-03T22:39:39.366791 #14420]  WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information.
W, [2013-07-03T22:39:39.367228 #14420]  WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository
W, [2013-07-03T22:42:41.396296 #14420]  WARN -- : runner.rb:60:in `rescue in block in run' Exiting after signal: SIGTERM
I, [2013-07-03T22:42:41.927554 #28265]  INFO -- : mcollectived:31:in `<main>' The Marionette Collective 2.2.1 started logging at info level
E, [2013-07-03T22:42:42.405764 #28273] ERROR -- : pluginmanager.rb:171:in `rescue in loadclass' Failed to load MCollective::Agent::Openshift: cannot load such file -- openshift-origin-node/plugins/unix_user_observer
E, [2013-07-03T22:42:42.406001 #28273] ERROR -- : agents.rb:71:in `rescue in loadagent' Loading agent openshift failed: cannot load such file -- openshift-origin-node/plugins/unix_user_observer
I, [2013-07-03T22:48:19.725715 #28273]  INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal
E, [2013-07-03T22:48:19.742779 #28273] ERROR -- : pluginmanager.rb:171:in `rescue in loadclass' Failed to load MCollective::Agent::Openshift: cannot load such file -- openshift-origin-node/plugins/unix_user_observer
E, [2013-07-03T22:48:19.742971 #28273] ERROR -- : agents.rb:71:in `rescue in loadagent' Loading agent openshift failed: cannot load such file -- openshift-origin-node/plugins/unix_user_observer
Comment 2 Adam Miller 2013-07-05 16:50:40 UTC 
Issue appears to originate from https://github.com/openshift/li/pull/1682 commit id 448219f
Comment 3 Adam Miller 2013-07-09 13:15:30 UTC 
This issue is believed to be resolved in INT but is masked by the fact that INT is blocked on a cgroups issue now which is also causing the nodes to fail on app creations. This should be cleared up next successful INT deployment.
Comment 4 Zhe Wang 2013-07-10 01:17:57 UTC 
Verified in INT(devenv_3470)

Running Command - rhc app create snapshotbdyu jbossews-2.0 -l xtian+test3 -p 123456  --insecure --timeout 360 -s 
Application Options
-------------------
  Namespace:  7k3nxoifru
  Cartridges: jbossews-2.0
  Gear Size:  default
  Scaling:    yes

Creating application 'snapshotbdyu' ... done

  HAProxy instance is started

Waiting for your DNS name to be available ... done

Warning: Permanently added 'snapshotbdyu-7k3nxoifru.int.rhcloud.com,107.21.181.30' (RSA) to the list of known hosts.
Cloning into 'snapshotbdyu'...

Your application 'snapshotbdyu' is now available.

  URL:        http://snapshotbdyu-7k3nxoifru.int.rhcloud.com/
  SSH to:     51dcb5996cec0e553900022d.rhcloud.com
  Git remote: ssh://51dcb5996cec0e553900022d.rhcloud.com/~/git/snapshotbdyu.git/
  Cloned to:  /home/slave4/workspace/OpenShift_TestingMatrix/workspace/snapshotbdyu

Run 'rhc show-app snapshotbdyu' for more details about your app.