Description of problem: It always fails to create an apps with either REST API or RHC in INT with 504 error. Version-Release number of selected component (if applicable): INT(devenv_3448) rhc-1.11.2 How reproducible: always Steps to Reproduce: 1. create an app in INT with RHC rhc app create phptst php-5.3 2. create an app in INT with REST API curl -k --user zhewang+1:redhat https://int.openshift.redhat.com/broker/rest/domains/zwint/applications -d name=phptst3 -d cartridges=php-5.3 -XPOST | python -m json.tool 3. try other rhc commands and REST APIs, e.g., rhc cartridge list curl -k --user zhewang+1:redhat https://int.openshift.redhat.com/broker/rest/domains/zwint/ XPOST | python -m json.tool Actual results: Creating apps in Step 1 and 2 fail with 504 error: Creating application 'phptst' ... Server returned an unexpected error code: 504 However, INT server responds normally in Step 3. Expected results: Server should respond normally without any errors Additional info:
This is the last 100 lines from /var/log/mcollective.log on ex-sml-node1.int.rhcloud.com ... I suspect this to be part of the issue: I, [2013-07-03T22:38:13.958915 #14420] INFO -- : openshift.rb:151:in `execute_action' Executing action [app-state-show] using method oo_app_state_show with args [{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"e2e9213a4e6c7d7ab0230857377a0803"}] I, [2013-07-03T22:38:13.959603 #14420] INFO -- : openshift.rb:166:in `execute_action' Finished executing action [app-state-show] (0) I, [2013-07-03T22:38:13.959758 #14420] INFO -- : openshift.rb:198:in `execute_parallel_action' execute_parallel_action call - [{:tag=>"", :gear=>"51d3664a03ef64060600000f", :job=>{:cartridge=>"openshift-origin-node", :action=>"app-state-show", :args=>{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"e2e9213a4e6c7d7ab0230857377a0803"}}, :result_stdout=>"started", :result_stderr=>"", :result_exit_code=>0}] I, [2013-07-03T22:39:14.016028 #14420] INFO -- : openshift.rb:181:in `execute_parallel_action' execute_parallel_action call / action: execute_parallel, agent=openshift, data={"ex-sml-node1.int.rhcloud.com"=> [{:tag=>"", :gear=>"51d3664a03ef64060600000f", :job=> {:cartridge=>"openshift-origin-node", :action=>"app-state-show", :args=> {"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"0853a468524097ccf71c34595ea74ae4"}}, :result_stdout=>"", :result_stderr=>"", :result_exit_code=>""}], :process_results=>true} I, [2013-07-03T22:39:14.016511 #14420] INFO -- : openshift.rb:151:in `execute_action' Executing action [app-state-show] using method oo_app_state_show with args [{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"0853a468524097ccf71c34595ea74ae4"}] I, [2013-07-03T22:39:14.024835 #14420] INFO -- : openshift.rb:166:in `execute_action' Finished executing action [app-state-show] (0) I, [2013-07-03T22:39:14.025047 #14420] INFO -- : openshift.rb:198:in `execute_parallel_action' execute_parallel_action call - [{:tag=>"", :gear=>"51d3664a03ef64060600000f", :job=>{:cartridge=>"openshift-origin-node", :action=>"app-state-show", :args=>{"--with-app-uuid"=>"51d3664a03ef64060600000f", "--with-app-name"=>"nagiosapp", "--with-container-uuid"=>"51d3664a03ef64060600000f", "--with-container-name"=>"nagiosapp", "--with-namespace"=>"nagiosmonitor", "--with-uid"=>4728, "--with-request-id"=>"0853a468524097ccf71c34595ea74ae4"}}, :result_stdout=>"started", :result_stderr=>"", :result_exit_code=>0}] I, [2013-07-03T22:39:18.235713 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:18.270397 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:18.270974 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:19.296047 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:19.331938 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:19.332381 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:20.250690 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:20.264132 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:20.264613 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:21.244428 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:21.276572 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:21.277039 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:22.217334 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:22.236455 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:22.236965 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:23.167040 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:23.193845 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:23.194669 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:24.117519 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:24.176152 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:24.176605 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:25.095928 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:25.111928 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:25.112364 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:26.075342 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:26.087851 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:26.107710 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:27.227803 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:27.246358 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:27.246802 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:28.187100 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:28.210078 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:28.214915 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:29.152668 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:29.279458 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:29.280038 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:30.228979 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:30.244945 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:30.245411 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:31.181024 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:31.194894 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:31.195343 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:32.201860 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:32.240827 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:32.241233 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:33.125948 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:33.152999 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:33.153418 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:34.222911 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:34.286761 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:34.287306 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:35.176785 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:35.196295 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:35.204034 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:36.206649 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:36.224306 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:36.231050 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:37.231090 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:37.269422 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:37.269882 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:38.328009 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:38.340238 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:38.340708 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository I, [2013-07-03T22:39:39.345382 #14420] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal W, [2013-07-03T22:39:39.366791 #14420] WARN -- : agent.rb:234:in `metadata' openshift.rb:23:in `<class:Openshift>': setting meta data in agents have been deprecated, DDL files are now being used for this information. W, [2013-07-03T22:39:39.367228 #14420] WARN -- : agents.rb:95:in `rescue in activate_agent?' Agent activation check for openshift failed: NameError: uninitialized constant OpenShift::Runtime::CartridgeRepository W, [2013-07-03T22:42:41.396296 #14420] WARN -- : runner.rb:60:in `rescue in block in run' Exiting after signal: SIGTERM I, [2013-07-03T22:42:41.927554 #28265] INFO -- : mcollectived:31:in `<main>' The Marionette Collective 2.2.1 started logging at info level E, [2013-07-03T22:42:42.405764 #28273] ERROR -- : pluginmanager.rb:171:in `rescue in loadclass' Failed to load MCollective::Agent::Openshift: cannot load such file -- openshift-origin-node/plugins/unix_user_observer E, [2013-07-03T22:42:42.406001 #28273] ERROR -- : agents.rb:71:in `rescue in loadagent' Loading agent openshift failed: cannot load such file -- openshift-origin-node/plugins/unix_user_observer I, [2013-07-03T22:48:19.725715 #28273] INFO -- : runner.rb:23:in `block in initialize' Reloading all agents after receiving USR1 signal E, [2013-07-03T22:48:19.742779 #28273] ERROR -- : pluginmanager.rb:171:in `rescue in loadclass' Failed to load MCollective::Agent::Openshift: cannot load such file -- openshift-origin-node/plugins/unix_user_observer E, [2013-07-03T22:48:19.742971 #28273] ERROR -- : agents.rb:71:in `rescue in loadagent' Loading agent openshift failed: cannot load such file -- openshift-origin-node/plugins/unix_user_observer
Issue appears to originate from https://github.com/openshift/li/pull/1682 commit id 448219f
This issue is believed to be resolved in INT but is masked by the fact that INT is blocked on a cgroups issue now which is also causing the nodes to fail on app creations. This should be cleared up next successful INT deployment.
Verified in INT(devenv_3470) Running Command - rhc app create snapshotbdyu jbossews-2.0 -l xtian+test3 -p 123456 --insecure --timeout 360 -s Application Options ------------------- Namespace: 7k3nxoifru Cartridges: jbossews-2.0 Gear Size: default Scaling: yes Creating application 'snapshotbdyu' ... done HAProxy instance is started Waiting for your DNS name to be available ... done Warning: Permanently added 'snapshotbdyu-7k3nxoifru.int.rhcloud.com,107.21.181.30' (RSA) to the list of known hosts. Cloning into 'snapshotbdyu'... Your application 'snapshotbdyu' is now available. URL: http://snapshotbdyu-7k3nxoifru.int.rhcloud.com/ SSH to: 51dcb5996cec0e553900022d.rhcloud.com Git remote: ssh://51dcb5996cec0e553900022d.rhcloud.com/~/git/snapshotbdyu.git/ Cloned to: /home/slave4/workspace/OpenShift_TestingMatrix/workspace/snapshotbdyu Run 'rhc show-app snapshotbdyu' for more details about your app.