Description of problem: When I run rhc create-app on my all-on-one OpenShift Origin installation, I get error message Creating application 'perl19988' ... Unable to complete the requested operation due to: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.. Reference ID: fd7ab2f1979184e8b5226be47c41984e The application can be listed via rhc apps but making HTTP request to its URL will fail with Name or service not known error. Version-Release number of selected component (if applicable): OpenShift Origin from today's nightly repo on Fedora 19. How reproducible: Deterministic, for various cartridges (tried perl and php). Steps to Reproduce: 1. Have OpenShift Origin installation in the example.com domain. 2. Run rhc setup and register the namespace test. 3. Run time rhc create-app -n test -a php2243 -t php-5.5 4. List applications with rhc apps 5. Try to make request to the application using curl http://php2243-test.example.com/ Actual results: # rhc create-app -n test -a php2243 -t php-5.5 Application Options ------------------- Namespace: test Cartridges: php-5.5 Gear Size: default Scaling: no Creating application 'php2243' ... Unable to complete the requested operation due to: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.. Reference ID: 25ccf605b309d8a9a12bb1f088493caa # rhc apps php2243 @ http://php2243-test.example.com/ (uuid: 51e3f8fb6892df85d600002e) --------------------------------------------------------------------------- Created: 9:28 AM Gears: 1 (defaults to small) Git URL: ssh://51e3f8fb6892df85d600002e.com/~/git/php2243.git/ SSH: 51e3f8fb6892df85d600002e.com php-5.5 (PHP 5.5) ----------------- Gears: 1 small # curl http://php2243-test.example.com/ curl: (6) Could not resolve host: php2243-test.example.com; Name or service not known # Expected results: No error for create-app and curl returning the boilerplate page of my new application. Additional info:
Created attachment 773763 [details] Hopefully the relevant part of /var/log/openshift/broker/development.log from the machine. I can attach other logs if you tell me which might be useful.
Fails with this error: uninitialized class variable @@container_plugin_class in OpenShift::Runtime::ApplicationContainer
Got the same error on Origin CI servers. It is related to https://github.com/openshift/origin-server/pull/3113/files I am working on a patch
https://github.com/openshift/origin-server/pull/3119
I also have the same problem when I deploy openshift-origin on fedora19 today. I will attach my log files for more detail.
Created attachment 787181 [details] log file /var/log/mcollective.log
Created attachment 787182 [details] log file /var/log/openshift/broker/development.log
openshift-tc service was being enabled. This has been fixed in puppet script.
Verified on latest origin build, the applications can be created with rhc app-create [root@broker ~]# service openshift-tc status Redirecting to /bin/systemctl status openshift-tc.service openshift-tc.service - OpenShift Traffic Control Loaded: loaded (/etc/systemd/system/openshift-tc.service; enabled) Active: inactive (dead) [root@broker ~]# rhc apps app1 @ http://app1-bmeng.example.com/ (uuid: 5285da626892df3f9b000007) ---------------------------------------------------------------------- Domain: bmeng Created: 4:25 PM Gears: 1 (defaults to small) Git URL: ssh://5285da626892df3f9b000007.com/~/git/app1.git/ SSH: 5285da626892df3f9b000007.com Deployment: auto (on git push) perl-5.16 (Perl 5.16) --------------------- Gears: 1 small
Moving back to ASSIGNED -- the issue is back: # rhc create-app -n test -a perl20724 -t perl-5.16 Application Options ------------------- Domain: test Cartridges: perl-5.16 Gear Size: default Scaling: no Creating application 'perl20724' ... Unable to complete the requested operation due to: An invalid exit code (129) was returned from the server broker.example.com. This indicates an unexpected problem during the execution of your request.. Reference ID: de100ffc00c8f8d5862036b85588ae06 2013-12-18 21:14:19.526 [INFO ] Parameters: {"name"=>"perl20724", "cartridges"=>["perl-5.16"], "domain_id"=>"test"} (pid:18346) 2013-12-18 21:14:25.495 [ERROR] An invalid exit code (129) was returned from the server broker.example.com. This indicates an unexpected problem during the execution of your request. (pid:18346) 2013-12-18 21:14:25.495 [ERROR] ["/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.18.0/lib/openshift/mcollective_application_container_proxy.rb:2727:in `parse_result'", "/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.18.0/lib/openshift/mcollective_application_container_proxy.rb:397:in `block in create'", "/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.18.0/lib/openshift/mcollective_application_container_proxy.rb:383:in `each'", "/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.18.0/lib/openshift/mcollective_application_container_proxy.rb:383:in `create'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/gear.rb:105:in `create_gear'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/pending_ops/create_gear_op.rb:9:in `execute'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/pending_app_op_group.rb:107:in `block in execute'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/pending_app_op_group.rb:96:in `each'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/pending_app_op_group.rb:96:in `execute'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/application.rb:1564:in `run_jobs'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/application.rb:684:in `block in add_features'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/application.rb:1630:in `run_in_application_lock'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/application.rb:666:in `add_features'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/models/application.rb:251:in `create_app'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/app/controllers/applications_controller.rb:157:in `create'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal/implicit_render.rb:4:in `send_action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/abstract_controller/base.rb:167:in `process_action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal/rendering.rb:10:in `process_action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/abstract_controller/callbacks.rb:18:in `block in process_action'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:481:in `block in _run__1504494359893083587__process_action__763507186306986740__callbacks'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_395'", "/usr/share/gems/gems/openshift-origin-controller-1.18.0/lib/openshift/controller/action_log.rb:80:in `set_logged_request'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_395'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:403:in `_run__1504494359893083587__process_action__763507186306986740__callbacks'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/usr/share/gems/gems/actionpack-3.2.13/lib/abstract_controller/callbacks.rb:17:in `process_action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal/rescue.rb:29:in `process_action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/notifications.rb:123:in `block in instrument'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/notifications/instrumenter.rb:20:in `instrument'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/notifications.rb:123:in `instrument'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal/instrumentation.rb:29:in `process_action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/abstract_controller/base.rb:121:in `process'", "/usr/share/gems/gems/actionpack-3.2.13/lib/abstract_controller/rendering.rb:45:in `process'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal.rb:203:in `dispatch'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_controller/metal.rb:246:in `block in action'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/routing/route_set.rb:73:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/routing/route_set.rb:36:in `call'", "/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'", "/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'", "/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/routing/route_set.rb:612:in `call'", "/usr/share/gems/gems/mongoid-3.1.4/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'", "/usr/share/gems/gems/mongoid-3.1.4/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'", "/usr/share/gems/gems/mongoid-3.1.4/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'", "/usr/share/gems/gems/sass-3.2.6/lib/sass/plugin/rack.rb:54:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'", "/usr/share/gems/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'", "/usr/share/gems/gems/rack-1.4.5/lib/rack/conditionalget.rb:35:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/head.rb:14:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/params_parser.rb:21:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/flash.rb:242:in `call'", "/usr/share/gems/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'", "/usr/share/gems/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/cookies.rb:341:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `_run__1281377470192371489__call__66849862331411855__callbacks'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:27:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'", "/usr/share/gems/gems/railties-3.2.13/lib/rails/rack/logger.rb:32:in `call_app'", "/usr/share/gems/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `block in call'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/tagged_logging.rb:22:in `tagged'", "/usr/share/gems/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/request_id.rb:22:in `call'", "/usr/share/gems/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'", "/usr/share/gems/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'", "/usr/share/gems/gems/activesupport-3.2.13/lib/active_support/cache/strategy/local_cache.rb:72:in `call'", "/usr/share/gems/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'", "/usr/share/gems/gems/actionpack-3.2.13/lib/action_dispatch/middleware/static.rb:63:in `call'", "/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'", "/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'", "/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'", "/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'", "/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'", "/usr/share/gems/gems/railties-3.2.13/lib/rails/engine.rb:479:in `call'", "/usr/share/gems/gems/railties-3.2.13/lib/rails/application.rb:223:in `call'", "/usr/share/gems/gems/railties-3.2.13/lib/rails/railtie/configurable.rb:30:in `method_missing'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/request_handler.rb:97:in `process_request'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:521:in `accept_and_process_next_request'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:171:in `block in handle_spawn_application'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/utils.rb:470:in `safe_fork'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:180:in `start'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/rack/application_spawner.rb:129:in `start'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in spawn_rack_application'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_application'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in synchronize'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/usr/share/gems/gems/passenger-3.0.21/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/usr/share/gems/gems/passenger-3.0.21/helper-scripts/passenger-spawn-server:102:in `<main>'"] (pid:18346) 2013-12-18 21:14:25.495 [ERROR] Encountered error during execute 'An invalid exit code (129) was returned from the server broker.example.com. This indicates an unexpected problem during the execution of your request.' (pid:18346) 2013-12-18 21:14:25.682 [INFO ] Completed 500 Internal Server Error in 6156ms (Views: 0.7ms) (pid:18346)
As for openshift-tc, I am not sure if it is supposed to be enabled or not. But attempt to start the service fails: [root@broker ~]# service openshift-tc start Redirecting to /bin/systemctl start openshift-tc.service Job for openshift-tc.service failed. See 'systemctl status openshift-tc.service' and 'journalctl -xn' for details. [root@broker ~]# systemctl status openshift-tc.service openshift-tc.service - OpenShift Traffic Control Loaded: loaded (/etc/systemd/system/openshift-tc.service; enabled) Active: failed (Result: exit-code) since Wed 2013-12-18 21:16:33 EST; 9s ago Process: 32305 ExecStart=/usr/sbin/oo-admin-ctl-tc start (code=exited, status=1/FAILURE) Dec 18 21:16:33 broker.example.com oo-admin-ctl-tc[32305]: /usr/share/gems/gems/openshift-origin-node-1.18.0/lib/openshift-origin-node/utils/tc.rb:103:in `get_interf...imeError) Dec 18 21:16:33 broker.example.com oo-admin-ctl-tc[32305]: from /usr/share/gems/gems/openshift-origin-node-1.18.0/lib/openshift-origin-node/utils/tc.rb:77:in `initialize' Dec 18 21:16:33 broker.example.com oo-admin-ctl-tc[32305]: from /usr/sbin/oo-admin-ctl-tc:24:in `new' Dec 18 21:16:33 broker.example.com oo-admin-ctl-tc[32305]: from /usr/sbin/oo-admin-ctl-tc:24:in `<main>' Dec 18 21:16:33 broker.example.com systemd[1]: openshift-tc.service: control process exited, code=exited status=1 Dec 18 21:16:33 broker.example.com systemd[1]: Failed to start OpenShift Traffic Control. Dec 18 21:16:33 broker.example.com systemd[1]: Unit openshift-tc.service entered failed state.
This is on Fedora 19 with Origin installed yesterday.
On system where rhc create-app failed, oo-accept-node reported FAIL: Could not get SELinux context for mcollective FAIL: Could not get SELinux context for oddjobd FAIL: tc htb qdisc not configured 3 ERRORS When I did service openshift-tc start and it passed, the oo-accept-node output changed to FAIL: Could not get SELinux context for mcollective FAIL: Could not get SELinux context for oddjobd 2 ERRORS and rhc create-app now passes there. That service status is now # service openshift-tc status | cat Redirecting to /bin/systemctl status openshift-tc.service openshift-tc.service - OpenShift Traffic Control Loaded: loaded (/etc/systemd/system/openshift-tc.service; enabled) Active: active (exited) since Thu 2013-12-19 00:58:44 EST; 3min 8s ago Process: 28427 ExecStart=/usr/sbin/oo-admin-ctl-tc start (code=exited, status=0/SUCCESS) Dec 19 00:58:44 broker.example.com systemd[1]: Starting OpenShift Traffic Control... Dec 19 00:58:44 broker.example.com oo-admin-ctl-tc[28427]: Starting bandwidth shaping: [OK] Dec 19 00:58:44 broker.example.com systemd[1]: Started OpenShift Traffic Control. However, on the other Fedora 19 machine where things were passing from the very beginning, I see [root@broker ~]# oo-accept-node FAIL: Could not get SELinux context for mcollective FAIL: Could not get SELinux context for oddjobd 2 ERRORS [root@broker ~]# service openshift-tc status Redirecting to /bin/systemctl status openshift-tc.service openshift-tc.service - OpenShift Traffic Control Loaded: loaded (/etc/systemd/system/openshift-tc.service; enabled) Active: inactive (dead) So is openshift-tc needed and should I run it or does it play any role here?
One thing different between those two systems is that the one where things passed has eth0, the other one does not. So I thought that it might be an instance of bug 1009257. But on the failing one, I have # grep conf_node_external_eth_dev configure_origin.pp conf_node_external_eth_dev => 'em1', so it might not be the case. Unless of course the handling of the conf_node_external_eth_dev parameter is not completely correct. Could QE please try to test on a system which does *not* have eth0 network interface to confirm/rule out this theory?
openshift-tc is needed but it is a one per boot setup script and exits after doing it setup. That is why you see that it is enabled by exited. Puppet script should enable it during setup. If it doesnt then that is a bug. You can ignore: (Luke is fixing this) FAIL: Could not get SELinux context for mcollective FAIL: Could not get SELinux context for oddjobd
(In reply to Krishna Raman from comment #18) > openshift-tc is needed but it is a one per boot setup script and exits after > doing it setup. That is why you see that it is enabled by exited. But on the other machine, I see it running, if I start it manually. Isn't that strange?
We apologize, however, we do not plan to address this report at this time. The majority of our active development is for the v3 version of OpenShift. If you would like for Red Hat to reconsider this decision, please reach out to your support representative. We are very sorry for any inconvenience this may cause.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days