Bug 977519 - Broker fails to handle exceptions on malformed manifest.yml
Broker fails to handle exceptions on malformed manifest.yml
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
:
Depends On:
Blocks: 983297
  Show dependency treegraph
 
Reported: 2013-06-24 15:05 EDT by Andy Grimm
Modified: 2016-11-07 22:47 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 983297 (view as bug list)
Environment:
Last Closed: 2013-07-22 11:18:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Example of a malformed YAML file (801 bytes, application/x-yaml)
2013-06-24 15:06 EDT, Andy Grimm
no flags Details
Broker log (56.13 KB, text/x-log)
2013-06-28 07:06 EDT, Jianwei Hou
no flags Details

  None (edit)
Description Andy Grimm 2013-06-24 15:05:27 EDT
Description of problem:

When a user adds a downloadable cartridge with malformed YAML to an app, the broker returns an ISE on all subsequent calls. 

How reproducible:
Always

Steps to Reproduce:
1. rhc app create dlcart jbossas-7
2. rhc cartridge add https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml
3. Try to use the web console or any rhc commands.

Actual results:
rhc returns "The server did not respond correctly."
The web console shows "We appear to be having technical difficulties"

Expected results:
The broker should handle exceptions.  The cartridge addition should fail, or the user should be able to remove it to get back into a working state.
Comment 1 Andy Grimm 2013-06-24 15:06:40 EDT
Created attachment 764756 [details]
Example of a malformed YAML file
Comment 2 Andy Grimm 2013-06-24 15:09:14 EDT
Example traceback:

2013-06-24 14:17:13.888 [ERROR] Reference ID: 2c5401ab906d71cbe3e0f5219727ce2f - undefined method `each' for nil:NilClass
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.8.10/lib/openshift-origin-common/models/profile.rb:62:in `from_descriptor'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.8.10/lib/openshift-origin-common/models/cartridge.rb:142:in `from_descriptor'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:211:in `block (2 levels) in downloaded_cartridges'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/helpers/cartridge_cache.rb:156:in `block in foreach_cart_version'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/helpers/cartridge_cache.rb:148:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/helpers/cartridge_cache.rb:148:in `foreach_cart_version'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:210:in `block in downloaded_cartridges'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:208:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:208:in `downloaded_cartridges'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:155:in `create_app'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/controllers/applications_controller.rb:98:in `create'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:481:in `block in _run__1059127462102703738__process_action__1866958233807456506__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_370'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/lib/openshift/controller/action_log.rb:34:in `set_logged_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_370'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__1059127462102703738__process_action__1866958233807456506__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__4029681894463033093__call__228790197012769644__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:517:in `accept_and_process_next_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/utils.rb:470:in `safe_fork'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'
Comment 3 Abhishek Gupta 2013-06-26 13:18:22 EDT
Either the fetch_community_carts method or the manifest initialization code (below) needs to be wrapped within begin-rescue blocks to handle exceptions.
Comment 4 Rajat Chopra 2013-06-26 18:48:26 EDT
Fixed in stage with rev#a6cb4f474fde8fb8930f097eb6ed428b9783f468
Fixed in master with rev#4a7c640248f238d59e1304e526b3f92383fb7c3f
Comment 5 Jianwei Hou 2013-06-26 23:12:16 EDT
I was not able to reproduce this issue. The provided cartridge can be added and removed properly with no exception, I think we are expecting a properly handled exception when the cartridge is malformed, looks like the broker think the provided cartridge is OK.

I have created a sample malformed cartridge and tested it, got same result, is this acceptable? https://raw.github.com/jhou1/downloadable-perl/master/metadata/manifest.yml

$ rhc cartridge add https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml -a dlcart
The cartridge 'https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml' will be downloaded and
installed
Adding https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml to application 'dlcart' ... Success

tok-irc-connector-1.0 (ircconn)
-------------------------------
  From:  https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml
  Gears: Located with jbossas-7

Added tok-irc-connector-1.0 to application dlcart

~$ rhc cartridge remove tok-irc-connector-1.0 -a dlcart
Removing a cartridge is a destructive operation that may result in loss of data associated with the cartridge.

Are you sure you wish to remove tok-irc-connector-1.0 from 'dlcart'? (yes|no): yes

Removing tok-irc-connector-1.0 from 'dlcart' ... removed
Comment 6 Jianwei Hou 2013-06-28 06:05:45 EDT
Verified on STG

Steps: 
1. rhc app create dlcart jbossas-7
2. rhc cartridge add https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml -a dlcart

Result:
The cartridge 'https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml' will be downloaded and installed
Adding https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml to application 'dlcart' ... Error in cartridge url - undefined method `each' for nil:NilClass
Comment 7 Jianwei Hou 2013-06-28 07:05:04 EDT
After further dig, I have created a malformed cartridge and try to add it to a redhat cartridge. When the client reports exception, the malformed cartridge is created on gear, but not added in mongo. Below is the environment and reproduce steps

Test env:
STG, devenv-stage_382, devenv_3427

Steps:
1. Create a php app
rhc app create php1 php-5.3
2. Create a malformed cartridge and add it the the application
rhc cartridge add https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml -a php1 
3. ssh into gear to debug

Result:
After step 2:
The cartridge 'https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml' will be downloaded and installed
Adding https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml to application 'php1' ... can't convert nil into String

After step 3:
The cartridge is created on node with incomplete environment vairables(which is intended). 
Run app-show of the app, the cartridge is not present, so there is no way we can remove the disordered cartridge.
[php1-jhou6.stg.rhcloud.com 51cd6a17dbd93cc2cb00042e]\> ls
app-root  git  jhou-mysql  php
[php1-jhou6.stg.rhcloud.com 51cd6a17dbd93cc2cb00042e]\> env|grep MYSQL
OPENSHIFT_MYSQL_DIR=/var/lib/openshift/51cd6a17dbd93cc2cb00042e/jhou-mysql/
OPENSHIFT_MYSQL_DB_LOG_DIR.erb=<%= ENV['OPENSHIFT_MYSQL_DIR'] %>/log/
OPENSHIFT_MYSQL_IDENT=jhou:jhou-mysql:5.1:0.2.0
OPENSHIFT_MYSQL_DB_SOCKET.erb=<%= ENV['OPENSHIFT_MYSQL_DIR'] %>/socket/mysql.sock


rhc app-show php1
php1 @ http://php1-jhou6.stg.rhcloud.com/ (uuid: 51cd6a17dbd93cc2cb00042e)
--------------------------------------------------------------------------
  Created: 6:48 PM
  Gears:   1 (defaults to small)
  Git URL: ssh://51cd6a17dbd93cc2cb00042e@php1-jhou6.stg.rhcloud.com/~/git/php1.git/
  SSH:     51cd6a17dbd93cc2cb00042e@php1-jhou6.stg.rhcloud.com

  php-5.3 (PHP 5.3)
  -----------------
    Gears: 1 small
Comment 8 Jianwei Hou 2013-06-28 07:06:16 EDT
Created attachment 766517 [details]
Broker log

Added development.log from broker
The malformed cartridge is https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml
Comment 9 Rajat Chopra 2013-06-28 13:18:35 EDT
I doubt if all such cases can be handled. If it is a downloaded cartridge, all the broker can do is issue a 'configure' or 'deconfigure'. If the cartridge fails to do either of them properly (i.e. leave stuff behind in terms of files etc), there is little that the platform can do.

What is clearly handled is that the broker understood that there was an error and then cleans up mongo accordingly.

Keeping this bug open so that some better experience can be provided in certain error situations. Lowering the severity though.
Comment 10 Rajat Chopra 2013-07-11 17:36:44 EDT
Fixed by Abhishek with pull request#https://github.com/openshift/origin-server/pull/3068
Comment 11 Jianwei Hou 2013-07-11 22:04:46 EDT
Verified on devenv_3489

Follow comment 7 to add the malformed cartridge again, this cartridge will be rejected now:
1. Create a php app
rhc app create php1 php-5.3
2. Create a malformed cartridge and add it the the application
rhc cartridge add https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml -a php1 

The cartridge 'https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml' will be downloaded and
installed
Adding https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml to application 'php1' ... Unexpected error: can't convert nil into String

$ rhc app-show php1
php1 @ http://php1-jhou.dev.rhcloud.com/ (uuid: 51df61dba4cf8f357d00000f)
-------------------------------------------------------------------------
  Created: 9:54 AM
  Gears:   1 (defaults to small)
  Git URL: ssh://51df61dba4cf8f357d00000f@php1-jhou.dev.rhcloud.com/~/git/php1.git/
  SSH:     51df61dba4cf8f357d00000f@php1-jhou.dev.rhcloud.com

  php-5.3 (PHP 5.3)
  -----------------
    Gears: 1 small

Note You need to log in before you can comment on or make changes to this bug.