Bug 977519

Summary: Broker fails to handle exceptions on malformed manifest.yml
Product: OpenShift Online Reporter: Andy Grimm <agrimm>
Component: PodAssignee: Rajat Chopra <rchopra>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: unspecified    
Version: 2.xCC: agrimm, jgoulding, jhou, sannam, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 983297 (view as bug list) Environment:
Last Closed: 2013-07-22 15:18:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 983297    
Attachments:
Description Flags
Example of a malformed YAML file
none
Broker log none

Description Andy Grimm 2013-06-24 19:05:27 UTC 
Description of problem:

When a user adds a downloadable cartridge with malformed YAML to an app, the broker returns an ISE on all subsequent calls. 

How reproducible:
Always

Steps to Reproduce:
1. rhc app create dlcart jbossas-7
2. rhc cartridge add https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml
3. Try to use the web console or any rhc commands.

Actual results:
rhc returns "The server did not respond correctly."
The web console shows "We appear to be having technical difficulties"

Expected results:
The broker should handle exceptions.  The cartridge addition should fail, or the user should be able to remove it to get back into a working state.
Comment 1 Andy Grimm 2013-06-24 19:06:40 UTC 
Created attachment 764756 [details]
Example of a malformed YAML file
Comment 2 Andy Grimm 2013-06-24 19:09:14 UTC 
Example traceback:

2013-06-24 14:17:13.888 [ERROR] Reference ID: 2c5401ab906d71cbe3e0f5219727ce2f - undefined method `each' for nil:NilClass
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.8.10/lib/openshift-origin-common/models/profile.rb:62:in `from_descriptor'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.8.10/lib/openshift-origin-common/models/cartridge.rb:142:in `from_descriptor'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:211:in `block (2 levels) in downloaded_cartridges'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/helpers/cartridge_cache.rb:156:in `block in foreach_cart_version'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/helpers/cartridge_cache.rb:148:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/helpers/cartridge_cache.rb:148:in `foreach_cart_version'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:210:in `block in downloaded_cartridges'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:208:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:208:in `downloaded_cartridges'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/models/application.rb:155:in `create_app'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/app/controllers/applications_controller.rb:98:in `create'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:481:in `block in _run__1059127462102703738__process_action__1866958233807456506__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_370'
  /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.17/lib/openshift/controller/action_log.rb:34:in `set_logged_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_370'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__1059127462102703738__process_action__1866958233807456506__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
  /opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongo_mapper-0.11.1/lib/mongo_mapper/middleware/identity_map.rb:10:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/unit_of_work.rb:39:in `unit_of_work'
  /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/rack/mongoid/middleware/identity_map.rb:34:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__4029681894463033093__call__228790197012769644__callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
  /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
  /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:517:in `accept_and_process_next_request'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:79:in `block in spawn_application'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/utils.rb:470:in `safe_fork'
  /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:64:in `spawn_application'
Comment 3 Abhishek Gupta 2013-06-26 17:18:22 UTC 
Either the fetch_community_carts method or the manifest initialization code (below) needs to be wrapped within begin-rescue blocks to handle exceptions.
Comment 4 Rajat Chopra 2013-06-26 22:48:26 UTC 
Fixed in stage with rev#a6cb4f474fde8fb8930f097eb6ed428b9783f468
Fixed in master with rev#4a7c640248f238d59e1304e526b3f92383fb7c3f
Comment 5 Jianwei Hou 2013-06-27 03:12:16 UTC 
I was not able to reproduce this issue. The provided cartridge can be added and removed properly with no exception, I think we are expecting a properly handled exception when the cartridge is malformed, looks like the broker think the provided cartridge is OK.

I have created a sample malformed cartridge and tested it, got same result, is this acceptable? https://raw.github.com/jhou1/downloadable-perl/master/metadata/manifest.yml

$ rhc cartridge add https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml -a dlcart
The cartridge 'https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml' will be downloaded and
installed
Adding https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml to application 'dlcart' ... Success

tok-irc-connector-1.0 (ircconn)
-------------------------------
  From:  https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml
  Gears: Located with jbossas-7

Added tok-irc-connector-1.0 to application dlcart

~$ rhc cartridge remove tok-irc-connector-1.0 -a dlcart
Removing a cartridge is a destructive operation that may result in loss of data associated with the cartridge.

Are you sure you wish to remove tok-irc-connector-1.0 from 'dlcart'? (yes|no): yes

Removing tok-irc-connector-1.0 from 'dlcart' ... removed
Comment 6 Jianwei Hou 2013-06-28 10:05:45 UTC 
Verified on STG

Steps: 
1. rhc app create dlcart jbossas-7
2. rhc cartridge add https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml -a dlcart

Result:
The cartridge 'https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml' will be downloaded and installed
Adding https://raw.github.com/Tok/OpenShift-IRC-cartridge/master/metadata/manifest.yml to application 'dlcart' ... Error in cartridge url - undefined method `each' for nil:NilClass
Comment 7 Jianwei Hou 2013-06-28 11:05:04 UTC 
After further dig, I have created a malformed cartridge and try to add it to a redhat cartridge. When the client reports exception, the malformed cartridge is created on gear, but not added in mongo. Below is the environment and reproduce steps

Test env:
STG, devenv-stage_382, devenv_3427

Steps:
1. Create a php app
rhc app create php1 php-5.3
2. Create a malformed cartridge and add it the the application
rhc cartridge add https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml -a php1 
3. ssh into gear to debug

Result:
After step 2:
The cartridge 'https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml' will be downloaded and installed
Adding https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml to application 'php1' ... can't convert nil into String

After step 3:
The cartridge is created on node with incomplete environment vairables(which is intended). 
Run app-show of the app, the cartridge is not present, so there is no way we can remove the disordered cartridge.
[php1-jhou6.stg.rhcloud.com 51cd6a17dbd93cc2cb00042e]\> ls
app-root  git  jhou-mysql  php
[php1-jhou6.stg.rhcloud.com 51cd6a17dbd93cc2cb00042e]\> env|grep MYSQL
OPENSHIFT_MYSQL_DIR=/var/lib/openshift/51cd6a17dbd93cc2cb00042e/jhou-mysql/
OPENSHIFT_MYSQL_DB_LOG_DIR.erb=<%= ENV['OPENSHIFT_MYSQL_DIR'] %>/log/
OPENSHIFT_MYSQL_IDENT=jhou:jhou-mysql:5.1:0.2.0
OPENSHIFT_MYSQL_DB_SOCKET.erb=<%= ENV['OPENSHIFT_MYSQL_DIR'] %>/socket/mysql.sock


rhc app-show php1
php1 @ http://php1-jhou6.stg.rhcloud.com/ (uuid: 51cd6a17dbd93cc2cb00042e)
--------------------------------------------------------------------------
  Created: 6:48 PM
  Gears:   1 (defaults to small)
  Git URL: ssh://51cd6a17dbd93cc2cb00042e.rhcloud.com/~/git/php1.git/
  SSH:     51cd6a17dbd93cc2cb00042e.rhcloud.com

  php-5.3 (PHP 5.3)
  -----------------
    Gears: 1 small
Comment 8 Jianwei Hou 2013-06-28 11:06:16 UTC 
Created attachment 766517 [details]
Broker log

Added development.log from broker
The malformed cartridge is https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml
Comment 9 Rajat Chopra 2013-06-28 17:18:35 UTC 
I doubt if all such cases can be handled. If it is a downloaded cartridge, all the broker can do is issue a 'configure' or 'deconfigure'. If the cartridge fails to do either of them properly (i.e. leave stuff behind in terms of files etc), there is little that the platform can do.

What is clearly handled is that the broker understood that there was an error and then cleans up mongo accordingly.

Keeping this bug open so that some better experience can be provided in certain error situations. Lowering the severity though.
Comment 10 Rajat Chopra 2013-07-11 21:36:44 UTC 
Fixed by Abhishek with pull request#https://github.com/openshift/origin-server/pull/3068
Comment 11 Jianwei Hou 2013-07-12 02:04:46 UTC 
Verified on devenv_3489

Follow comment 7 to add the malformed cartridge again, this cartridge will be rejected now:
1. Create a php app
rhc app create php1 php-5.3
2. Create a malformed cartridge and add it the the application
rhc cartridge add https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml -a php1 

The cartridge 'https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml' will be downloaded and
installed
Adding https://raw.github.com/jhou1/downloadable-malformed/master/metadata/manifest.yml to application 'php1' ... Unexpected error: can't convert nil into String

$ rhc app-show php1
php1 @ http://php1-jhou.dev.rhcloud.com/ (uuid: 51df61dba4cf8f357d00000f)
-------------------------------------------------------------------------
  Created: 9:54 AM
  Gears:   1 (defaults to small)
  Git URL: ssh://51df61dba4cf8f357d00000f.rhcloud.com/~/git/php1.git/
  SSH:     51df61dba4cf8f357d00000f.rhcloud.com

  php-5.3 (PHP 5.3)
  -----------------
    Gears: 1 small