Bug 1195979 - Content Promotion with 200+ Puppet Environments Hangs or Fails (Old puppet environments aren't deleted)
Summary: Content Promotion with 200+ Puppet Environments Hangs or Fails (Old puppet en...
Keywords:
Status: CLOSED DUPLICATE of bug 1117376
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.0.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Katello Bug Bin
QA Contact: Katello QA List
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-24 22:50 UTC by Justin Sherrill
Modified: 2017-02-23 20:26 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-31 19:44:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 8932 0 None None None 2016-04-22 16:10:45 UTC

Description Justin Sherrill 2015-02-24 22:50:12 UTC
We have 32 content views and 4 environments, which has generated 223 Puppet Environments.  With this number of Puppet Environments promotions are hanging during the Dynflow finalization step "Content Update" and often failing with 406 errors.  If we resume the task (2-3 times), it will eventually finish.

The following Dynflow step is where we encounter the error:
<pre>
466: Actions::Katello::Foreman::ContentUpdate (error) [ 19.80s / 19.80s ]

Error:

ProxyAPI::ProxyException

ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42 ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://xxxxxxx.xxxxxxx.biz:9090/puppet

---
- /usr/share/foreman/lib/proxy_api/puppet.rb:27:in `rescue in classes'
- /usr/share/foreman/lib/proxy_api/puppet.rb:21:in `classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:155:in `actual_classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:78:in `new_classes_for'
- /usr/share/foreman/app/services/puppet_class_importer.rb:25:in `block in changes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `each'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `changes'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/services/katello/puppet_class_importer_extensions.rb:22:in
  `update_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/foreman.rb:37:in
  `update_puppet_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/katello/foreman/content_update.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:465:in
  `block (2 levels) in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:58:in
  `block (2 levels) in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:85:in
  `set_pulp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:44:in
  `as_pulp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:57:in
  `block in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:92:in
  `set_cp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:39:in
  `as_cp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:56:in
  `as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:30:in
  `with_progress_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:22:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:35:in
  `with_locale'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:464:in
  `block in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `block in with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:463:in
  `execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:230:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
  `block (2 levels) in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `with_meta_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
  `block in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
  `open_action'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:72:in
  `run_step'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:57:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `block in run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `all?'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:53:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:28:in
  `block (2 levels) in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:27:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/transaction_adapters/active_record.rb:5:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:24:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:23:in
  `block in on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in
  `block in assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `tap'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in
  `match_value'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in
  `block in match'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `match'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:17:in
  `on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:82:in
  `on_envelope'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:72:in
  `receive'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block (2 levels) in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `loop'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:13:in
  `block in initialize'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `block in create_with_logging_context'
</pre>


From foreman-proxy/proxy.log:
<pre>
10.60.141.42 - - [13/Jan/2015 12:51:42] "GET /environments/KT_xxxxxxx_prod_centos_6_server_x86_64_cv__16/classes HTTP/1.1" 200 2 2.4716
I, [2015-01-13T12:51:43.053304 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:43.146193 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.108623 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.141426 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.143053 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.250840 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.252718 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.414303 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
E, [2015-01-13T12:51:45.414820 #63199] ERROR -- : Failed to show puppet classes: Error converting value for param 'modulepath': Error converting value for param 'basemodulepath': Could not find value for $confdir
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42/classes HTTP/1.1" 406 162 2.3751
I, [2015-01-13T12:51:45.433601 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.522164 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_qa_rhel_5_8_server_x86_64_cv_69/classes HTTP/1.1" 200 2 2.7583
I, [2015-01-13T12:51:45.603873 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.702921 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.561855 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.662264 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.731477 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.748059 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.830031 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
</pre>

Comment 1 Justin Sherrill 2015-02-24 22:50:14 UTC
Created from redmine issue http://projects.theforeman.org/issues/8932

Comment 3 Bryan Kearney 2015-03-31 19:44:20 UTC

*** This bug has been marked as a duplicate of bug 1117376 ***

Comment 4 Justin Sherrill 2015-03-31 20:02:32 UTC
Note this is also partially duplicated by https://bugzilla.redhat.com/show_bug.cgi?id=1185838


Note You need to log in before you can comment on or make changes to this bug.