Bug 1195979

Summary:	Content Promotion with 200+ Puppet Environments Hangs or Fails (Old puppet environments aren't deleted)
Product:	Red Hat Satellite	Reporter:	Justin Sherrill <jsherril>
Component:	Content Management	Assignee:	Katello Bug Bin <katello-bugs>
Status:	CLOSED DUPLICATE	QA Contact:	Katello QA List <katello-qa-list>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.0.4	CC:	bbuckingham, bkearney, cwelton
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
URL:	http://projects.theforeman.org/issues/8932
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-03-31 19:44:20 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Justin Sherrill 2015-02-24 22:50:12 UTC

We have 32 content views and 4 environments, which has generated 223 Puppet Environments.  With this number of Puppet Environments promotions are hanging during the Dynflow finalization step "Content Update" and often failing with 406 errors.  If we resume the task (2-3 times), it will eventually finish.

The following Dynflow step is where we encounter the error:
<pre>
466: Actions::Katello::Foreman::ContentUpdate (error) [ 19.80s / 19.80s ]

Error:

ProxyAPI::ProxyException

ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42 ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://xxxxxxx.xxxxxxx.biz:9090/puppet

---
- /usr/share/foreman/lib/proxy_api/puppet.rb:27:in `rescue in classes'
- /usr/share/foreman/lib/proxy_api/puppet.rb:21:in `classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:155:in `actual_classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:78:in `new_classes_for'
- /usr/share/foreman/app/services/puppet_class_importer.rb:25:in `block in changes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `each'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `changes'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/services/katello/puppet_class_importer_extensions.rb:22:in
  `update_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/foreman.rb:37:in
  `update_puppet_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/katello/foreman/content_update.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:465:in
  `block (2 levels) in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:58:in
  `block (2 levels) in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:85:in
  `set_pulp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:44:in
  `as_pulp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:57:in
  `block in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:92:in
  `set_cp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:39:in
  `as_cp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:56:in
  `as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:30:in
  `with_progress_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:22:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:35:in
  `with_locale'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:464:in
  `block in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `block in with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:463:in
  `execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:230:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
  `block (2 levels) in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `with_meta_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
  `block in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
  `open_action'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:72:in
  `run_step'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:57:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `block in run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `all?'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:53:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:28:in
  `block (2 levels) in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:27:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/transaction_adapters/active_record.rb:5:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:24:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:23:in
  `block in on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in
  `block in assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `tap'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in
  `match_value'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in
  `block in match'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `match'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:17:in
  `on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:82:in
  `on_envelope'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:72:in
  `receive'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block (2 levels) in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `loop'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:13:in
  `block in initialize'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `block in create_with_logging_context'
</pre>


From foreman-proxy/proxy.log:
<pre>
10.60.141.42 - - [13/Jan/2015 12:51:42] "GET /environments/KT_xxxxxxx_prod_centos_6_server_x86_64_cv__16/classes HTTP/1.1" 200 2 2.4716
I, [2015-01-13T12:51:43.053304 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:43.146193 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.108623 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.141426 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.143053 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.250840 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.252718 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.414303 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
E, [2015-01-13T12:51:45.414820 #63199] ERROR -- : Failed to show puppet classes: Error converting value for param 'modulepath': Error converting value for param 'basemodulepath': Could not find value for $confdir
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42/classes HTTP/1.1" 406 162 2.3751
I, [2015-01-13T12:51:45.433601 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.522164 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_qa_rhel_5_8_server_x86_64_cv_69/classes HTTP/1.1" 200 2 2.7583
I, [2015-01-13T12:51:45.603873 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.702921 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.561855 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.662264 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.731477 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.748059 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.830031 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
</pre>

Comment 1 Justin Sherrill 2015-02-24 22:50:14 UTC

Created from redmine issue http://projects.theforeman.org/issues/8932

Comment 3 Bryan Kearney 2015-03-31 19:44:20 UTC


*** This bug has been marked as a duplicate of bug 1117376 ***

Comment 4 Justin Sherrill 2015-03-31 20:02:32 UTC

Note this is also partially duplicated by https://bugzilla.redhat.com/show_bug.cgi?id=1185838