Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1195979 - Content Promotion with 200+ Puppet Environments Hangs or Fails (Old puppet environments aren't deleted)
Summary: Content Promotion with 200+ Puppet Environments Hangs or Fails (Old puppet en...
Keywords:
Status: CLOSED DUPLICATE of bug 1117376
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.0.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Katello Bug Bin
QA Contact: Katello QA List
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-24 22:50 UTC by Justin Sherrill
Modified: 2017-02-23 20:26 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-31 19:44:20 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 8932 0 None None None 2016-04-22 16:10:45 UTC

Description Justin Sherrill 2015-02-24 22:50:12 UTC 
We have 32 content views and 4 environments, which has generated 223 Puppet Environments.  With this number of Puppet Environments promotions are hanging during the Dynflow finalization step "Content Update" and often failing with 406 errors.  If we resume the task (2-3 times), it will eventually finish.

The following Dynflow step is where we encounter the error:
<pre>
466: Actions::Katello::Foreman::ContentUpdate (error) [ 19.80s / 19.80s ]

Error:

ProxyAPI::ProxyException

ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42 ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://xxxxxxx.xxxxxxx.biz:9090/puppet

---
- /usr/share/foreman/lib/proxy_api/puppet.rb:27:in `rescue in classes'
- /usr/share/foreman/lib/proxy_api/puppet.rb:21:in `classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:155:in `actual_classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:78:in `new_classes_for'
- /usr/share/foreman/app/services/puppet_class_importer.rb:25:in `block in changes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `each'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `changes'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/services/katello/puppet_class_importer_extensions.rb:22:in
  `update_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/foreman.rb:37:in
  `update_puppet_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/katello/foreman/content_update.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:465:in
  `block (2 levels) in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:58:in
  `block (2 levels) in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:85:in
  `set_pulp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:44:in
  `as_pulp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:57:in
  `block in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:92:in
  `set_cp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:39:in
  `as_cp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:56:in
  `as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:30:in
  `with_progress_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:22:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:35:in
  `with_locale'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:464:in
  `block in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `block in with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:463:in
  `execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:230:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
  `block (2 levels) in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `with_meta_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
  `block in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
  `open_action'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:72:in
  `run_step'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:57:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `block in run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `all?'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:53:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:28:in
  `block (2 levels) in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:27:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/transaction_adapters/active_record.rb:5:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:24:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:23:in
  `block in on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in
  `block in assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `tap'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in
  `match_value'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in
  `block in match'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `match'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:17:in
  `on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:82:in
  `on_envelope'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:72:in
  `receive'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block (2 levels) in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `loop'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:13:in
  `block in initialize'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `block in create_with_logging_context'
</pre>


From foreman-proxy/proxy.log:
<pre>
10.60.141.42 - - [13/Jan/2015 12:51:42] "GET /environments/KT_xxxxxxx_prod_centos_6_server_x86_64_cv__16/classes HTTP/1.1" 200 2 2.4716
I, [2015-01-13T12:51:43.053304 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:43.146193 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.108623 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.141426 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.143053 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.250840 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.252718 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.414303 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
E, [2015-01-13T12:51:45.414820 #63199] ERROR -- : Failed to show puppet classes: Error converting value for param 'modulepath': Error converting value for param 'basemodulepath': Could not find value for $confdir
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42/classes HTTP/1.1" 406 162 2.3751
I, [2015-01-13T12:51:45.433601 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.522164 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_qa_rhel_5_8_server_x86_64_cv_69/classes HTTP/1.1" 200 2 2.7583
I, [2015-01-13T12:51:45.603873 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.702921 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.561855 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.662264 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.731477 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.748059 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.830031 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
</pre>
Comment 1 Justin Sherrill 2015-02-24 22:50:14 UTC 
Created from redmine issue http://projects.theforeman.org/issues/8932
Comment 3 Bryan Kearney 2015-03-31 19:44:20 UTC 

*** This bug has been marked as a duplicate of bug 1117376 ***
Comment 4 Justin Sherrill 2015-03-31 20:02:32 UTC 
Note this is also partially duplicated by https://bugzilla.redhat.com/show_bug.cgi?id=1185838
Note You need to log in before you can comment on or make changes to this bug.