Hide Forgot
Description of problem: `foreman-rake facts:clean` does not perform batched cleanup, causing `ERROR: invalid message format` when count(facts) is very large Build up of too many facts can cause performance degradation in Satellite. This issue prevents facts from being cleaned in that scenario. Version-Release number of selected component (if applicable): foreman-1.22.0.32-1.el7sat.noarch satellite-6.6.0-7.el7sat.noarch How reproducible: Occurs in Satellite deployments with many facts Steps to Reproduce: 1. Create many facts 2. Run `foreman-rake facts:clean` Actual results: ============================================================================================= rake aborted! ActiveRecord::StatementInvalid: PG::ProtocolViolation: ERROR: invalid message format ****SNIP**** /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all' /usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts' /usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!' /usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>' Caused by: PG::ProtocolViolation: ERROR: invalid message format /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all' /usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts' /usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!' /usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>' Tasks: TOP => facts:clean (See full trace by running task with --trace) Starting orphaned facts clean up... ============================================================================================= Expected results: Facts are cleaned Additional info: Issue is resolved upstream using rolling/batch facts cleanup https://projects.theforeman.org/issues/25227 https://github.com/theforeman/foreman/pull/6153
Upstream bug assigned to lzap@redhat.com
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25227 has been resolved.
Hotfix RPM is available for Satellite 6.5.3: To obtain the hotfix: 1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment. 2. Take a complete snapshot or backup of your Satellite server. 3. Copy the hotfix RPM to your Satellite server 4. # rpm -Uvh --nodeps foreman-1.20.1.36-2.HOTFIXRHBZ1773133.el7sat.noarch.rpm No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.
Connecting redmine issue https://projects.theforeman.org/issues/25227 from this bug
Hotfix RPM is available for Satellite 6.6.1 which fixes this BZ and BZ1781340: To obtain the hotfix: 1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment. 2. Take a complete snapshot or backup of your Satellite server. 3. Copy the hotfix RPM to your Satellite server 4. # rpm -Uvh --nodeps foreman-1.22.0.33-2.HOTFIXRHBZ1773133_1781340.el7sat.noarch.rpm No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake. !!!NOTE!!!: This hotfix can be installed on Satellite 6.6.0 as well but it will be necessary to downgrade it before updating to Satellite 6.6.1 or later: To remove the hotfix from Satellite 6.6.0 before updating: 1. # satellite-maintain packages unlock 2. # yum downgrade foreman-1.22.0.32 3. # satellite-maintain packages lock
Verified with (almost) 55000 orphaned facts on Satellite 6.7 Snap 9. $ foreman-rake facts:clean --trace --verbose ** Invoke facts:clean (first_time) ** Invoke environment (first_time) ** Execute environment ** Execute facts:clean Starting orphaned facts clean up Finished, cleaned 54999 facts No issues during the cleanup.