Bug 1773133 - Foreman-rake facts:cleanup batch/rolling support
Summary: Foreman-rake facts:cleanup batch/rolling support
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Fact
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
high
urgent vote
Target Milestone: 6.7.0
Assignee: Lukas Zapletal
QA Contact: Radovan Drazny
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-15 23:51 UTC by wclark
Modified: 2020-02-11 17:26 UTC (History)
5 users (show)

Fixed In Version: foreman-1.23.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1791495 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 25227 Normal Closed Foreman-rake facts:cleanup batch/rolling support 2020-02-20 09:47:52 UTC
Github theforeman foreman pull 6153 'None' closed Fixes #25227 - facts:cleanup batch support 2020-02-20 09:47:52 UTC

Description wclark 2019-11-15 23:51:49 UTC
Description of problem:


`foreman-rake facts:clean` does not perform batched cleanup, causing `ERROR:  invalid message format` when count(facts) is very large

Build up of too many facts can cause performance degradation in Satellite. This issue prevents facts from being cleaned in that scenario.


Version-Release number of selected component (if applicable):

foreman-1.22.0.32-1.el7sat.noarch
satellite-6.6.0-7.el7sat.noarch

How reproducible:

Occurs in Satellite deployments with many facts

Steps to Reproduce:
1. Create many facts
2. Run `foreman-rake facts:clean`

Actual results:


=============================================================================================
rake aborted!
ActiveRecord::StatementInvalid: PG::ProtocolViolation: ERROR:  invalid message format

****SNIP****

/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::ProtocolViolation: ERROR:  invalid message format
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => facts:clean
(See full trace by running task with --trace)
Starting orphaned facts clean up...
=============================================================================================



Expected results:

Facts are cleaned

Additional info:

Issue is resolved upstream using rolling/batch facts cleanup

https://projects.theforeman.org/issues/25227

https://github.com/theforeman/foreman/pull/6153

Comment 3 Bryan Kearney 2019-11-16 01:01:10 UTC
Upstream bug assigned to lzap@redhat.com

Comment 4 Bryan Kearney 2019-11-16 01:01:11 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25227 has been resolved.

Comment 6 wclark 2019-11-25 19:18:21 UTC
Hotfix RPM is available for Satellite 6.5.3:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.20.1.36-2.HOTFIXRHBZ1773133.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.

Comment 10 Tomer Brisker 2019-12-24 13:42:53 UTC
Connecting redmine issue https://projects.theforeman.org/issues/25227 from this bug

Comment 11 wclark 2019-12-24 15:50:27 UTC
Hotfix RPM is available for Satellite 6.6.1 which fixes this BZ and BZ1781340:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.22.0.33-2.HOTFIXRHBZ1773133_1781340.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.

!!!NOTE!!!: This hotfix can be installed on Satellite 6.6.0 as well but it will be necessary to downgrade it before updating to Satellite 6.6.1 or later:

To remove the hotfix from Satellite 6.6.0 before updating:

1. # satellite-maintain packages unlock

2. # yum downgrade foreman-1.22.0.32

3. # satellite-maintain packages lock

Comment 15 Radovan Drazny 2020-01-30 10:25:42 UTC
Verified with (almost) 55000 orphaned facts on Satellite 6.7 Snap 9. 

$ foreman-rake facts:clean --trace --verbose
** Invoke facts:clean (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute facts:clean
Starting orphaned facts clean up
Finished, cleaned 54999 facts

No issues during the cleanup.


Note You need to log in before you can comment on or make changes to this bug.