Bug 1773133 - Foreman-rake facts:cleanup batch/rolling support
Summary: Foreman-rake facts:cleanup batch/rolling support
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Fact
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: 6.7.0
Assignee: Lukas Zapletal
QA Contact: Radovan Drazny
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-15 23:51 UTC by wclark
Modified: 2020-05-17 11:42 UTC (History)
7 users (show)

Fixed In Version: foreman-1.23.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1791495 (view as bug list)
Environment:
Last Closed: 2020-04-14 13:27:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 25227 0 Normal Closed Foreman-rake facts:cleanup batch/rolling support 2020-09-30 15:03:48 UTC
Github theforeman foreman pull 6153 0 'None' closed Fixes #25227 - facts:cleanup batch support 2020-09-30 15:03:48 UTC
Red Hat Product Errata RHSA-2020:1454 0 None None None 2020-04-14 13:27:19 UTC

Description wclark 2019-11-15 23:51:49 UTC
Description of problem:


`foreman-rake facts:clean` does not perform batched cleanup, causing `ERROR:  invalid message format` when count(facts) is very large

Build up of too many facts can cause performance degradation in Satellite. This issue prevents facts from being cleaned in that scenario.


Version-Release number of selected component (if applicable):

foreman-1.22.0.32-1.el7sat.noarch
satellite-6.6.0-7.el7sat.noarch

How reproducible:

Occurs in Satellite deployments with many facts

Steps to Reproduce:
1. Create many facts
2. Run `foreman-rake facts:clean`

Actual results:


=============================================================================================
rake aborted!
ActiveRecord::StatementInvalid: PG::ProtocolViolation: ERROR:  invalid message format

****SNIP****

/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::ProtocolViolation: ERROR:  invalid message format
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => facts:clean
(See full trace by running task with --trace)
Starting orphaned facts clean up...
=============================================================================================



Expected results:

Facts are cleaned

Additional info:

Issue is resolved upstream using rolling/batch facts cleanup

https://projects.theforeman.org/issues/25227

https://github.com/theforeman/foreman/pull/6153

Comment 3 Bryan Kearney 2019-11-16 01:01:10 UTC
Upstream bug assigned to lzap@redhat.com

Comment 4 Bryan Kearney 2019-11-16 01:01:11 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25227 has been resolved.

Comment 6 wclark 2019-11-25 19:18:21 UTC
Hotfix RPM is available for Satellite 6.5.3:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.20.1.36-2.HOTFIXRHBZ1773133.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.

Comment 10 Tomer Brisker 2019-12-24 13:42:53 UTC
Connecting redmine issue https://projects.theforeman.org/issues/25227 from this bug

Comment 11 wclark 2019-12-24 15:50:27 UTC
Hotfix RPM is available for Satellite 6.6.1 which fixes this BZ and BZ1781340:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.22.0.33-2.HOTFIXRHBZ1773133_1781340.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.

!!!NOTE!!!: This hotfix can be installed on Satellite 6.6.0 as well but it will be necessary to downgrade it before updating to Satellite 6.6.1 or later:

To remove the hotfix from Satellite 6.6.0 before updating:

1. # satellite-maintain packages unlock

2. # yum downgrade foreman-1.22.0.32

3. # satellite-maintain packages lock

Comment 15 Radovan Drazny 2020-01-30 10:25:42 UTC
Verified with (almost) 55000 orphaned facts on Satellite 6.7 Snap 9. 

$ foreman-rake facts:clean --trace --verbose
** Invoke facts:clean (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute facts:clean
Starting orphaned facts clean up
Finished, cleaned 54999 facts

No issues during the cleanup.

Comment 19 Lukas Zapletal 2020-03-02 08:20:30 UTC
There is another problem associated with this bug which was fixed in 1.24 (6.7) https://bugzilla.redhat.com/show_bug.cgi?id=1781340 (Cleaning orphaned facts fails with undefined method map)

If you need a backport raise new BZ please, it's a trivial one.

Comment 22 errata-xmlrpc 2020-04-14 13:27:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454


Note You need to log in before you can comment on or make changes to this bug.