Bug 1773133

Summary: Foreman-rake facts:cleanup batch/rolling support
Product: Red Hat Satellite Reporter: wclark
Component: FactAssignee: Lukas Zapletal <lzap>
Status: CLOSED ERRATA QA Contact: Radovan Drazny <rdrazny>
Severity: urgent Docs Contact:
Priority: high    
Version: 6.6.0CC: cmarinea, egolov, ktordeur, mhulan, mjia, mmccune, pmoravec
Target Milestone: 6.7.0Keywords: PrioBumpField, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-1.23.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1791495 (view as bug list) Environment:
Last Closed: 2020-04-14 13:27:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description wclark 2019-11-15 23:51:49 UTC
Description of problem:


`foreman-rake facts:clean` does not perform batched cleanup, causing `ERROR:  invalid message format` when count(facts) is very large

Build up of too many facts can cause performance degradation in Satellite. This issue prevents facts from being cleaned in that scenario.


Version-Release number of selected component (if applicable):

foreman-1.22.0.32-1.el7sat.noarch
satellite-6.6.0-7.el7sat.noarch

How reproducible:

Occurs in Satellite deployments with many facts

Steps to Reproduce:
1. Create many facts
2. Run `foreman-rake facts:clean`

Actual results:


=============================================================================================
rake aborted!
ActiveRecord::StatementInvalid: PG::ProtocolViolation: ERROR:  invalid message format

****SNIP****

/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::ProtocolViolation: ERROR:  invalid message format
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => facts:clean
(See full trace by running task with --trace)
Starting orphaned facts clean up...
=============================================================================================



Expected results:

Facts are cleaned

Additional info:

Issue is resolved upstream using rolling/batch facts cleanup

https://projects.theforeman.org/issues/25227

https://github.com/theforeman/foreman/pull/6153

Comment 3 Bryan Kearney 2019-11-16 01:01:10 UTC
Upstream bug assigned to lzap

Comment 4 Bryan Kearney 2019-11-16 01:01:11 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25227 has been resolved.

Comment 6 wclark 2019-11-25 19:18:21 UTC
Hotfix RPM is available for Satellite 6.5.3:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.20.1.36-2.HOTFIXRHBZ1773133.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.

Comment 10 Tomer Brisker 2019-12-24 13:42:53 UTC
Connecting redmine issue https://projects.theforeman.org/issues/25227 from this bug

Comment 11 wclark 2019-12-24 15:50:27 UTC
Hotfix RPM is available for Satellite 6.6.1 which fixes this BZ and BZ1781340:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.22.0.33-2.HOTFIXRHBZ1773133_1781340.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.

!!!NOTE!!!: This hotfix can be installed on Satellite 6.6.0 as well but it will be necessary to downgrade it before updating to Satellite 6.6.1 or later:

To remove the hotfix from Satellite 6.6.0 before updating:

1. # satellite-maintain packages unlock

2. # yum downgrade foreman-1.22.0.32

3. # satellite-maintain packages lock

Comment 15 Radovan Drazny 2020-01-30 10:25:42 UTC
Verified with (almost) 55000 orphaned facts on Satellite 6.7 Snap 9. 

$ foreman-rake facts:clean --trace --verbose
** Invoke facts:clean (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute facts:clean
Starting orphaned facts clean up
Finished, cleaned 54999 facts

No issues during the cleanup.

Comment 19 Lukas Zapletal 2020-03-02 08:20:30 UTC
There is another problem associated with this bug which was fixed in 1.24 (6.7) https://bugzilla.redhat.com/show_bug.cgi?id=1781340 (Cleaning orphaned facts fails with undefined method map)

If you need a backport raise new BZ please, it's a trivial one.

Comment 22 errata-xmlrpc 2020-04-14 13:27:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454