Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1773133 - Foreman-rake facts:cleanup batch/rolling support
Summary: Foreman-rake facts:cleanup batch/rolling support
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Fact
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: 6.7.0
Assignee: Lukas Zapletal
QA Contact: Radovan Drazny
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-15 23:51 UTC by wclark
Modified: 2023-10-06 18:47 UTC (History)
7 users (show)

Fixed In Version: foreman-1.23.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1791495 (view as bug list)
Environment:
Last Closed: 2020-04-14 13:27:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 25227 0 Normal Closed Foreman-rake facts:cleanup batch/rolling support 2020-09-30 15:03:48 UTC
Github theforeman foreman pull 6153 0 'None' closed Fixes #25227 - facts:cleanup batch support 2020-09-30 15:03:48 UTC
Red Hat Product Errata RHSA-2020:1454 0 None None None 2020-04-14 13:27:19 UTC

Description wclark 2019-11-15 23:51:49 UTC 
Description of problem:


`foreman-rake facts:clean` does not perform batched cleanup, causing `ERROR:  invalid message format` when count(facts) is very large

Build up of too many facts can cause performance degradation in Satellite. This issue prevents facts from being cleaned in that scenario.


Version-Release number of selected component (if applicable):

foreman-1.22.0.32-1.el7sat.noarch
satellite-6.6.0-7.el7sat.noarch

How reproducible:

Occurs in Satellite deployments with many facts

Steps to Reproduce:
1. Create many facts
2. Run `foreman-rake facts:clean`

Actual results:


=============================================================================================
rake aborted!
ActiveRecord::StatementInvalid: PG::ProtocolViolation: ERROR:  invalid message format

****SNIP****

/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::ProtocolViolation: ERROR:  invalid message format
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:590:in `execute_and_clear'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:163:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `delete'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:401:in `delete_all'
/usr/share/foreman/app/services/fact_cleaner.rb:73:in `delete_excluded_facts'
/usr/share/foreman/app/services/fact_cleaner.rb:5:in `clean!'
/usr/share/foreman/lib/tasks/facts.rake:14:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => facts:clean
(See full trace by running task with --trace)
Starting orphaned facts clean up...
=============================================================================================



Expected results:

Facts are cleaned

Additional info:

Issue is resolved upstream using rolling/batch facts cleanup

https://projects.theforeman.org/issues/25227

https://github.com/theforeman/foreman/pull/6153
Comment 3 Bryan Kearney 2019-11-16 01:01:10 UTC 
Upstream bug assigned to lzap
Comment 4 Bryan Kearney 2019-11-16 01:01:11 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25227 has been resolved.
Comment 6 wclark 2019-11-25 19:18:21 UTC 
Hotfix RPM is available for Satellite 6.5.3:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.20.1.36-2.HOTFIXRHBZ1773133.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.
Comment 10 Tomer Brisker 2019-12-24 13:42:53 UTC 
Connecting redmine issue https://projects.theforeman.org/issues/25227 from this bug
Comment 11 wclark 2019-12-24 15:50:27 UTC 
Hotfix RPM is available for Satellite 6.6.1 which fixes this BZ and BZ1781340:

To obtain the hotfix:

1. Open a support ticket to obtain the RPM, since the hotfix RPM is too large to be provided as a bugzilla attachment.

2. Take a complete snapshot or backup of your Satellite server.

3. Copy the hotfix RPM to your Satellite server

4. # rpm -Uvh --nodeps foreman-1.22.0.33-2.HOTFIXRHBZ1773133_1781340.el7sat.noarch.rpm 

No service restart should be necessary for the change to take effect, since facts cleanup is done via Rake.

!!!NOTE!!!: This hotfix can be installed on Satellite 6.6.0 as well but it will be necessary to downgrade it before updating to Satellite 6.6.1 or later:

To remove the hotfix from Satellite 6.6.0 before updating:

1. # satellite-maintain packages unlock

2. # yum downgrade foreman-1.22.0.32

3. # satellite-maintain packages lock
Comment 15 Radovan Drazny 2020-01-30 10:25:42 UTC 
Verified with (almost) 55000 orphaned facts on Satellite 6.7 Snap 9. 

$ foreman-rake facts:clean --trace --verbose
** Invoke facts:clean (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute facts:clean
Starting orphaned facts clean up
Finished, cleaned 54999 facts

No issues during the cleanup.
Comment 19 Lukas Zapletal 2020-03-02 08:20:30 UTC 
There is another problem associated with this bug which was fixed in 1.24 (6.7) https://bugzilla.redhat.com/show_bug.cgi?id=1781340 (Cleaning orphaned facts fails with undefined method map)

If you need a backport raise new BZ please, it's a trivial one.
Comment 22 errata-xmlrpc 2020-04-14 13:27:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454
Note You need to log in before you can comment on or make changes to this bug.