Bug 1341860 - Fact import triggers ip conflicts checks, which drives cpu utilization to 100% on smart-proxy
Summary: Fact import triggers ip conflicts checks, which drives cpu utilization to 100...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Orchestration
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Sachin Ghai
URL:
Whiteboard:
: 1341974 (view as bug list)
Depends On:
Blocks: 1335807 1337493
TreeView+ depends on / blocked
 
Reported: 2016-06-01 22:22 UTC by Brad Buckingham
Modified: 2019-09-26 17:32 UTC (History)
5 users (show)

Fixed In Version: foreman-1.11.0.34-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 11:13:36 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 12425 0 None None None 2016-06-01 22:22:43 UTC

Description Brad Buckingham 2016-06-01 22:22:42 UTC
This is an expensive call, and performing lots of them has a serious impact on smart-proxy. See http://projects.theforeman.org/issues/12392 for more details and discussion.

Comment 1 Brad Buckingham 2016-06-01 22:22:46 UTC
Created from redmine issue http://projects.theforeman.org/issues/12425

Comment 2 Brad Buckingham 2016-06-01 22:25:11 UTC
The changes associated with this bug may be required in order to support the solution provided by bug 1337462.  The reason being that bug utilizes 'importing_facts' attribute added by this one.

Comment 4 Marek Hulan 2016-06-02 09:42:43 UTC
*** Bug 1341974 has been marked as a duplicate of this bug. ***

Comment 5 Sachin Ghai 2016-06-03 07:42:39 UTC
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7.

I don't see issue reported in one of its duplicate bz 1341974 and related to following exception:

undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>/opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/attribute_methods.rb:435:in `method_missing'

Upgrade was completed successfully:


SATELLITE UPGRADE started at: Fri Jun  3 02:18:11 2016
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] run: satellite-installer --scenario satellite --upgrade
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrading...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: stop_services...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_databases...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: update_http_conf...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_pulp...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_pulp_httpd_conf...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_httpd...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_qpidd...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_pulp...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_candlepin...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_tomcat...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_katello_settings_file...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_foreman...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_nodes_distributors...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: Running installer...
Done                                               [100%] []Installing             Done                                               [100%] []
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out:   The full log is at /var/log/foreman-installer/satellite.log
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: restart_services...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: db_seed...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_package_groups (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_rpms (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_distributions (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_puppet_modules (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_subscriptions (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: elasticsearch_message...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Elasticsearch has been removed as a dependency.  The database files can be removed manually with #rm -rf /var/lib/elasticsearch.  Some packages are no longer needed and can be removed:  #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_docker_v1_content (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_gutterball...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Katello upgrade completed!

Comment 6 Sachin Ghai 2016-06-03 07:44:06 UTC
@Brad: Could you please confirm if anything specific needs to test for verification of this bz ?

Comment 7 Brad Buckingham 2016-06-03 11:42:02 UTC
Marek, do you have any recommendations for QE on additional tests that should be performed with the inclusion of this fix?

Comment 8 Marek Hulan 2016-06-03 13:19:59 UTC
As per upstream ticket, this fixes the behavior during importing of facts. To verify, trigger a puppet run and make sure that the capsule does not receive any request from satellite regarding DHCP. That can be verified in capsule log (with debug log level).

Comment 9 Sachin Ghai 2016-07-05 12:49:57 UTC
I don't see any logs when I run "puppet agent -t" on a host. I see some logs on satellite under "/var/log/foreman/production.log" but no logs under "/var/log/foreman-proxy/proxy.log"

steps taken:

1. I ran "puppet agent -t" on a host registered with satellite

]# puppet agent -t
Warning: Local environment: "production" doesn't match server specified node environment "KT_Default_Organization_DEV_cv_rhel6_3", switching agent to "KT_Default_Organization_DEV_cv_rhel6_3".
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for mac001a4a3ea60f.satqe.lab.eng.rdu2.redhat.com
Info: Applying configuration version '1467722487'
Notice: Finished catalog run in 0.18 seconds

2. checked "/var/log/foreman/production.log" but no logs under "/var/log/foreman-proxy/proxy.log"

Comment 11 Sachin Ghai 2016-07-05 13:06:15 UTC
Hey Marek,

Could you please take a look at comment9 to see if steps taken to reproduce the bz are correct ? thanks

Comment 12 Marek Hulan 2016-07-14 14:16:09 UTC
Sounds correct, just make sure your capsule log level is set to debug and restart it before testing. If not DHCP related log appears, it works.

Comment 13 Sachin Ghai 2016-07-19 06:32:56 UTC
Thanks Marek for reply.

I set the capsule log level to :DEBUG and restarted foreman-proxy on capsule. Later, I ran "puppet agent -t" on registered host and I don't see any logs related to DHCP logs under "/var/log/foreman-proxy/proxy.log"


[root@cloud-qe-21 ~]# cat /etc/foreman-proxy/settings.yml | grep DEBUG
# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
:log_level: DEBUG
 ~]# service foreman-proxy restart
Redirecting to /bin/systemctl restart  foreman-proxy.service


Moving this to verified.

Comment 14 Bryan Kearney 2016-07-27 11:13:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501


Note You need to log in before you can comment on or make changes to this bug.