Bug 1341860

Summary: Fact import triggers ip conflicts checks, which drives cpu utilization to 100% on smart-proxy
Product: Red Hat Satellite Reporter: Brad Buckingham <bbuckingham>
Component: OrchestrationAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, cwelton, ehelms, mhulan, sghai
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: foreman- Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:13:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1335807, 1337493    

Description Brad Buckingham 2016-06-01 22:22:42 UTC
This is an expensive call, and performing lots of them has a serious impact on smart-proxy. See http://projects.theforeman.org/issues/12392 for more details and discussion.

Comment 1 Brad Buckingham 2016-06-01 22:22:46 UTC
Created from redmine issue http://projects.theforeman.org/issues/12425

Comment 2 Brad Buckingham 2016-06-01 22:25:11 UTC
The changes associated with this bug may be required in order to support the solution provided by bug 1337462.  The reason being that bug utilizes 'importing_facts' attribute added by this one.

Comment 4 Marek Hulan 2016-06-02 09:42:43 UTC
*** Bug 1341974 has been marked as a duplicate of this bug. ***

Comment 5 Sachin Ghai 2016-06-03 07:42:39 UTC
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14.2 on rhel7.

I don't see issue reported in one of its duplicate bz 1341974 and related to following exception:

undefined method `importing_facts=' for #<Host::Managed:0x00000008cf1e78>/opt/rh/rh-ror41/root/usr/share/gems/gems/activemodel-4.1.5/lib/active_model/attribute_methods.rb:435:in `method_missing'

Upgrade was completed successfully:

SATELLITE UPGRADE started at: Fri Jun  3 02:18:11 2016
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] run: satellite-installer --scenario satellite --upgrade
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrading...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: stop_services...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_databases...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: update_http_conf...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_pulp...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_pulp_httpd_conf...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_httpd...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_qpidd...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_pulp...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_candlepin...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: start_tomcat...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: fix_katello_settings_file...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: migrate_foreman...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_nodes_distributors...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: Running installer...
Done                                               [100%] []Installing             Done                                               [100%] []
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out:   The full log is at /var/log/foreman-installer/satellite.log
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: restart_services...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: db_seed...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_package_groups (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_rpms (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_distributions (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_puppet_modules (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: import_subscriptions (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: elasticsearch_message...
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Elasticsearch has been removed as a dependency.  The database files can be removed manually with #rm -rf /var/lib/elasticsearch.  Some packages are no longer needed and can be removed:  #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_docker_v1_content (this may take a while) ...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade Step: remove_gutterball...

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Katello upgrade completed!

Comment 6 Sachin Ghai 2016-06-03 07:44:06 UTC
@Brad: Could you please confirm if anything specific needs to test for verification of this bz ?

Comment 7 Brad Buckingham 2016-06-03 11:42:02 UTC
Marek, do you have any recommendations for QE on additional tests that should be performed with the inclusion of this fix?

Comment 8 Marek Hulan 2016-06-03 13:19:59 UTC
As per upstream ticket, this fixes the behavior during importing of facts. To verify, trigger a puppet run and make sure that the capsule does not receive any request from satellite regarding DHCP. That can be verified in capsule log (with debug log level).

Comment 9 Sachin Ghai 2016-07-05 12:49:57 UTC
I don't see any logs when I run "puppet agent -t" on a host. I see some logs on satellite under "/var/log/foreman/production.log" but no logs under "/var/log/foreman-proxy/proxy.log"

steps taken:

1. I ran "puppet agent -t" on a host registered with satellite

]# puppet agent -t
Warning: Local environment: "production" doesn't match server specified node environment "KT_Default_Organization_DEV_cv_rhel6_3", switching agent to "KT_Default_Organization_DEV_cv_rhel6_3".
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for mac001a4a3ea60f.satqe.lab.eng.rdu2.redhat.com
Info: Applying configuration version '1467722487'
Notice: Finished catalog run in 0.18 seconds

2. checked "/var/log/foreman/production.log" but no logs under "/var/log/foreman-proxy/proxy.log"

Comment 11 Sachin Ghai 2016-07-05 13:06:15 UTC
Hey Marek,

Could you please take a look at comment9 to see if steps taken to reproduce the bz are correct ? thanks

Comment 12 Marek Hulan 2016-07-14 14:16:09 UTC
Sounds correct, just make sure your capsule log level is set to debug and restart it before testing. If not DHCP related log appears, it works.

Comment 13 Sachin Ghai 2016-07-19 06:32:56 UTC
Thanks Marek for reply.

I set the capsule log level to :DEBUG and restarted foreman-proxy on capsule. Later, I ran "puppet agent -t" on registered host and I don't see any logs related to DHCP logs under "/var/log/foreman-proxy/proxy.log"

[root@cloud-qe-21 ~]# cat /etc/foreman-proxy/settings.yml | grep DEBUG
:log_level: DEBUG
 ~]# service foreman-proxy restart
Redirecting to /bin/systemctl restart  foreman-proxy.service

Moving this to verified.

Comment 14 Bryan Kearney 2016-07-27 11:13:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.