Bug 1728402 - large number of interfaces cause slow puppet executions due to fact generation
Summary: large number of interfaces cause slow puppet executions due to fact generation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.0 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Upstream M2
: 16.0 (Train on RHEL 8.1)
Assignee: Alex Schultz
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks: 1711267 1731208 1731209 1731210
TreeView+ depends on / blocked
 
Reported: 2019-07-09 20:56 UTC by Alex Schultz
Modified: 2020-02-06 14:42 UTC (History)
2 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.1.1-0.20190822173502.e6b598f.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1731208 1731209 1731210 (view as bug list)
Environment:
Last Closed: 2020-02-06 14:41:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1835959 0 None None None 2019-07-09 20:58:31 UTC
OpenStack gerrit 671524 0 'None' MERGED Re-Add facter cache for container configurations 2021-01-12 22:27:52 UTC
OpenStack gerrit 672831 0 'None' MERGED Set selinux type for facter.conf 2021-01-12 22:27:52 UTC
Red Hat Product Errata RHEA-2020:0283 0 None None None 2020-02-06 14:42:16 UTC

Description Alex Schultz 2019-07-09 20:56:56 UTC
Description of problem:
When a compute node (or controller) has many interfaces (bridges/tun), during the application of updates the puppet processes start thrashing trying to generate the network puppet facts. This affects both facter 2 and facter 3 based systems due to the fact that multiple processes end up trying to execute ip commands to gather the networking facts. This can be worked around by caching the facts for the life of the container-puppet.py (or docker-puppet.py) executions. This affects all versions of TripleO since Ocata. Prior to Ocata, facter 2 is super slow when there are more than than 1000 interfaces on a host. At ~1500, it can take >= 10 minutes to generate the facts.

for i in $(seq 1 380); do ip tuntap add name dummy_tun$i mode tun; done
for i in $(seq 1 1274); do ip link add name dummy_br$i type bridge; done

$ time facter

facter2
real 9m51.817s
user 7m8.936s
sys 2m42.702s

facter3
real 0m2.954s
user 0m1.111s
sys 0m1.721s

$ time puppet facts

facter2
real 12m10.936s
user 8m16.478s
sys 3m54.138s

facter3
real 0m11.169s
user 0m5.522s
sys 0m4.002s


Steps to Reproduce:
1. deploy basic undercloud
2. deploy default overcloud 1ctlr+1compute
3. Apply dummy network interfaces to compute
for i in $(seq 1 380); do ip tuntap add name dummy_tun$i mode tun; done
for i in $(seq 1 1274); do ip link add name dummy_br$i type bridge; done
4. deploy stack update with no configuration changes

Actual results:
Stack update will take an excessive amount of time (almost 4 hours in my test) compared to a basic 1ctlr+1compute update with no interfaces (~30 mins in my test)

Expected results:
Stack update may take a bit longer but it shouldn't approach timeouts.

Comment 4 Jad Haj Yahya 2019-11-25 07:31:59 UTC
Verified on RHOS_TRUNK-16.0-RHEL-8-20191122.n.2

Comment 7 errata-xmlrpc 2020-02-06 14:41:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283


Note You need to log in before you can comment on or make changes to this bug.