Bug 1731209

Summary: [OSP14] large number of interfaces cause slow puppet executions due to fact generation
Product: Red Hat OpenStack Reporter: Alex Schultz <aschultz>
Component: openstack-tripleo-heat-templatesAssignee: Alex Schultz <aschultz>
Status: CLOSED EOL QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 14.0 (Rocky)CC: augol, emacchi, mburns, ssmolyak
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: openstack-tripleo-heat-templates-9.3.1-0.20190513171776.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1728402 Environment:
Last Closed: 2020-08-26 20:59:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1726325, 1728402, 1731210    
Bug Blocks: 1711267, 1731208    

Description Alex Schultz 2019-07-18 15:38:40 UTC
+++ This bug was initially created as a clone of Bug #1728402 +++

Description of problem:
When a compute node (or controller) has many interfaces (bridges/tun), during the application of updates the puppet processes start thrashing trying to generate the network puppet facts. This affects both facter 2 and facter 3 based systems due to the fact that multiple processes end up trying to execute ip commands to gather the networking facts. This can be worked around by caching the facts for the life of the container-puppet.py (or docker-puppet.py) executions. This affects all versions of TripleO since Ocata. Prior to Ocata, facter 2 is super slow when there are more than than 1000 interfaces on a host. At ~1500, it can take >= 10 minutes to generate the facts.

for i in $(seq 1 380); do ip tuntap add name dummy_tun$i mode tun; done
for i in $(seq 1 1274); do ip link add name dummy_br$i type bridge; done

$ time facter

real 9m51.817s
user 7m8.936s
sys 2m42.702s

real 0m2.954s
user 0m1.111s
sys 0m1.721s

$ time puppet facts

real 12m10.936s
user 8m16.478s
sys 3m54.138s

real 0m11.169s
user 0m5.522s
sys 0m4.002s

Steps to Reproduce:
1. deploy basic undercloud
2. deploy default overcloud 1ctlr+1compute
3. Apply dummy network interfaces to compute
for i in $(seq 1 380); do ip tuntap add name dummy_tun$i mode tun; done
for i in $(seq 1 1274); do ip link add name dummy_br$i type bridge; done
4. deploy stack update with no configuration changes

Actual results:
Stack update will take an excessive amount of time (almost 4 hours in my test) compared to a basic 1ctlr+1compute update with no interfaces (~30 mins in my test)

Expected results:
Stack update may take a bit longer but it shouldn't approach timeouts.

Comment 1 Alex Schultz 2019-07-18 15:42:37 UTC
It should be noted that OSP14 uses facter2 which will still have a larger delay than with facter3 (osp15+). We are currently investigating upgrading facter as well for this version but this should improve the existing performance.

Comment 2 Emilien Macchi 2019-08-29 14:33:28 UTC
This is a work in progress as you can see in the dependencies being rebased. Moving to ON_DEV. Alex please move it to POST/MODIFIED if you think it's ready. Thanks!