Bug 1304986 - Too many environments makes "puppet cert" execution very slow
Too many environments makes "puppet cert" execution very slow
Status: CLOSED CURRENTRELEASE
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Performance (Show other bugs)
6.1.4
All Unspecified
urgent Severity high (vote)
: Unspecified
: --
Assigned To: satellite6-bugs
: Triaged
Depends On:
Blocks: 1296845 1315284
  Show dependency treegraph
 
Reported: 2016-02-05 03:25 EST by sgarciam
Modified: 2017-04-26 06:42 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1315284 (view as bug list)
Environment:
Last Closed: 2016-12-13 10:08:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description sgarciam 2016-02-05 03:25:54 EST
Description of problem:
"puppet cert" becomes slower as many directories exist in /etc/puppet/environments.


How reproducible:
Always


Steps to Reproduce:
1.Run 
for each in `seq 1 200`; do ls -l | wc -l; mkdir /etc/puppet/environments/dummy_env_$each; time puppet cert list --all &>/dev/null; done
2. Wait


Actual results:
As number of directories in /etc/puppet/environment increases, "puppet cert list --all" time increases.


Expected results:
Listing certificates information (which actually resides in /var/lib/puppet/ssl) doesn't should be affected by the number of existing environments.

Additional info:
1. This might affect to provisioning hosts because downloading a ks through /unattended/provision URL revokes the host certificate and it may derivate in a timeout in foreman-proxy. There's a support case open related.
2. This situation can be very likely to happen because deleting a content view doesn't delete the related Puppet environment. RFE to be submitted to wipe Puppet environments related to deleted CVs or link both actions.
Comment 1 Pavel Moravec 2016-02-05 03:44:18 EST
I second this request - execution time of _any_ "puppet cert .." command is directly proportional to the number of /etc/puppet/environment subdirs.

Improved reproducer:

for i in `seq 1 10`; do
  for j in `seq 1 10`; do
    mkdir /etc/puppet/environments/dummy_env_${i}_${j}
  done
  echo "environments=$(ls /etc/puppet/environments/ | wc -l)"
  time /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --clean whatever.insane.hostname > /dev/null
done

Check how the time of cleaning even non-existing cert increases when # of subdirs in /etc/puppet/environments/ increases.
Comment 6 Peter Vreman 2016-02-05 08:18:40 EST
I opened created Case01577779 for it of 2016-02-03. The case has attached a strace that shows that many stat() calls are being done:

grep 'stat("/etc/puppet/environments' /tmp/cert-clean-strace | wc -l
838050

There is also a Puppet ticket https://tickets.puppetlabs.com/browse/PUP-3389 that mentions it is fixed in Puppet 3.7.4
Comment 7 Peter Vreman 2016-02-05 08:21:05 EST
Also note that this startup performance is also for catalog compiles. We are seeing with Sat6.1.x also regular timeouts (3+ minutes). With Sat6.0.x there was never a problem. It is introduced with Sat6.1.x that started to use the environmentpath variable in puppet.conf
Comment 8 Peter Vreman 2016-02-05 08:25:22 EST
When upgrading puppet also take into consideration https://tickets.puppetlabs.com/browse/PUP-5380
Comment 9 Peter Vreman 2016-02-05 08:27:52 EST
And also https://tickets.puppetlabs.com/browse/PUP-5547 that is related to puppet environments caching
Comment 12 Stephen Benjamin 2016-12-13 10:08:31 EST
On 6.1, I do indeed see this behavior.  However, on 6.2 (Puppet 3.8.6), I can confirm this is fixed. I've tried with up to 10,000 environments: 

[root@sat-rhel7 environments]# for i in `seq 1 10000`; do mkdir -p Environment$i/modules; done 

[root@sat-rhel7 environments]# time puppet cert list --all + "sat-rhel7.example.com" (SHA256) 58:3A:31:45:B4:98:9C:35:9C:F6:39:06:75:E6:43:6A:54:62:8B:7B:EC:46:11:21:A3:8E:B2:4F:D1:A0:CA:FD (alt names: "DNS:puppet", "DNS:puppet.example.com", "DNS:sat-rhel7.example.com")
real 0m0.881s user 0m0.835s sys 0m0.044s

Note You need to log in before you can comment on or make changes to this bug.