Red Hat Bugzilla – Bug 1304986
Too many environments makes "puppet cert" execution very slow
Last modified: 2017-04-26 06:42:32 EDT
Description of problem: "puppet cert" becomes slower as many directories exist in /etc/puppet/environments. How reproducible: Always Steps to Reproduce: 1.Run for each in `seq 1 200`; do ls -l | wc -l; mkdir /etc/puppet/environments/dummy_env_$each; time puppet cert list --all &>/dev/null; done 2. Wait Actual results: As number of directories in /etc/puppet/environment increases, "puppet cert list --all" time increases. Expected results: Listing certificates information (which actually resides in /var/lib/puppet/ssl) doesn't should be affected by the number of existing environments. Additional info: 1. This might affect to provisioning hosts because downloading a ks through /unattended/provision URL revokes the host certificate and it may derivate in a timeout in foreman-proxy. There's a support case open related. 2. This situation can be very likely to happen because deleting a content view doesn't delete the related Puppet environment. RFE to be submitted to wipe Puppet environments related to deleted CVs or link both actions.
I second this request - execution time of _any_ "puppet cert .." command is directly proportional to the number of /etc/puppet/environment subdirs. Improved reproducer: for i in `seq 1 10`; do for j in `seq 1 10`; do mkdir /etc/puppet/environments/dummy_env_${i}_${j} done echo "environments=$(ls /etc/puppet/environments/ | wc -l)" time /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --clean whatever.insane.hostname > /dev/null done Check how the time of cleaning even non-existing cert increases when # of subdirs in /etc/puppet/environments/ increases.
I opened created Case01577779 for it of 2016-02-03. The case has attached a strace that shows that many stat() calls are being done: grep 'stat("/etc/puppet/environments' /tmp/cert-clean-strace | wc -l 838050 There is also a Puppet ticket https://tickets.puppetlabs.com/browse/PUP-3389 that mentions it is fixed in Puppet 3.7.4
Also note that this startup performance is also for catalog compiles. We are seeing with Sat6.1.x also regular timeouts (3+ minutes). With Sat6.0.x there was never a problem. It is introduced with Sat6.1.x that started to use the environmentpath variable in puppet.conf
When upgrading puppet also take into consideration https://tickets.puppetlabs.com/browse/PUP-5380
And also https://tickets.puppetlabs.com/browse/PUP-5547 that is related to puppet environments caching
On 6.1, I do indeed see this behavior. However, on 6.2 (Puppet 3.8.6), I can confirm this is fixed. I've tried with up to 10,000 environments: [root@sat-rhel7 environments]# for i in `seq 1 10000`; do mkdir -p Environment$i/modules; done [root@sat-rhel7 environments]# time puppet cert list --all + "sat-rhel7.example.com" (SHA256) 58:3A:31:45:B4:98:9C:35:9C:F6:39:06:75:E6:43:6A:54:62:8B:7B:EC:46:11:21:A3:8E:B2:4F:D1:A0:CA:FD (alt names: "DNS:puppet", "DNS:puppet.example.com", "DNS:sat-rhel7.example.com") real 0m0.881s user 0m0.835s sys 0m0.044s