Bug 1304986 - Too many environments makes "puppet cert" execution very slow
Summary: Too many environments makes "puppet cert" execution very slow
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Performance
Version: 6.1.4
Hardware: All
OS: Unspecified
urgent
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1296845 1315284
TreeView+ depends on / blocked
 
Reported: 2016-02-05 08:25 UTC by Sergio G.
Modified: 2020-05-14 15:06 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1315284 (view as bug list)
Environment:
Last Closed: 2016-12-13 15:08:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sergio G. 2016-02-05 08:25:54 UTC
Description of problem:
"puppet cert" becomes slower as many directories exist in /etc/puppet/environments.


How reproducible:
Always


Steps to Reproduce:
1.Run 
for each in `seq 1 200`; do ls -l | wc -l; mkdir /etc/puppet/environments/dummy_env_$each; time puppet cert list --all &>/dev/null; done
2. Wait


Actual results:
As number of directories in /etc/puppet/environment increases, "puppet cert list --all" time increases.


Expected results:
Listing certificates information (which actually resides in /var/lib/puppet/ssl) doesn't should be affected by the number of existing environments.

Additional info:
1. This might affect to provisioning hosts because downloading a ks through /unattended/provision URL revokes the host certificate and it may derivate in a timeout in foreman-proxy. There's a support case open related.
2. This situation can be very likely to happen because deleting a content view doesn't delete the related Puppet environment. RFE to be submitted to wipe Puppet environments related to deleted CVs or link both actions.

Comment 1 Pavel Moravec 2016-02-05 08:44:18 UTC
I second this request - execution time of _any_ "puppet cert .." command is directly proportional to the number of /etc/puppet/environment subdirs.

Improved reproducer:

for i in `seq 1 10`; do
  for j in `seq 1 10`; do
    mkdir /etc/puppet/environments/dummy_env_${i}_${j}
  done
  echo "environments=$(ls /etc/puppet/environments/ | wc -l)"
  time /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --clean whatever.insane.hostname > /dev/null
done

Check how the time of cleaning even non-existing cert increases when # of subdirs in /etc/puppet/environments/ increases.

Comment 6 Peter Vreman 2016-02-05 13:18:40 UTC
I opened created Case01577779 for it of 2016-02-03. The case has attached a strace that shows that many stat() calls are being done:

grep 'stat("/etc/puppet/environments' /tmp/cert-clean-strace | wc -l
838050

There is also a Puppet ticket https://tickets.puppetlabs.com/browse/PUP-3389 that mentions it is fixed in Puppet 3.7.4

Comment 7 Peter Vreman 2016-02-05 13:21:05 UTC
Also note that this startup performance is also for catalog compiles. We are seeing with Sat6.1.x also regular timeouts (3+ minutes). With Sat6.0.x there was never a problem. It is introduced with Sat6.1.x that started to use the environmentpath variable in puppet.conf

Comment 8 Peter Vreman 2016-02-05 13:25:22 UTC
When upgrading puppet also take into consideration https://tickets.puppetlabs.com/browse/PUP-5380

Comment 9 Peter Vreman 2016-02-05 13:27:52 UTC
And also https://tickets.puppetlabs.com/browse/PUP-5547 that is related to puppet environments caching

Comment 12 Stephen Benjamin 2016-12-13 15:08:31 UTC
On 6.1, I do indeed see this behavior.  However, on 6.2 (Puppet 3.8.6), I can confirm this is fixed. I've tried with up to 10,000 environments: 

[root@sat-rhel7 environments]# for i in `seq 1 10000`; do mkdir -p Environment$i/modules; done 

[root@sat-rhel7 environments]# time puppet cert list --all + "sat-rhel7.example.com" (SHA256) 58:3A:31:45:B4:98:9C:35:9C:F6:39:06:75:E6:43:6A:54:62:8B:7B:EC:46:11:21:A3:8E:B2:4F:D1:A0:CA:FD (alt names: "DNS:puppet", "DNS:puppet.example.com", "DNS:sat-rhel7.example.com")
real 0m0.881s user 0m0.835s sys 0m0.044s


Note You need to log in before you can comment on or make changes to this bug.