Bug 1304986

Summary:	Too many environments makes "puppet cert" execution very slow
Product:	Red Hat Satellite	Reporter:	Sergio G. <sgarciam>
Component:	Performance	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	urgent
Version:	6.1.4	CC:	ahoness, bbuckingham, bkearney, cfouant, chorn, ealcaniz, egolov, ghares, peter.vreman, pmoravec, riehecky, stbenjam, sthirugn, xdmoon
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1315284 (view as bug list)		Environment:
Last Closed:	2016-12-13 15:08:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1296845, 1315284

Description Sergio G. 2016-02-05 08:25:54 UTC

Description of problem:
"puppet cert" becomes slower as many directories exist in /etc/puppet/environments.


How reproducible:
Always


Steps to Reproduce:
1.Run 
for each in `seq 1 200`; do ls -l | wc -l; mkdir /etc/puppet/environments/dummy_env_$each; time puppet cert list --all &>/dev/null; done
2. Wait


Actual results:
As number of directories in /etc/puppet/environment increases, "puppet cert list --all" time increases.


Expected results:
Listing certificates information (which actually resides in /var/lib/puppet/ssl) doesn't should be affected by the number of existing environments.

Additional info:
1. This might affect to provisioning hosts because downloading a ks through /unattended/provision URL revokes the host certificate and it may derivate in a timeout in foreman-proxy. There's a support case open related.
2. This situation can be very likely to happen because deleting a content view doesn't delete the related Puppet environment. RFE to be submitted to wipe Puppet environments related to deleted CVs or link both actions.

Comment 1 Pavel Moravec 2016-02-05 08:44:18 UTC

I second this request - execution time of _any_ "puppet cert .." command is directly proportional to the number of /etc/puppet/environment subdirs.

Improved reproducer:

for i in `seq 1 10`; do
  for j in `seq 1 10`; do
    mkdir /etc/puppet/environments/dummy_env_${i}_${j}
  done
  echo "environments=$(ls /etc/puppet/environments/ | wc -l)"
  time /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --clean whatever.insane.hostname > /dev/null
done

Check how the time of cleaning even non-existing cert increases when # of subdirs in /etc/puppet/environments/ increases.

Comment 6 Peter Vreman 2016-02-05 13:18:40 UTC

I opened created Case01577779 for it of 2016-02-03. The case has attached a strace that shows that many stat() calls are being done:

grep 'stat("/etc/puppet/environments' /tmp/cert-clean-strace | wc -l
838050

There is also a Puppet ticket https://tickets.puppetlabs.com/browse/PUP-3389 that mentions it is fixed in Puppet 3.7.4

Comment 7 Peter Vreman 2016-02-05 13:21:05 UTC

Also note that this startup performance is also for catalog compiles. We are seeing with Sat6.1.x also regular timeouts (3+ minutes). With Sat6.0.x there was never a problem. It is introduced with Sat6.1.x that started to use the environmentpath variable in puppet.conf

Comment 8 Peter Vreman 2016-02-05 13:25:22 UTC

When upgrading puppet also take into consideration https://tickets.puppetlabs.com/browse/PUP-5380

Comment 9 Peter Vreman 2016-02-05 13:27:52 UTC

And also https://tickets.puppetlabs.com/browse/PUP-5547 that is related to puppet environments caching

Comment 12 Stephen Benjamin 2016-12-13 15:08:31 UTC

On 6.1, I do indeed see this behavior.  However, on 6.2 (Puppet 3.8.6), I can confirm this is fixed. I've tried with up to 10,000 environments: 

[root@sat-rhel7 environments]# for i in `seq 1 10000`; do mkdir -p Environment$i/modules; done 

[root@sat-rhel7 environments]# time puppet cert list --all + "sat-rhel7.example.com" (SHA256) 58:3A:31:45:B4:98:9C:35:9C:F6:39:06:75:E6:43:6A:54:62:8B:7B:EC:46:11:21:A3:8E:B2:4F:D1:A0:CA:FD (alt names: "DNS:puppet", "DNS:puppet.example.com", "DNS:sat-rhel7.example.com")
real 0m0.881s user 0m0.835s sys 0m0.044s