1304986 – Too many environments makes "puppet cert" execution very slow

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1304986 - Too many environments makes "puppet cert" execution very slow

Summary: Too many environments makes "puppet cert" execution very slow

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Performance
Sub Component:
Version:	6.1.4
Hardware:	All
OS:	Unspecified
Priority:	urgent
Severity:	high
Target Milestone:	Unspecified
Assignee:	satellite6-bugs
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1296845 1315284
TreeView+	depends on / blocked

Reported:	2016-02-05 08:25 UTC by Sergio G.
Modified:	2020-05-14 15:06 UTC (History)
CC List:	14 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1315284 (view as bug list)
Environment:
Last Closed:	2016-12-13 15:08:31 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Sergio G. 2016-02-05 08:25:54 UTC

Description of problem:
"puppet cert" becomes slower as many directories exist in /etc/puppet/environments.


How reproducible:
Always


Steps to Reproduce:
1.Run 
for each in `seq 1 200`; do ls -l | wc -l; mkdir /etc/puppet/environments/dummy_env_$each; time puppet cert list --all &>/dev/null; done
2. Wait


Actual results:
As number of directories in /etc/puppet/environment increases, "puppet cert list --all" time increases.


Expected results:
Listing certificates information (which actually resides in /var/lib/puppet/ssl) doesn't should be affected by the number of existing environments.

Additional info:
1. This might affect to provisioning hosts because downloading a ks through /unattended/provision URL revokes the host certificate and it may derivate in a timeout in foreman-proxy. There's a support case open related.
2. This situation can be very likely to happen because deleting a content view doesn't delete the related Puppet environment. RFE to be submitted to wipe Puppet environments related to deleted CVs or link both actions.

Comment 1 Pavel Moravec 2016-02-05 08:44:18 UTC

I second this request - execution time of _any_ "puppet cert .." command is directly proportional to the number of /etc/puppet/environment subdirs.

Improved reproducer:

for i in `seq 1 10`; do
  for j in `seq 1 10`; do
    mkdir /etc/puppet/environments/dummy_env_${i}_${j}
  done
  echo "environments=$(ls /etc/puppet/environments/ | wc -l)"
  time /usr/bin/puppet cert --ssldir /var/lib/puppet/ssl --clean whatever.insane.hostname > /dev/null
done

Check how the time of cleaning even non-existing cert increases when # of subdirs in /etc/puppet/environments/ increases.

Comment 6 Peter Vreman 2016-02-05 13:18:40 UTC

I opened created Case01577779 for it of 2016-02-03. The case has attached a strace that shows that many stat() calls are being done:

grep 'stat("/etc/puppet/environments' /tmp/cert-clean-strace | wc -l
838050

There is also a Puppet ticket https://tickets.puppetlabs.com/browse/PUP-3389 that mentions it is fixed in Puppet 3.7.4

Comment 7 Peter Vreman 2016-02-05 13:21:05 UTC

Also note that this startup performance is also for catalog compiles. We are seeing with Sat6.1.x also regular timeouts (3+ minutes). With Sat6.0.x there was never a problem. It is introduced with Sat6.1.x that started to use the environmentpath variable in puppet.conf

Comment 8 Peter Vreman 2016-02-05 13:25:22 UTC

When upgrading puppet also take into consideration https://tickets.puppetlabs.com/browse/PUP-5380

Comment 9 Peter Vreman 2016-02-05 13:27:52 UTC

And also https://tickets.puppetlabs.com/browse/PUP-5547 that is related to puppet environments caching

Comment 12 Stephen Benjamin 2016-12-13 15:08:31 UTC

On 6.1, I do indeed see this behavior.  However, on 6.2 (Puppet 3.8.6), I can confirm this is fixed. I've tried with up to 10,000 environments: 

[root@sat-rhel7 environments]# for i in `seq 1 10000`; do mkdir -p Environment$i/modules; done 

[root@sat-rhel7 environments]# time puppet cert list --all + "sat-rhel7.example.com" (SHA256) 58:3A:31:45:B4:98:9C:35:9C:F6:39:06:75:E6:43:6A:54:62:8B:7B:EC:46:11:21:A3:8E:B2:4F:D1:A0:CA:FD (alt names: "DNS:puppet", "DNS:puppet.example.com", "DNS:sat-rhel7.example.com")
real 0m0.881s user 0m0.835s sys 0m0.044s

Note You need to log in before you can comment on or make changes to this bug.