Description of problem: Each Puppet runs during the deploy are taking around 15-20 minutes for large and busy env with more than 1200 containers per controller Version-Release number of selected component (if applicable): Red Hat OpenStack Platform 13 How reproducible: Always Steps to Reproduce: 1. Deploy an overcloud and create more private networks with almost 1200 containers / controller. 2. Run a deploy command or ansible playbook to deploy / upgrade env. 3. Actual results: Puppet run is slower Expected results: Puppet runs should be faster and not loop in reading /proc/mounts multiple times. Additional info: Just controllers alone took 16 hours to upgrade Upon further checking puppet is kind of looping in reading /proc/mounts file which is really slow. Found the code responsible for slowness in /usr/share/ruby/vendor_ruby/puppet/util/selinux.rb ~~~~~~~~~~~~~~~~ def read_mounts mounts = "" begin if File.method_defined? "read_nonblock" # If possible we use read_nonblock in a loop rather than read to work- # a linux kernel bug. See ticket #1963 for details. mountfh = File.open("/proc/mounts") mounts += mountfh.read_nonblock(1024) while true << Here else # Otherwise we shell out and let cat do it for us mountfh = IO.popen("/bin/cat /proc/mounts") mounts = mountfh.read << Here end rescue EOFError # that's expected rescue return nil ensure mountfh.close if mountfh end ~~~~~~~~~~~~~~~~~~ After commenting both reads, the puppet run took 47 sec in busy env, non busy env is taking around 9s.
Created attachment 1694019 [details] strace of puppet apply
strace -o puppet-tripleo-with-strace.txt puppet apply --noop --summarize --detailed-exitcodes --verbose --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --color=false -e "class { 'tripleo::profile::base::docker': step => 1, }" --debug -v 2>&1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 13 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4387