Bug 1291307

Summary: 40-hiera-datafiles is slow
Product: Red Hat OpenStack Reporter: Attila Fazekas <afazekas>
Component: openstack-tripleo-puppet-elementsAssignee: RHOS Maint <rhos-maint>
Status: CLOSED WONTFIX QA Contact: Shai Revivo <srevivo>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: hbrock, jcoufal, jslagle, mburns, rhel-osp-director-maint
Target Milestone: ---   
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-10 03:22:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Attila Fazekas 2015-12-14 14:34:01 UTC
Description of problem:
/usr/libexec/os-refresh-config/configure.d/40-hiera-datafiles executed at every configuration phase. For example when you just change /usr/share/openstack-tripleo-heat-templates/environments/config-debug.yaml ConfigDebug parameter it is executed 4 times. At the first deployment, it is expected to run even more times.

In my small deployment it takes 22 sec to execute,
the execution includes 1081 * jq command and 40 * os-apply-config.

Both external command is for json(/yaml) manipulation,
the os-apply-config tooks ~0.5 sec, 0.25 sec just to load the required python

The jq is a relative fast C program 0.001..0.01 sec, but it is executed 1000 times, and while the process is under strace it's costs even more.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. 3 controller, one compute, isolated network default deployment
2. rerun the overcloud deploy and include (or change) /usr/share/openstack-tripleo-heat-templates/environments/config-debug.yaml 

For ex.:
$ source /home/stack/stackrc; time openstack overcloud deploy --log-file overcloud_deployment_48.log --templates --libvirt-type=qemu --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal --timeout=240 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e ~/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/config-debug.yaml

3. wait a lot
4. login to one controller as root
5. time /usr/libexec/os-refresh-config/configure.d/40-hiera-datafiles # -> 22 sec
6. time strace -Ff -e execve,clone,fork -v -ttt -s 4096  -o /tmp/execve-hiera  /usr/libexec/os-refresh-config/configure.d/40-hiera-datafiles # -> 44 sec

Actual results:
40-hiera-datafiles executed in 22 sec without tracing, 44 sec with trace

Expected results:
40-hiera-datafiles executes in 2.2 sec.

Additional info:
40-hiera-datafiles cloud be written in language which does not needs multiple external python process to complete. For example it can be written in python with anyjson, ujson usage.
Alternatively the external command os-apply-config must be significantly faster,
for ex. written in C or go.

Because at one execution, the external commands frequently reads the same file content multiple times, makes more sense to implement 40-hiera-datafiles in way which avoids unnecessary rereads.

Comment 2 Mike Burns 2016-04-07 21:00:12 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 4 Jaromir Coufal 2016-10-10 03:22:31 UTC
Obsolete, please re-open if re-appears.