Description of problem: Deployed the latest passed phase 1 build 2016-11-04.2 and ran some rally benchmarks. Everything seemed fine including the undercloud in terms of resource utilization. However, on an idle undercloud/overcloud ironci-conductor grows in memory abruptly to about 93 GB eating up all available memory and causing the undercloud to swap. There is nothing too interesting in the ironic logs at the same time (16:00 UTC on 11/8/16) except for the fact that it also complains about no free memory as can be seen here http://pastebin.test.redhat.com/428458.

Below is the link to the grafana dashboard showing ironic-conductor sudden memory spike:
http://norton.perf.lab.eng.rdu.redhat.com:3000/dashboard/snapshot/05rRcnI3560IH40bpBUSwkFG8OMkN9Dc

The entire set of dashboards showing undercloud resource utilization and OpenStack process resource utilization can be found at:

http://norton.perf.lab.eng.rdu.redhat.com:3000/dashboard/db/openstack-general-system-performance?var-Cloud=sai-latest&var-NodeType=*&var-Node=undercloud&var-Interface=interface-br-ctlplane&var-Disk=disk-sda&var-cpus0=All&var-cpus00=All&from=1478613866952&to=1478667866000

Version-Release number of selected component (if applicable):
RHOP 10 Build 2016-11-04.2

How reproducible: Happened once and remained the same way


Steps to Reproduce:
1. Deploy undercloud
2. Deploy overcloud
3.

Actual results: There's a sudden spike in ironic-conductor memory.


Expected results: This sort of sudden memory surge leading to undercloud swapping shouldn't happen.


Additional info: Ironic and mistral logs attached