Bug 1381561 - 'Expression consumed too much memory' at overcloud deploy
Summary: 'Expression consumed too much memory' at overcloud deploy
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-heat
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: 10.0 (Newton)
Assignee: Jon Schlueter
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-04 12:46 UTC by Attila Fazekas
Modified: 2023-02-22 23:02 UTC (History)
14 users (show)

Fixed In Version: puppet-heat-9.4.0-1.1.el7ost
Doc Type: Bug Fix
Doc Text:
The OpenStack Platform director exceeded the default memory limits for using OpenStack Orchestration (heat) YAQL expressions. This caused an "Expression consumed too much memory" error during an overcloud deployment and subsequent deployment failure. This fix increases the default memory limits for the director, which results in a error-free overcloud deployment.
Clone Of:
Environment:
Last Closed: 2016-12-14 16:07:29 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 378639 0 None None None 2016-10-04 13:41:27 UTC
Red Hat Product Errata RHEA-2016:2948 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC

Description Attila Fazekas 2016-10-04 12:46:09 UTC 
Description of problem:

With current  2016-10-04.1 snapshot, we got the above error from heat when we try to deploy the overcloud.


11:37:48 stdout: 2016-10-04 11:37:38Z [42]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [69]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [46]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [29]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [41]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [8]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [68]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [64]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [30]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [35]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [11]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [36]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [54]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [13]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [18]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [26]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:38Z [14]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [55]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [15]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [67]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [31]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [0]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [10]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [5]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [4]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [12]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [19]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [71]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [40]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [16]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [57]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [7]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [1]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [51]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [44]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [33]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [45]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [61]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [63]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [60]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [2]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [20]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [70]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [23]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [34]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [50]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [6]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [47]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [43]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [38]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [59]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [9]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:39Z [overcloud-ControllerServiceChain-3ngdcv5gekmz-ServiceChain-tf67457ua6cu]: CREATE_COMPLETE  Stack CREATE completed successfully
11:37:48 2016-10-04 11:37:40Z [ServiceChain]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:40Z [overcloud-ControllerServiceChain-3ngdcv5gekmz]: CREATE_COMPLETE  Stack CREATE completed successfully
11:37:48 2016-10-04 11:37:40Z [ControllerServiceChain]: CREATE_COMPLETE  state changed
11:37:48 2016-10-04 11:37:46Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Expression consumed too much memory
11:37:48 
11:37:48  Stack overcloud CREATE_FAILED 

The host is el7.3 beta, and it is our usual 1 cpu, 1 cnt, 1 ceph setup.

List of packages:
https://paste.fedoraproject.org/443155/47558504/



The 'Expression consumed too much memory' string found in the 'yaql' library,
we expected to have change merged which increased the memory limit.

The heat-engine log also has 'Expression consumed too much memory' string, but without any detail.
Comment 2 Attila Fazekas 2016-10-04 12:53:29 UTC 
Our version expected to be new enough to have this change:
https://github.com/openstack/instack-undercloud/commit/55ccd0e1e8c66c9f474112358063bc263720d84f  (not verifed)
Comment 3 Marius Cornea 2016-10-04 13:29:45 UTC 
(In reply to Attila Fazekas from comment #2)
> Our version expected to be new enough to have this change:
> https://github.com/openstack/instack-undercloud/commit/
> 55ccd0e1e8c66c9f474112358063bc263720d84f  (not verifed)

The fix appears to be there but the error still shows up. I hit it with 1 controller + 1 compute:

grep heat::yaql_memory_quota  /usr/share/instack-undercloud/puppet-stack-config/puppet-stack-config.yaml.template
heat::yaql_memory_quota: 100000

instack-undercloud.noarch                                                                              5.0.0-0.20160929132429.56e65af.el7ost
Comment 4 Zane Bitter 2016-10-04 13:41:28 UTC 
I gather that y'all have this instack-undercloud change:

https://review.openstack.org/377782

but are probably missing the corresponding puppet-heat change:

https://review.openstack.org/378639
Comment 5 Jon Schlueter 2016-10-04 13:43:53 UTC 
(In reply to Zane Bitter from comment #4)
> I gather that y'all have this instack-undercloud change:
> 
> https://review.openstack.org/377782
> 
> but are probably missing the corresponding puppet-heat change:
> 
> https://review.openstack.org/378639

yep that's it, working on build for it.
Comment 6 Alexandre Maumené 2016-10-05 07:41:23 UTC 
Hi,

With:
puppet-heat-9.4.0-1.1.el7ost.noarch
instack-undercloud-5.0.0-0.20160929132429.56e65af.el7ost.noarch

Result is still:
2016-10-05 07:36:38Z [overcloud-ControllerServiceChain-36ecvlqvrx3t-ServiceChain-jbopffivvg74]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-10-05 07:36:39Z [ServiceChain]: CREATE_COMPLETE  state changed
2016-10-05 07:36:39Z [overcloud-ControllerServiceChain-36ecvlqvrx3t]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-10-05 07:36:39Z [ControllerServiceChain]: CREATE_COMPLETE  state changed
2016-10-05 07:36:49Z [CephStorage]: CREATE_IN_PROGRESS  state changed
2016-10-05 07:36:49Z [Compute]: CREATE_IN_PROGRESS  state changed
2016-10-05 07:36:51Z [overcloud-Compute-agk5lybjxlmz]: CREATE_IN_PROGRESS  Stack CREATE started
2016-10-05 07:36:51Z [ObjectStorage]: CREATE_IN_PROGRESS  state changed
2016-10-05 07:36:51Z [1]: CREATE_IN_PROGRESS  state changed
2016-10-05 07:36:52Z [CephStorage]: CREATE_FAILED  CREATE aborted
2016-10-05 07:36:52Z [Compute]: CREATE_FAILED  CREATE aborted
2016-10-05 07:36:52Z [ObjectStorage]: CREATE_FAILED  CREATE aborted
2016-10-05 07:36:52Z [overcloud]: CREATE_FAILED  Resource CREATE failed: Expression consumed too much memory
2016-10-05 07:36:54Z [0]: CREATE_IN_PROGRESS  state changed
2016-10-05 07:36:54Z [overcloud-Compute-agk5lybjxlmz-1-e6nmqr4svmv7]: CREATE_IN_PROGRESS  Stack CREATE started
2016-10-05 07:36:54Z [NodeAdminUserData]: CREATE_IN_PROGRESS  state changed
2016-10-05 07:36:54Z [UpdateConfig]: CREATE_IN_PROGRESS  state changed
2016-10-05 07:36:55Z [NodeUserData]: CREATE_IN_PROGRESS  state changed

 Stack overcloud CREATE_FAILED 

Heat Stack create failed.
Comment 7 Thierry Vignaud 2016-10-05 07:50:49 UTC 
Moving back to dev stage then
Comment 8 Marius Cornea 2016-10-05 07:55:53 UTC 
(In reply to Alexandre Maumené from comment #6)
> Hi,
> 
> With:
> puppet-heat-9.4.0-1.1.el7ost.noarch
> instack-undercloud-5.0.0-0.20160929132429.56e65af.el7ost.noarch
> 
> Result is still:
> 2016-10-05 07:36:38Z
> [overcloud-ControllerServiceChain-36ecvlqvrx3t-ServiceChain-jbopffivvg74]:
> CREATE_COMPLETE  Stack CREATE completed successfully
> 2016-10-05 07:36:39Z [ServiceChain]: CREATE_COMPLETE  state changed
> 2016-10-05 07:36:39Z [overcloud-ControllerServiceChain-36ecvlqvrx3t]:
> CREATE_COMPLETE  Stack CREATE completed successfully
> 2016-10-05 07:36:39Z [ControllerServiceChain]: CREATE_COMPLETE  state changed
> 2016-10-05 07:36:49Z [CephStorage]: CREATE_IN_PROGRESS  state changed
> 2016-10-05 07:36:49Z [Compute]: CREATE_IN_PROGRESS  state changed
> 2016-10-05 07:36:51Z [overcloud-Compute-agk5lybjxlmz]: CREATE_IN_PROGRESS 
> Stack CREATE started
> 2016-10-05 07:36:51Z [ObjectStorage]: CREATE_IN_PROGRESS  state changed
> 2016-10-05 07:36:51Z [1]: CREATE_IN_PROGRESS  state changed
> 2016-10-05 07:36:52Z [CephStorage]: CREATE_FAILED  CREATE aborted
> 2016-10-05 07:36:52Z [Compute]: CREATE_FAILED  CREATE aborted
> 2016-10-05 07:36:52Z [ObjectStorage]: CREATE_FAILED  CREATE aborted
> 2016-10-05 07:36:52Z [overcloud]: CREATE_FAILED  Resource CREATE failed:
> Expression consumed too much memory
> 2016-10-05 07:36:54Z [0]: CREATE_IN_PROGRESS  state changed
> 2016-10-05 07:36:54Z [overcloud-Compute-agk5lybjxlmz-1-e6nmqr4svmv7]:
> CREATE_IN_PROGRESS  Stack CREATE started
> 2016-10-05 07:36:54Z [NodeAdminUserData]: CREATE_IN_PROGRESS  state changed
> 2016-10-05 07:36:54Z [UpdateConfig]: CREATE_IN_PROGRESS  state changed
> 2016-10-05 07:36:55Z [NodeUserData]: CREATE_IN_PROGRESS  state changed
> 
>  Stack overcloud CREATE_FAILED 
> 
> Heat Stack create failed.

Do you get the 'Expression consumed too much memory' message in the FAIL step? This BZ was targeting this particular error. If there's another issue we should open a different BZ for it.
Comment 9 Alexandre Maumené 2016-10-05 07:57:04 UTC 
Ok I think I posted to quickly... I didn't run "openstack undercloud install" to update the parameters in /etc/heat/heat.conf after updating the packages. It should be fixed and I'm testing it atm. I'll update this BZ to confirm.
Comment 10 Thierry Vignaud 2016-10-05 09:20:48 UTC 
Moving back to MODIFIED then
Comment 11 Alexandre Maumené 2016-10-05 09:22:27 UTC 
I confirm that I'm not hitting this issue any more after updating the packages. Thanks.
Comment 14 errata-xmlrpc 2016-12-14 16:07:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html
Note You need to log in before you can comment on or make changes to this bug.