Bug 1261512
| Summary: | heat stack create error "Data too long for column 'resource_properties' at row 1" | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marios Andreou <mandreou> | ||||
| Component: | openstack-heat | Assignee: | Zane Bitter <zbitter> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Amit Ugol <augol> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 7.0 (Kilo) | CC: | ddomingo, flfuchs, jcoufal, mandreou, mburns, ohochman, rhel-osp-director-maint, rhosp-bugs-internal, sbaker, shardy, yeylon | ||||
| Target Milestone: | z2 | Keywords: | ZStream | ||||
| Target Release: | 7.0 (Kilo) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-heat-2015.1.1-2.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
On each resource event, the Orchestration service stores the current properties of the resource to enable debugging and auditing of the stack's history. However, the Orchestration service can only store 64kB (encoded) of such information. As such, if a stack contained a resource with particularly large properties (for example, a SoftwareConfig with a large script), the Orchestration service could not record events for the resource in questions, thereby preventing the stack from being created.
With this release, properties that are too large to store in the event table are elided. This, in turn, allows users to create stacks regardless of the size of each resources' properties.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-10-08 12:21:09 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Marios Andreou
2015-09-09 13:46:54 UTC
upstream bug report at https://bugs.launchpad.net/heat/+bug/1493858 We have basically agreed on a medium-term upstream fix: https://review.openstack.org/#/c/222352/ Care to add steps to reproduce please? Hi Amit, this was happening on all deploys until the fix landed (i.e. didn't need to do anything to reproduce, just deploy). Looks like the band aid landed already (openstack-heat-2015.1.1-2.el7ost fixed in version above, not sure if that is correct) so may not be able to reproduce anymore, thanks, marios I reproduced this by just generating a template with a huge property and trying to create a stack with it. This Python code should create something you can test with:
templ_data = {
'heat_template_version': '2013-05-23',
'resources': {
'huge': {
'type': 'OS::Heat::SoftwareConfig',
'properties': {'config': 'A' * (1 << 16)},
}
}
}
with open('huge.yaml', 'w') as op:
import yaml
op.write(yaml.safe_dump(templ_data))
The original fix is replaced with the upstream version (which makes it easier for clients to detect the missing data) in openstack-heat-2015.1.1-4.el7ost above example with 16b works well. 1MB is the limit and above it the user gets a JSON size limit as expected. I still found some occurrences of the "Data too long for column 'resource_properties' at row 1" error in a current undercloud heat log. (HA deployments from ui are failing on that undercloud).
See the attached file. ("grep 'oslo_db' /var/log/heat/heat-engine.log > heat-engine-oslo_db.log")
The heat packages currently installed:
openstack-heat-engine-2015.1.1-4.el7ost.noarch
openstack-heat-api-2015.1.1-4.el7ost.noarch
Created attachment 1076549 [details]
Log entries from heat-engine.log containing 'oslo_db'
We now handle this issue by catching the exception and retrying, so it's possible that something is just logging the exception even though it is handled appropriately. So I wouldn't necessarily read too much into the presence of those log entried. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:1865 Note to avoid confusion: the log message described here *always* appears in the logs for TripleO. If you see it, that does *not* mean that the bug is still a problem. We now catch and handle the exception appropriately (by storing the event but omitting the huge data), but not before it is logged. |