Bug 1501852 - "puppet apply" failures are not reported back to heat
Summary: "puppet apply" failures are not reported back to heat
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 12.0 (Pike)
Assignee: Martin André
QA Contact: Artem Hrechanychenko
Depends On:
Blocks: 1494455 1505424
TreeView+ depends on / blocked
Reported: 2017-10-13 10:51 UTC by Damien Ciabrini
Modified: 2018-02-05 19:15 UTC (History)
12 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.3-4.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-12-13 22:13:59 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Launchpad 1723163 0 None None None 2017-10-13 10:52:43 UTC
OpenStack gerrit 513242 0 None None None 2017-11-08 10:03:14 UTC
OpenStack gerrit 513244 0 None None None 2017-11-08 10:06:25 UTC
OpenStack gerrit 513261 0 None None None 2017-10-20 09:02:56 UTC
OpenStack gerrit 514727 0 None None None 2017-11-08 10:01:14 UTC
OpenStack gerrit 514728 0 None None None 2017-11-08 10:07:58 UTC
OpenStack gerrit 515326 0 None None None 2017-11-08 10:09:39 UTC
OpenStack gerrit 517022 0 None None None 2017-11-09 07:25:52 UTC
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Damien Ciabrini 2017-10-13 10:51:11 UTC
Description of problem:
In Pike puppet runs are triggered differently than before, and they currently fail to report failure code to the callers.

Unless puppet is called with --detailed-exitcodes, it may fail to report error code when puppet Resources go in error [1].

[1] https://tickets.puppetlabs.com/browse/PUP-2754

How reproducible:

Actual results:
The error is not catch and the deployment continues unexpectedly and fails much later for unrelated reasons.

Expected results:
The overcloud deployment should fail if any resource fail unexpectedly during the puppet run.

Additional info:

Comment 2 Alex Schultz 2017-10-13 21:51:00 UTC
Specifically the impact for OSP is that the docker-puppet.py script was not properly using --detailed-exitcodes like the heat agent does.

Comment 3 Omri Hochman 2017-10-18 13:30:36 UTC
Adding blocker - as this Bz is blocking #1494455 , which blocks OSP12 scenario for : "Controller replacement" .

Comment 4 Fabio Massimo Di Nitto 2017-10-19 10:12:59 UTC

what is the rationale for assigning this bug back to PIDONE? this is a generic framework issue.

BTW it´s not the first I notice this ping pong with bugs towards PIDONE. I would like to remind that when you reassign a but to another DFG, you need to explain why you believe that it is somebody else problem.

Comment 5 Fabio Massimo Di Nitto 2017-10-19 10:14:04 UTC

similar to you as comment #4, why was the bug bounced to DFG:Containers without any explanation?

Comment 6 Fabio Massimo Di Nitto 2017-10-19 10:26:31 UTC
After talking to Michele and Damien, this is an issue with the framework in general and not specific to PIDONE. While PIDONE is helping already fixing the problem (as Michele mentioned on IRC) the bug is DF responsibility as originally identified by Mike Burns.

Comment 7 Omri Hochman 2017-10-19 12:52:15 UTC
(In reply to Fabio Massimo Di Nitto from comment #4)
> Omri,
> what is the rationale for assigning this bug back to PIDONE? this is a
> generic framework issue.
> BTW it´s not the first I notice this ping pong with bugs towards PIDONE. I
> would like to remind that when you reassign a but to another DFG, you need
> to explain why you believe that it is somebody else problem.

Thanks Fabio,  Sure will do for next time.

Comment 8 Alex Schultz 2017-10-19 14:08:21 UTC
So I moved it to DFG:Containers because the issue is with docker-puppet.py (as indicated in comment #2) and this was created by the containers team and is only necessary for the container effort so it was a bug in their feature.  So moving it back once again.  The DFG is the default assignee for THT but all DFGs have some elements they own within this component. TripleO components are special such that it may not always be DFG:DF but we end up with it by default.

Comment 9 Michele Baldessari 2017-10-20 09:02:57 UTC
So here is a current recap of the reviews needed here:
https://review.openstack.org/#/c/513261/ puppet-tripleo - tuned-adm fix (merged in pike)
https://review.openstack.org/#/c/512033/ tht - swift xined (merged in pike)
https://review.openstack.org/#/c/512082/ puppet-pacemaker rule order revert (merged and no stable branches)
https://review.openstack.org/#/c/511509/ tht - Add --detailed-exitcodes  not merged yet because we are trying to fix all the additional CI breakages that this is causing

Comment 12 Damien Ciabrini 2017-11-08 10:12:51 UTC
Quick status update, almost all reviews have merged in pike upstream

We still need 511509 backport [1] to merge in pike upstream, then we'll be good to move to MODIFIED.

[1] https://review.openstack.org/#/c/517022

Comment 13 Damien Ciabrini 2017-11-09 07:45:56 UTC
Quick explanation for all the tracked reviews

https://review.openstack.org/517022 has just landed in pike and is the main fix for this bugzilla.

The other tracked reviews are all the fixes in pike for the bugs that are uncovered once the main fix is in use.

Comment 14 Michele Baldessari 2017-11-09 08:38:34 UTC
Here is a recap of all the reviews:
https://review.openstack.org/#/c/517022 - tht - Add --detailed-exitcodes when running puppet via ansible (stable/pike)
https://review.openstack.org/#/c/515326 - tht - Run containerized mistral-api eventlet  (stable/pike)
https://review.openstack.org/#/c/514728 - puppet-tripleo - Add option to disable running mistral-api via wsgi (stable/pike)
https://review.openstack.org/#/c/514727 - tht - Disable MongoDB in scenario002  (stable/pike)
https://review.openstack.org/#/c/513261 - puppet-tripleo - Make sure tuned package is installed before calling tuned-adm (stable/pike)
https://review.openstack.org/#/c/513244 - tht - Disable SwiftDispersion when using docker (stable/pike)
https://review.openstack.org/#/c/513242 - tht - Disable xinetd class when creating swift-storage puppet configuration (stable/pike)

Note that we already ship puppet-pacemaker-0.6.1-0.20171024215340.9a46ecd.el7ost.src.rpm so the revert at
I607927c2ee5c29b605e18e9294b0f91d37337680 is not needed, hence we're not even linking it.

Moving this one to POST now.

Comment 15 Artem Hrechanychenko 2017-11-13 15:00:20 UTC
Hi folks,

I see that all of patches from https://bugzilla.redhat.com/show_bug.cgi?id=1501852#c14 was merge

Comment 19 Artem Hrechanychenko 2017-11-17 19:00:03 UTC


Comment 22 errata-xmlrpc 2017-12-13 22:13:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.