Bug 1259905 - Integrate yum updates of overcloud with Puppet
Integrate yum updates of overcloud with Puppet
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity high
: y1
: 7.0 (Kilo)
Assigned To: Jay Dobies
Alexander Chuzhoy
:
: 1235705 (view as bug list)
Depends On: 1259900 1261921
Blocks: 1257642 1262189
  Show dependency treegraph
 
Reported: 2015-09-03 14:38 EDT by Zane Bitter
Modified: 2015-10-19 09:36 EDT (History)
11 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-57.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-08 08:18:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 190918 None None None Never
OpenStack gerrit 193394 None None None Never
OpenStack gerrit 194348 None None None Never

  None (edit)
Description Zane Bitter 2015-09-03 14:38:21 EDT
Puppet is responsible for managing many of the packages on the system, so when doing a yum update on the overcloud we should leave that part to Puppet to ensure that services are e.g. restarted in the correct order.

There are upstream patches to implement this:

https://review.openstack.org/#/c/194348 (Write package names out to flat files)

This allows us to make a list of packages that are Puppet's responsibility. Note that this depends on the changes for bug 1259900.

https://review.openstack.org/#/c/190918 (Ensure present/latest for puppet driven package updates)

This puts Puppet (instead of yum) in charge of updating packages that it knows about.
Comment 1 Zane Bitter 2015-09-03 14:56:30 EDT
We'll likely also need:

https://review.openstack.org/#/c/193394/ (wire in tripleo::packages)
Comment 3 Zane Bitter 2015-09-18 10:52:47 EDT
Verification here would probably involve looking at the logs on the servers after they've been updated and confirming (1) that none of the packages managed by puppet are updated by the initial call to yum, (2) that puppet runs, and (3) that puppet pulls in any package updates to the packages it is managing.

The ultimate test of this is really the CI that upgrades to the latest puddle and verifies with Tempest that everything still works afterwards.
Comment 4 Zane Bitter 2015-09-18 14:59:51 EDT
We're considering a different approach to this problem in light of the fact that we deploy with Pacemaker. See https://bugzilla.redhat.com/show_bug.cgi?id=1261921#c15
Comment 5 Udi 2015-09-22 03:58:38 EDT
Need clarifications: 
1) Which packages are puppet's responsibility, and how will yum know the difference?
2) Which logs should I be inspecting while the update runs?
3) What should I be interested to find in the logs?
4) What do you mean that we deploy with pacemaker? It's just a part that's in the deployment, right?

There is also a problem to test the updates in the latest puddle, since there is nothing that really needs updating. Is there a recommendation about what to do with that?
Comment 6 Zane Bitter 2015-09-23 09:34:02 EDT
(In reply to Udi from comment #5)
> Need clarifications: 
> 1) Which packages are puppet's responsibility, and how will yum know the
> difference?

All of the packages that Puppet knows about (i.e. they have corresponding modules in Puppet). This includes all of the OpenStack services for a start. Puppet writes lists of these packages in the directory /var/lib/tripleo/installed-packages/. Yum ignores them because we pass them with the --exclude flag when we run yum.

> 2) Which logs should I be inspecting while the update runs?

Yum & Puppet

> 3) What should I be interested to find in the logs?

"confirming (1) that none of the packages managed by puppet are updated by the initial call to yum, (2) that puppet runs, and (3) that puppet pulls in any package updates to the packages it is managing."

> 4) What do you mean that we deploy with pacemaker? It's just a part that's
> in the deployment, right?

I mean that Pacemaker (and not Puppet or systemd) manages the starting and stopping of services. So on controller nodes, we're ditching the idea of having Puppet update stuff in order to control the restart ordering - instead we'll have yum update everything and stop Pacemaker (and all the services it manages) before and restart it after. It's worth reading all the comments on bug 1261921, and in fact there is probably no point testing this one until the patch for that has landed, since it completely changes the update script.

> There is also a problem to test the updates in the latest puddle, since
> there is nothing that really needs updating. Is there a recommendation about
> what to do with that?

Maybe do the same as the CI - install the GA bits and then update to the latest puddle. (Unfortunately this probably involves installing the GA undercloud as well and then updating it first. It would be nice to be able to install just the old overcloud with a new undercloud and then only update the overcloud, but that's probably tricky to get right.)
Comment 7 Alexander Chuzhoy 2015-10-02 11:11:56 EDT
Verified:

Environment:
openstack-tripleo-heat-templates-0.8.6-71.el7ost.noarch

So this narrows down to checking the compute nodes.
Checked a compute node and verified that the packages meant to be excluded during stack update, were actually excluded and were updated later by puppet.
Comment 9 errata-xmlrpc 2015-10-08 08:18:00 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1862
Comment 10 Mike Burns 2015-10-19 09:36:41 EDT
*** Bug 1235705 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.