Created attachment 1471196 [details]
Journal entries showing failed update (monitor failed to restart)

During a fast-forward upgrade, the 'openstack overcloud upgrade run --roles Controller --skip-tags validation' command updates both Ceph and OSP RPMs.  On slower systems, the Ceph monitor service can fail to restart.

The sequence seems to be:

1. First "wave" of Ceph RPMs are updated - libradosstriper1, ceph-common,
   ceph-base, ceph-selinux.

2. ceph-selinux update causes restart of all Ceph services (ceph-mon in
   this case).

3. Because the ceph-mon package hasn't yet been updated, it fails with an
   undefined symbol error.

4. Many other packages are updated ...

5. systemd tries to start the Ceph monitor several times, but it keeps
   failing (see #3).  Finally, systemd gives up.

6. More packages are updated ...

7. Eventually the remaining Ceph packages are updated - ceph-radosgw,
   ceph-mon, and ceph-osd.  Now the Ceph monitor could be started, but
   it's too late; systemd has already given up.

When the Ceph RPMs are updated by themselves, the undefined symbol still occurs, but the ceph-mon package then gets updated and the services starts successfully on the next attempt.

So this is a problem for the OSP FFU/upgrade workflow because of the large number of packages that are updated at the same time as the Ceph packages.  (It's also exposed by running on slower systems.)