Bug 1349890
| Summary: | rhel-osp-director: 9.0 Update to async fails - os-collect-config restarted during yum update | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | os-collect-config | Assignee: | Ben Nemec <bnemec> |
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 9.0 (Mitaka) | CC: | apevec, dbecker, jcoufal, jjoyce, jstransk, lhh, mburns, morazi, nlevinki, ohochman, rhel-osp-director-maint, sasha, srevivo, tvignaud |
| Target Milestone: | ga | Keywords: | Triaged |
| Target Release: | 9.0 (Mitaka) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | os-collect-config-0.1.37-5.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-08-11 11:33:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Reproduced. Takes a long time until the update fails. Hi Sasha, as promised and just in time for 'Wednesday' squeezed in a quick first look&triage today. I think the error you have pasted about the unset update identifier is a red herring (notice it is stdout and not stderr). During minor update UpdateIdentifier is only set when it is time for the given node to be updated. Looking at controller-0 for example /var/log/messages, there is one occurrence of it being unset but then another where the update occurs here:
Jun 23 19:09:45 localhost os-collect-config: [2016-06-23 19:09:45,400] (heat-config) [INFO] {"deploy_stdout": "Started yum_update.sh on server e85cb0d5-f5cd-4e81-b96b-6d451cc9ce5e at Thu Jun 23 19:09:45 EDT 2016\nNot running due to unset update_identifier\n", "deploy_stderr": "", "update_managed_packages": "false", "deploy_status_code": 0}
...
Jun 24 01:47:40 overcloud-controller-0 os-collect-config: [2016-06-24 01:47:40,348] (heat-config) [INFO] {"deploy_stdout": "Started yum_update.sh on server e85cb0d5-f5cd-4e81-b96b-6d451cc9ce5e at Fri Jun 24 01:40:44 UTC 2016\nPacemaker running, stopping cluster node and doing full package update\nStopping Cluster (pacemaker)... Stopping Cluster (corosync)...\nRunning: yum -q -y update \nDelta RP
I also see the following in the trace, which is https://bugzilla.redhat.com/show_bug.cgi?id=1343905 afaics (also OSP9 minor update and is fixed upstream in the resource agents). Not clear this is the root yet:
Jun 24 01:45:24 overcloud-controller-0 rabbitmq-cluster(rabbitmq)[19434]: INFO: node failed to join even after reseting local data. Check SELINUX policy
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ Error: {aborted,{no_exists,[rabbit_user, ]
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ [{{internal_user,'$1','_','_'}, ]
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ [{'/=','$1',<<"guest">>}], ]
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ ['$_']}]]}} ]
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]: notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
Jun 24 01:45:24 overcloud-controller-0 crmd[18508]: notice: Operation rabbitmq_start_0: unknown error (node=overcloud-controller-0, call=245, rc=1, cib-update=66, confirmed=true)
Jun 24 01:45:24 overcloud-controller-0 crmd[18508]: notice: overcloud-controller-0-rabbitmq_start_0:245 [ User defined signal 2\nUser defined signal 2\nError: {aborted,{no_exists,[rabbit_user,\n [{{internal_user,'$1','_','_'},\n [{'/=','$1',<<"guest">>}],\n ['$_']}]]}}\nUser defined signal 2\nUser defined signal 2\n ]
Jun 24 01:45:25 overcloud-controller-0 galera(galera)[23083]: ERROR: MySQL is not running
As always with these bugs can you also please include the full deployment and update commands you used (can't recover these from the sos files, though I could guess but you could also just tell me :) )
thanks, marios
I noticed that yum update ran on only one machine.
I see the following on it:
[root@overcloud-controller-2 ~]# yum history
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
14 | System <unset> | 2016-06-30 13:50 | I, U | 108 **
13 | Cloud User <heat-admin> | 2016-06-30 13:41 | Install | 1
12 | Cloud User <heat-admin> | 2016-06-30 13:41 | Install | 1
11 | root <root> | 2016-06-16 18:19 | Install | 1
10 | root <root> | 2016-06-16 18:18 | Install | 7 EE
9 | root <root> | 2016-06-16 18:17 | Install | 1
8 | root <root> | 2016-06-16 18:15 | Install | 35
7 | root <root> | 2016-06-16 17:38 | Install | 622 EE
6 | root <root> | 2016-06-16 17:33 | Install | 2
5 | root <root> | 2016-06-16 17:23 | Install | 119
4 | root <root> | 2016-06-16 17:22 | Install | 2
3 | root <root> | 2016-06-16 17:22 | Erase | 1
2 | root <root> | 2016-06-16 17:21 | Install | 13
1 | System <unset> | 2016-06-16 16:36 | Install | 338
Warning: RPMDB altered outside of yum.
** Found 63 pre-existing rpmdb problem(s), 'yum check' output follows:
1:NetworkManager-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-1.0.6-29.el7_2.x86_64
1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-config-server-1.0.6-29.el7_2.x86_64
1:NetworkManager-libnm-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-libnm-1.0.6-29.el7_2.x86_64
1:NetworkManager-team-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-team-1.0.6-29.el7_2.x86_64
1:NetworkManager-tui-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-tui-1.0.6-29.el7_2.x86_64
corosync-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosync-2.3.4-7.el7_2.1.x86_64
corosynclib-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosynclib-2.3.4-7.el7_2.1.x86_64
7:device-mapper-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-1.02.107-5.el7_2.2.x86_64
7:device-mapper-event-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-1.02.107-5.el7_2.2.x86_64
7:device-mapper-event-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64
7:device-mapper-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-libs-1.02.107-5.el7_2.2.x86_64
device-mapper-multipath-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-0.4.9-85.el7_2.4.x86_64
device-mapper-multipath-libs-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64
dracut-033-360.el7_2.1.x86_64 is a duplicate with dracut-033-360.el7_2.x86_64
dracut-config-generic-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-generic-033-360.el7_2.x86_64
dracut-config-rescue-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-rescue-033-360.el7_2.x86_64
glusterfs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-3.7.5-19.el7.x86_64
glusterfs-api-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-api-3.7.5-19.el7.x86_64
glusterfs-client-xlators-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-client-xlators-3.7.5-19.el7.x86_64
glusterfs-libs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-libs-3.7.5-19.el7.x86_64
iproute-3.10.0-54.el7_2.1.x86_64 is a duplicate with iproute-3.10.0-54.el7.x86_64
iscsi-initiator-utils-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-6.2.0.873-32.el7.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64
kernel-headers-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-headers-3.10.0-327.18.2.el7.x86_64
kernel-tools-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-3.10.0-327.18.2.el7.x86_64
kernel-tools-libs-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-libs-3.10.0-327.18.2.el7.x86_64
kpartx-0.4.9-85.el7_2.5.x86_64 is a duplicate with kpartx-0.4.9-85.el7_2.4.x86_64
libgudev1-219-19.el7_2.11.x86_64 is a duplicate with libgudev1-219-19.el7_2.9.x86_64
libvirt-client-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-client-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-lxc-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64
libxml2-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-2.9.1-6.el7_2.2.x86_64
libxml2-devel-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-devel-2.9.1-6.el7_2.2.x86_64
libxml2-python-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-python-2.9.1-6.el7_2.2.x86_64
7:lvm2-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-2.02.130-5.el7_2.2.x86_64
7:lvm2-libs-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-libs-2.02.130-5.el7_2.2.x86_64
1:nfs-utils-1.3.0-0.21.el7_2.1.x86_64 is a duplicate with 1:nfs-utils-1.3.0-0.21.el7_2.x86_64
1:openstack-nova-cert-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-cert-13.0.0-8.el7ost.noarch
1:openstack-nova-common-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-common-13.0.0-8.el7ost.noarch
1:openstack-nova-conductor-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-conductor-13.0.0-8.el7ost.noarch
1:openstack-nova-scheduler-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-scheduler-13.0.0-8.el7ost.noarch
os-apply-config-0.1.32-6.el7ost.noarch is a duplicate with os-apply-config-0.1.32-5.el7ost.noarch
os-collect-config-0.1.37-4.el7ost.noarch is a duplicate with os-collect-config-0.1.37-3.el7ost.noarch
os-refresh-config-0.1.11-4.el7ost.noarch is a duplicate with os-refresh-config-0.1.11-3.el7ost.noarch
polkit-0.112-7.el7_2.x86_64 is a duplicate with polkit-0.112-6.el7_2.x86_64
python-gnocchi-2.1.3-2.el7ost.noarch is a duplicate with python-gnocchi-2.1.3-1.el7ost.noarch
1:python-nova-13.1.0-1.el7ost.noarch is a duplicate with 1:python-nova-13.0.0-8.el7ost.noarch
python-oslo-concurrency-3.7.1-1.el7ost.noarch is a duplicate with python-oslo-concurrency-3.6.0-1.el7ost.noarch
python-perf-3.10.0-327.22.2.el7.x86_64 is a duplicate with python-perf-3.10.0-327.18.2.el7.x86_64
python-webtest-1.3.4-6.el7.noarch is a duplicate with python-webtest-1.3.4-5.el7.noarch
rpcbind-0.2.0-33.el7_2.1.x86_64 is a duplicate with rpcbind-0.2.0-33.el7_2.x86_64
selinux-policy-3.13.1-60.el7_2.7.noarch is a duplicate with selinux-policy-3.13.1-60.el7_2.3.noarch
systemd-219-19.el7_2.11.x86_64 is a duplicate with systemd-219-19.el7_2.9.x86_64
systemd-libs-219-19.el7_2.11.x86_64 is a duplicate with systemd-libs-219-19.el7_2.9.x86_64
systemd-sysv-219-19.el7_2.11.x86_64 is a duplicate with systemd-sysv-219-19.el7_2.9.x86_64
tzdata-2016e-1.el7.noarch is a duplicate with tzdata-2016d-1.el7.noarch
history list
(In reply to Alexander Chuzhoy from comment #5) > I noticed that yum update ran on only one machine. > > I see the following on it: > [root@overcloud-controller-2 ~]# yum history Hi Sasha, I am not entirely clear what you mean here. Do you mean the package update only ran on controller-2? Because from my comment #4 you can see it ran on controller-0. Do you mean you tried to reproduce and on this second attempt 'noticed that yum update ran on only one machine'? As always with these bugs can you also please include the full deployment and update commands you used (can't recover these from the sos files, though I could guess but you could also just tell me :) ) I will poke a bit at the environment you sent thanks, marios > Loaded plugins: product-id, search-disabled-repos, subscription-manager > This system is not registered to Red Hat Subscription Management. You can > use subscription-manager to register. > ID | Login user | Date and time | Action(s) | > Altered > ----------------------------------------------------------------------------- > -- > 14 | System <unset> | 2016-06-30 13:50 | I, U | 108 > ** > 13 | Cloud User <heat-admin> | 2016-06-30 13:41 | Install | 1 > > 12 | Cloud User <heat-admin> | 2016-06-30 13:41 | Install | 1 > > 11 | root <root> | 2016-06-16 18:19 | Install | 1 > > 10 | root <root> | 2016-06-16 18:18 | Install | 7 > EE > 9 | root <root> | 2016-06-16 18:17 | Install | 1 > > 8 | root <root> | 2016-06-16 18:15 | Install | 35 > > 7 | root <root> | 2016-06-16 17:38 | Install | 622 > EE > 6 | root <root> | 2016-06-16 17:33 | Install | 2 > > 5 | root <root> | 2016-06-16 17:23 | Install | 119 > > 4 | root <root> | 2016-06-16 17:22 | Install | 2 > > 3 | root <root> | 2016-06-16 17:22 | Erase | 1 > > 2 | root <root> | 2016-06-16 17:21 | Install | 13 > > 1 | System <unset> | 2016-06-16 16:36 | Install | 338 > > Warning: RPMDB altered outside of yum. > > ** Found 63 pre-existing rpmdb problem(s), 'yum check' output follows: > > 1:NetworkManager-1.0.6-30.el7_2.x86_64 is a duplicate with > 1:NetworkManager-1.0.6-29.el7_2.x86_64 > 1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64 is a duplicate with > 1:NetworkManager-config-server-1.0.6-29.el7_2.x86_64 > 1:NetworkManager-libnm-1.0.6-30.el7_2.x86_64 is a duplicate with > 1:NetworkManager-libnm-1.0.6-29.el7_2.x86_64 > 1:NetworkManager-team-1.0.6-30.el7_2.x86_64 is a duplicate with > 1:NetworkManager-team-1.0.6-29.el7_2.x86_64 > 1:NetworkManager-tui-1.0.6-30.el7_2.x86_64 is a duplicate with > 1:NetworkManager-tui-1.0.6-29.el7_2.x86_64 > corosync-2.3.4-7.el7_2.3.x86_64 is a duplicate with > corosync-2.3.4-7.el7_2.1.x86_64 > corosynclib-2.3.4-7.el7_2.3.x86_64 is a duplicate with > corosynclib-2.3.4-7.el7_2.1.x86_64 > 7:device-mapper-1.02.107-5.el7_2.5.x86_64 is a duplicate with > 7:device-mapper-1.02.107-5.el7_2.2.x86_64 > 7:device-mapper-event-1.02.107-5.el7_2.5.x86_64 is a duplicate with > 7:device-mapper-event-1.02.107-5.el7_2.2.x86_64 > 7:device-mapper-event-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with > 7:device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64 > 7:device-mapper-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with > 7:device-mapper-libs-1.02.107-5.el7_2.2.x86_64 > device-mapper-multipath-0.4.9-85.el7_2.5.x86_64 is a duplicate with > device-mapper-multipath-0.4.9-85.el7_2.4.x86_64 > device-mapper-multipath-libs-0.4.9-85.el7_2.5.x86_64 is a duplicate with > device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64 > dracut-033-360.el7_2.1.x86_64 is a duplicate with > dracut-033-360.el7_2.x86_64 > dracut-config-generic-033-360.el7_2.1.x86_64 is a duplicate with > dracut-config-generic-033-360.el7_2.x86_64 > dracut-config-rescue-033-360.el7_2.1.x86_64 is a duplicate with > dracut-config-rescue-033-360.el7_2.x86_64 > glusterfs-3.7.9-10.el7.x86_64 is a duplicate with > glusterfs-3.7.5-19.el7.x86_64 > glusterfs-api-3.7.9-10.el7.x86_64 is a duplicate with > glusterfs-api-3.7.5-19.el7.x86_64 > glusterfs-client-xlators-3.7.9-10.el7.x86_64 is a duplicate with > glusterfs-client-xlators-3.7.5-19.el7.x86_64 > glusterfs-libs-3.7.9-10.el7.x86_64 is a duplicate with > glusterfs-libs-3.7.5-19.el7.x86_64 > iproute-3.10.0-54.el7_2.1.x86_64 is a duplicate with > iproute-3.10.0-54.el7.x86_64 > iscsi-initiator-utils-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with > iscsi-initiator-utils-6.2.0.873-32.el7.x86_64 > iscsi-initiator-utils-iscsiuio-6.2.0.873-33.el7_2.1.x86_64 is a duplicate > with iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64 > kernel-headers-3.10.0-327.22.2.el7.x86_64 is a duplicate with > kernel-headers-3.10.0-327.18.2.el7.x86_64 > kernel-tools-3.10.0-327.22.2.el7.x86_64 is a duplicate with > kernel-tools-3.10.0-327.18.2.el7.x86_64 > kernel-tools-libs-3.10.0-327.22.2.el7.x86_64 is a duplicate with > kernel-tools-libs-3.10.0-327.18.2.el7.x86_64 > kpartx-0.4.9-85.el7_2.5.x86_64 is a duplicate with > kpartx-0.4.9-85.el7_2.4.x86_64 > libgudev1-219-19.el7_2.11.x86_64 is a duplicate with > libgudev1-219-19.el7_2.9.x86_64 > libvirt-client-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-client-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-lxc-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64 > libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 is a duplicate with > libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64 > libxml2-2.9.1-6.el7_2.3.x86_64 is a duplicate with > libxml2-2.9.1-6.el7_2.2.x86_64 > libxml2-devel-2.9.1-6.el7_2.3.x86_64 is a duplicate with > libxml2-devel-2.9.1-6.el7_2.2.x86_64 > libxml2-python-2.9.1-6.el7_2.3.x86_64 is a duplicate with > libxml2-python-2.9.1-6.el7_2.2.x86_64 > 7:lvm2-2.02.130-5.el7_2.5.x86_64 is a duplicate with > 7:lvm2-2.02.130-5.el7_2.2.x86_64 > 7:lvm2-libs-2.02.130-5.el7_2.5.x86_64 is a duplicate with > 7:lvm2-libs-2.02.130-5.el7_2.2.x86_64 > 1:nfs-utils-1.3.0-0.21.el7_2.1.x86_64 is a duplicate with > 1:nfs-utils-1.3.0-0.21.el7_2.x86_64 > 1:openstack-nova-cert-13.1.0-1.el7ost.noarch is a duplicate with > 1:openstack-nova-cert-13.0.0-8.el7ost.noarch > 1:openstack-nova-common-13.1.0-1.el7ost.noarch is a duplicate with > 1:openstack-nova-common-13.0.0-8.el7ost.noarch > 1:openstack-nova-conductor-13.1.0-1.el7ost.noarch is a duplicate with > 1:openstack-nova-conductor-13.0.0-8.el7ost.noarch > 1:openstack-nova-scheduler-13.1.0-1.el7ost.noarch is a duplicate with > 1:openstack-nova-scheduler-13.0.0-8.el7ost.noarch > os-apply-config-0.1.32-6.el7ost.noarch is a duplicate with > os-apply-config-0.1.32-5.el7ost.noarch > os-collect-config-0.1.37-4.el7ost.noarch is a duplicate with > os-collect-config-0.1.37-3.el7ost.noarch > os-refresh-config-0.1.11-4.el7ost.noarch is a duplicate with > os-refresh-config-0.1.11-3.el7ost.noarch > polkit-0.112-7.el7_2.x86_64 is a duplicate with polkit-0.112-6.el7_2.x86_64 > python-gnocchi-2.1.3-2.el7ost.noarch is a duplicate with > python-gnocchi-2.1.3-1.el7ost.noarch > 1:python-nova-13.1.0-1.el7ost.noarch is a duplicate with > 1:python-nova-13.0.0-8.el7ost.noarch > python-oslo-concurrency-3.7.1-1.el7ost.noarch is a duplicate with > python-oslo-concurrency-3.6.0-1.el7ost.noarch > python-perf-3.10.0-327.22.2.el7.x86_64 is a duplicate with > python-perf-3.10.0-327.18.2.el7.x86_64 > python-webtest-1.3.4-6.el7.noarch is a duplicate with > python-webtest-1.3.4-5.el7.noarch > rpcbind-0.2.0-33.el7_2.1.x86_64 is a duplicate with > rpcbind-0.2.0-33.el7_2.x86_64 > selinux-policy-3.13.1-60.el7_2.7.noarch is a duplicate with > selinux-policy-3.13.1-60.el7_2.3.noarch > systemd-219-19.el7_2.11.x86_64 is a duplicate with > systemd-219-19.el7_2.9.x86_64 > systemd-libs-219-19.el7_2.11.x86_64 is a duplicate with > systemd-libs-219-19.el7_2.9.x86_64 > systemd-sysv-219-19.el7_2.11.x86_64 is a duplicate with > systemd-sysv-219-19.el7_2.9.x86_64 > tzdata-2016e-1.el7.noarch is a duplicate with tzdata-2016d-1.el7.noarch > history list Hi Sasha: just had spent some time on the env you sent me via email. So I answered my own question from comment 6, indeed it looks like yum update only ran on controller-2 - so *must* be a redeployment? Before getting into the why (still not clear) can we please step back and be clear what this bug is about; for example I propose the title is changed since this isn't to do with unset update identifier, afaics and from my comment #4. Furthermore can we have a clear reproducer; deploy with OSP 9 like _foo_ and then update by doing _bar_. For this second run, did you start from a clean environment? Looking at the logs, I can see a few different errors that are from before the yum update on controller-2. Best I can tell and using [1] for reference: cluster stop on control-2 line 54 @[1]: 54028 Jun 30 09:49:21 localhost systemd: Stopped Pacemaker High Availability Cluster Manager. 54029 Jun 30 09:49:21 localhost systemd: Stopping Corosync Cluster Engine... 54044 Jun 30 09:49:21 localhost corosync[6724]: [MAIN ] Corosync Cluster Engine exiting normally package update on control-2 line 68 @[1] 54099 Jun 30 09:50:30 localhost yum[15213]: Updated: glusterfs-libs-3.7.9-10.el7.x86_64 54173 Jun 30 09:50:38 localhost yum[15213]: Updated: 7:lvm2-2.02.130-5.el7_2.5.x86_64 I dont see any errors after this update... just before it there is: 53472:Jun 30 09:47:38 localhost ceilometer-collector: error: [Errno 4] Interrupted system call 53559:Jun 30 09:47:46 localhost nova-conductor: Exception TypeError: "'NoneType' object is not callable" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:4> ignored As i said earlier, there are many *other* errors here but they come from way earlier, June 29th... for example this is in all controllers [2]: 10861:Jun 29 20:45:29 localhost gnocchi-metricd: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 And this cinder error on controllers 1 and 2: 40578:Jun 30 09:19:56 localhost cinder-scheduler: 2016-06-30 13:19:25.163 11498 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1205, u'Lock wait timeout exceeded; try restarting transaction') 40656:Jun 30 09:19:56 localhost cinder-scheduler: 2016-06-30 13:19:56.019 11498 ERROR cinder.service DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') [SQL: u'UPDATE services SET updated_at=%s, report_count=%s WHERE services.id = %s'] [parameters: (datetime.datetime(2016, 6, 30, 13, 18, 34, 872488), 13501, 1)] Finally on controller0 there is also this ceph error - but again, isn't this one day before the yum update on controller-2: 5403:Jun 29 20:37:23 localhost os-collect-config: #033[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.admin]/Exec[ceph-injectkey-client.admin]/unless: Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-overcloud-controller-0/keyring' auth get client.admin | grep AQAtYXRXAAAAABAAFa36Lk0MKfIH8zTjcPtWnw==" exceeded timeout#033[0m 5404:Jun 29 20:37:23 localhost os-collect-config: #033[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.openstack]/Exec[ceph-injectkey-client.openstack]/unless: Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-overcloud-controller-0/keyring' auth get client.openstack | grep AQAtYXRXAAAAABAAz4vP2YaVJrsxX9cvBNNtSQ==" exceeded timeout#033[0m 5405:Jun 29 20:37:23 localhost os-collect-config: #033[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/unless: Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph --name 'mon.' --keyring '/var/lib/ceph/mon/ceph-overcloud-controller-0/keyring' auth get client.bootstrap-osd | grep AQAtYXRXAAAAABAAFa36Lk0MKfIH8zTjcPtWnw==" exceeded timeout#033[0m 12922:Jun 29 20:50:11 localhost gnocchi-metricd: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 So can we please be clear about what we are debugging otherwise I'm really not sure what I'm looking for here. thanks, marios [1] https://github.com/openstack/tripleo-heat-templates/blob/c1958cc723f58c1f7f586eab79e36f2fbebe7492/extraconfig/tasks/yum_update.sh [2] [stack@instack ~]$ for i in $(nova list|grep ctlplane|awk -F' ' '{ print $12 }'|awk -F'=' '{ print $2 }'); do ssh heat-admin@$i 'hostname; echo "*******"; echo "" ;sudo grep -ni "error:" /var/log/messages' | grep -v "demote in redis"; echo ""; done Hi Marios,
The gnocci errors are there upon deployment. I see the following after a clean deployment:
Jul 5 11:10:13 localhost gnocchi-metricd: raise ImportError("Your rados python module does not support "
Jul 5 11:10:13 localhost gnocchi-metricd: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0
[stack@instack ~]$ openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml
starting package update on stack overcloud
IN_PROGRESS
WAITING
not_started: [u'overcloud-controller-1', u'overcloud-controller-0']
on_breakpoint: [u'overcloud-controller-2', u'overcloud-cephstorage-0', u'overcloud-novacompute-0']
Breakpoint reached, continue? Regexp or Enter=proceed (will clear 8ed88fa4-5d65-4def-b2d6-5bde3a8ccd3b), no=cancel update, C-c=quit interactive mode:
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
WAITING
completed: [u'overcloud-novacompute-0']
on_breakpoint: [u'overcloud-controller-1', u'overcloud-controller-2', u'overcloud-cephstorage-0', u'overcloud-controller-0']
Breakpoint reached, continue? Regexp or Enter=proceed (will clear 6d6d97f2-330e-41e6-8e4c-2a7f142c8b34), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
The above is what I get.
After many repeatitions of "IN_PROGRESS" the update then fails without being able to complete the rest of the OC nodes.
I checked the OC nodes. On all but one the yum history shows that the update wasn't attempted.
On overcloud-novacompute-0 the yum wasn't attempted and I was able to run yum update manually on it - without issues.
On overcloud-controller0 I see that yum update was already done, but I see the following:
[root@overcloud-controller-0 ~]# yum history
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
ID | Login user | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
14 | System <unset> | 2016-07-06 14:54 | I, U | 123 **
13 | Cloud User <heat-admin> | 2016-07-06 14:16 | Install | 1
12 | Cloud User <heat-admin> | 2016-07-06 14:16 | Install | 1
11 | root <root> | 2016-06-16 18:19 | Install | 1
10 | root <root> | 2016-06-16 18:18 | Install | 7 EE
9 | root <root> | 2016-06-16 18:17 | Install | 1
8 | root <root> | 2016-06-16 18:15 | Install | 35
7 | root <root> | 2016-06-16 17:38 | Install | 622 EE
6 | root <root> | 2016-06-16 17:33 | Install | 2
5 | root <root> | 2016-06-16 17:23 | Install | 119
4 | root <root> | 2016-06-16 17:22 | Install | 2
3 | root <root> | 2016-06-16 17:22 | Erase | 1
2 | root <root> | 2016-06-16 17:21 | Install | 13
1 | System <unset> | 2016-06-16 16:36 | Install | 338
Warning: RPMDB altered outside of yum.
** Found 78 pre-existing rpmdb problem(s), 'yum check' output follows:
1:NetworkManager-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-1.0.6-29.el7_2.x86_64
1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-config-server-1.0.6-29.el7_2.x86_64
1:NetworkManager-libnm-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-libnm-1.0.6-29.el7_2.x86_64
1:NetworkManager-team-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-team-1.0.6-29.el7_2.x86_64
1:NetworkManager-tui-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-tui-1.0.6-29.el7_2.x86_64
corosync-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosync-2.3.4-7.el7_2.1.x86_64
corosynclib-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosynclib-2.3.4-7.el7_2.1.x86_64
7:device-mapper-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-1.02.107-5.el7_2.2.x86_64
7:device-mapper-event-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-1.02.107-5.el7_2.2.x86_64
7:device-mapper-event-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64
7:device-mapper-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-libs-1.02.107-5.el7_2.2.x86_64
device-mapper-multipath-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-0.4.9-85.el7_2.4.x86_64
device-mapper-multipath-libs-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64
dracut-033-360.el7_2.1.x86_64 is a duplicate with dracut-033-360.el7_2.x86_64
dracut-config-generic-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-generic-033-360.el7_2.x86_64
dracut-config-rescue-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-rescue-033-360.el7_2.x86_64
glusterfs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-3.7.5-19.el7.x86_64
glusterfs-api-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-api-3.7.5-19.el7.x86_64
glusterfs-client-xlators-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-client-xlators-3.7.5-19.el7.x86_64
glusterfs-libs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-libs-3.7.5-19.el7.x86_64
iproute-3.10.0-54.el7_2.1.x86_64 is a duplicate with iproute-3.10.0-54.el7.x86_64
iscsi-initiator-utils-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-6.2.0.873-32.el7.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64
kernel-headers-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-headers-3.10.0-327.18.2.el7.x86_64
kernel-tools-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-3.10.0-327.18.2.el7.x86_64
kernel-tools-libs-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-libs-3.10.0-327.18.2.el7.x86_64
kpartx-0.4.9-85.el7_2.5.x86_64 is a duplicate with kpartx-0.4.9-85.el7_2.4.x86_64
libgudev1-219-19.el7_2.11.x86_64 is a duplicate with libgudev1-219-19.el7_2.9.x86_64
libvirt-client-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-client-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-lxc-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64
libxml2-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-2.9.1-6.el7_2.2.x86_64
libxml2-devel-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-devel-2.9.1-6.el7_2.2.x86_64
libxml2-python-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-python-2.9.1-6.el7_2.2.x86_64
7:lvm2-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-2.02.130-5.el7_2.2.x86_64
7:lvm2-libs-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-libs-2.02.130-5.el7_2.2.x86_64
1:mariadb-galera-common-5.5.42-2.el7ost.x86_64 is a duplicate with 1:mariadb-galera-common-5.5.42-1.el7ost.x86_64
1:mariadb-galera-server-5.5.42-2.el7ost.x86_64 is a duplicate with 1:mariadb-galera-server-5.5.42-1.el7ost.x86_64
1:nfs-utils-1.3.0-0.21.el7_2.1.x86_64 is a duplicate with 1:nfs-utils-1.3.0-0.21.el7_2.x86_64
1:openstack-ceilometer-api-6.1.0-1.el7ost.noarch is a duplicate with 1:openstack-ceilometer-api-6.0.0-2.el7ost.noarch
1:openstack-ceilometer-common-6.1.0-1.el7ost.noarch is a duplicate with 1:openstack-ceilometer-common-6.0.0-2.el7ost.noarch
openstack-gnocchi-common-2.1.3-2.el7ost.noarch is a duplicate with openstack-gnocchi-common-2.1.3-1.el7ost.noarch
openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch is a duplicate with openstack-gnocchi-statsd-2.1.3-1.el7ost.noarch
1:openstack-neutron-common-8.1.0-8.el7ost.noarch is a duplicate with 1:openstack-neutron-common-8.1.0-6.el7ost.noarch
1:openstack-neutron-metering-agent-8.1.0-8.el7ost.noarch is a duplicate with 1:openstack-neutron-metering-agent-8.1.0-6.el7ost.noarch
1:openstack-nova-api-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-api-13.0.0-8.el7ost.noarch
1:openstack-nova-cert-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-cert-13.0.0-8.el7ost.noarch
1:openstack-nova-common-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-common-13.0.0-8.el7ost.noarch
1:openstack-nova-conductor-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-conductor-13.0.0-8.el7ost.noarch
1:openstack-nova-novncproxy-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-novncproxy-13.0.0-8.el7ost.noarch
1:openstack-nova-scheduler-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-scheduler-13.0.0-8.el7ost.noarch
1:openstack-puppet-modules-8.1.2-1.el7ost.noarch is a duplicate with 1:openstack-puppet-modules-8.1.1-1.el7ost.noarch
os-apply-config-0.1.32-6.el7ost.noarch is a duplicate with os-apply-config-0.1.32-5.el7ost.noarch
os-collect-config-0.1.37-5.el7ost.noarch is a duplicate with os-collect-config-0.1.37-3.el7ost.noarch
os-refresh-config-0.1.11-4.el7ost.noarch is a duplicate with os-refresh-config-0.1.11-3.el7ost.noarch
polkit-0.112-7.el7_2.x86_64 is a duplicate with polkit-0.112-6.el7_2.x86_64
1:python-ceilometer-6.1.0-1.el7ost.noarch is a duplicate with 1:python-ceilometer-6.0.0-2.el7ost.noarch
1:python-django-horizon-9.0.1-1.el7ost.noarch is a duplicate with 1:python-django-horizon-9.0.0-5.el7ost.noarch
python-gnocchi-2.1.3-2.el7ost.noarch is a duplicate with python-gnocchi-2.1.3-1.el7ost.noarch
1:python-neutron-8.1.0-8.el7ost.noarch is a duplicate with 1:python-neutron-8.1.0-6.el7ost.noarch
1:python-nova-13.1.0-1.el7ost.noarch is a duplicate with 1:python-nova-13.0.0-8.el7ost.noarch
python-oslo-concurrency-3.7.1-1.el7ost.noarch is a duplicate with python-oslo-concurrency-3.6.0-1.el7ost.noarch
python-perf-3.10.0-327.22.2.el7.x86_64 is a duplicate with python-perf-3.10.0-327.18.2.el7.x86_64
python-webtest-1.3.4-6.el7.noarch is a duplicate with python-webtest-1.3.4-5.el7.noarch
rpcbind-0.2.0-33.el7_2.1.x86_64 is a duplicate with rpcbind-0.2.0-33.el7_2.x86_64
selinux-policy-3.13.1-60.el7_2.7.noarch is a duplicate with selinux-policy-3.13.1-60.el7_2.3.noarch
selinux-policy-targeted-3.13.1-60.el7_2.7.noarch is a duplicate with selinux-policy-targeted-3.13.1-60.el7_2.3.noarch
systemd-219-19.el7_2.11.x86_64 is a duplicate with systemd-219-19.el7_2.9.x86_64
systemd-libs-219-19.el7_2.11.x86_64 is a duplicate with systemd-libs-219-19.el7_2.9.x86_64
systemd-sysv-219-19.el7_2.11.x86_64 is a duplicate with systemd-sysv-219-19.el7_2.9.x86_64
tzdata-2016e-1.el7.noarch is a duplicate with tzdata-2016d-1.el7.noarch
history list
[root@overcloud-controller-0 ~]# yum update
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No packages marked for update
[root@overcloud-controller-0 ~]#
Got an opportunity to investigate an environment with this issue today. The hypothesis we discussed yesterday on IRC was correct, it's indeed caused by bug 1350489. There was an update of os-collect-config done by os-collect-config, and the postun script of the older RPM postun script restarted os-collect-config, resulting in the update never being reported to Heat and never finishing. See bug 1350489 for further details and links. To work around this for the time being, run 'sudo yum -y update os-collect-config' right before you trigger the minor update via Heat. The root cause is already fixed in os-collect-config RPM -- if you do the initial deploy with os-collect-config-0.1.37-5 or newer, this problem will not happen. However, we still need to deal with how to safely update os-collect-config from the broken version, but we'll do that earlier than in OSP 9 minor update (we'll do that during OSP 8 -> OSP 9 major upgrade or earlier within OSP 8). I'm not closing this as a duplicate of bug 1350489 (yet?) because that one is about major upgrades. It would still be nice to test that minor updates can succeed too when we start on os-collect-config-0.1.37-5 or newer, so i'm keeping this open. Following up with snippets to show that the root cause is indeed the same as bug 1350489: [root@overcloud-controller-0 ~]# yum history info 15 | grep os-collect-config -A1 Updated os-collect-config-0.1.37-3.el7ost.noarch @rhos-9.0-director-puddle Update 0.1.37-5.el7ost.noarch installed -- 7 warning: %postun(os-collect-config-0.1.37-3.el7ost.noarch) scriptlet failed, signal 15 history info A snippet from os-collect-config log that shows it got restarted during yum update: Jul 12 19:28:48 overcloud-controller-0.localdomain yum[22424]: Updated: python-webtest-1.3.4-6.el7.noarch Jul 12 19:28:49 overcloud-controller-0.localdomain yum[22424]: Updated: tzdata-2016f-1.el7.noarch Jul 12 19:29:08 overcloud-controller-0.localdomain yum[22424]: Installed: kernel-devel-3.10.0-327.22.2.el7.x86_64 Jul 12 19:29:15 overcloud-controller-0.localdomain yum[22424]: Updated: 1:openstack-puppet-modules-8.1.2-1.el7ost.noarch Jul 12 19:29:15 overcloud-controller-0.localdomain yum[22424]: Updated: python-perf-3.10.0-327.22.2.el7.x86_64 Jul 12 19:29:17 overcloud-controller-0.localdomain yum[22424]: Updated: 1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64 Jul 12 19:29:17 overcloud-controller-0.localdomain yum[22424]: Updated: os-apply-config-0.1.32-6.el7ost.noarch Jul 12 19:29:18 overcloud-controller-0.localdomain yum[22424]: Updated: kernel-headers-3.10.0-327.22.2.el7.x86_64 Jul 12 19:29:18 overcloud-controller-0.localdomain yum[22424]: Updated: os-refresh-config-0.1.11-4.el7ost.noarch Jul 12 19:29:20 overcloud-controller-0.localdomain systemd[1]: Stopping Collect metadata and run hook commands.... Jul 12 19:29:20 overcloud-controller-0.localdomain systemd[1]: Started Collect metadata and run hook commands.. Jul 12 19:29:20 overcloud-controller-0.localdomain systemd[1]: Starting Collect metadata and run hook commands.... Jul 12 19:29:21 overcloud-controller-0.localdomain os-collect-config[3187]: /var/lib/os-collect-config/local-data not found. Skipping Jul 12 19:29:21 overcloud-controller-0.localdomain os-collect-config[3187]: No local metadata found (['/var/lib/os-collect-config/local-data']) Jul 12 19:29:22 overcloud-controller-0.localdomain os-collect-config[3187]: [2016-07-12 19:29:22,030] (os-refresh-config) [INFO] Starting phase pre-configure Verified: Environment: os-collect-config-0.1.37-6.el7ost.noarch The reported issue doesn't reproduce. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-1599.html |
rhel-osp-director: 9.0 Update to async fails: "deploy_stdout": "Started yum_update.sh on server cb7f31ba-49f6-4641-be53-6411010c88c8 at Thu Jun 23 19:09:32 EDT 2016\nNot running due to unset update_identifier\n"," Environment: instack-undercloud-4.0.0-4.el7ost.noarch openstack-tripleo-heat-templates-2.0.0-11.el7ost.noarch openstack-puppet-modules-8.1.1-1.el7ost.noarch Steps to reproduce: 1. Deploy 9.0 2. Attempt to update to async version. Result: After a long time: IN_PROGRESS IN_PROGRESS IN_PROGRESS ERROR: Authentication failed: Authentication required [stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | CephStorage | 54319496-df48-4cca-97b0-ea4faf165be1 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-06-24T01:36:44 | overcloud | | 0 | 7484a4e5-c270-42e8-98c8-4b193d1d9115 | OS::TripleO::CephStorage | UPDATE_FAILED | 2016-06-24T01:36:47 | overcloud-CephStorage-y7oejcga2avx | | Controller | 8aa5c7e5-2830-463d-9df7-b301aa75a8e3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-06-24T01:37:03 | overcloud | | 2 | 89e6dcac-181c-491e-9d2b-6025907db94b | OS::TripleO::Controller | UPDATE_FAILED | 2016-06-24T01:37:13 | overcloud-Controller-g4klenufuufc | | UpdateDeployment | 5abc7bac-5a5c-483f-ae1f-941dfad602f3 | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2016-06-24T01:59:23 | overcloud-Controller-g4klenufuufc-2-xpht4e4owh7w | +-----------------------------------------------+------------------------------- [stack@instack ~]$ heat deployment-show 5abc7bac-5a5c-483f-ae1f-941dfad602f3 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "IN_PROGRESS", "server_id": "cb7f31ba-49f6-4641-be53-6411010c88c8", "config_id": "6776f747-918e-444e-bbb3-f5f0ab7223a0", "output_values": { "deploy_stdout": "Started yum_update.sh on server cb7f31ba-49f6-4641-be53-6411010c88c8 at Thu Jun 23 19:09:32 EDT 2016\nNot running due to unset update_identifier\n", "deploy_stderr": "", "update_managed_packages": "false", "deploy_status_code": 0 }, "creation_time": "2016-06-23T23:08:47", "updated_time": "2016-06-24T01:59:23", "input_values": {}, "action": "UPDATE", "status_reason": "Deploy data available", "id": "5abc7bac-5a5c-483f-ae1f-941dfad602f3" } Expected result: Successful update.