Bug 1349890 - rhel-osp-director: 9.0 Update to async fails - os-collect-config restarted during yum update
Summary: rhel-osp-director: 9.0 Update to async fails - os-collect-config restarted du...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: os-collect-config
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Ben Nemec
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-24 13:07 UTC by Alexander Chuzhoy
Modified: 2016-10-11 06:21 UTC (History)
14 users (show)

Fixed In Version: os-collect-config-0.1.37-5.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-11 11:33:37 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:1599 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 director Release Candidate Advisory 2016-08-11 15:25:37 UTC

Description Alexander Chuzhoy 2016-06-24 13:07:27 UTC
rhel-osp-director: 9.0 Update to async fails: "deploy_stdout": "Started yum_update.sh on server cb7f31ba-49f6-4641-be53-6411010c88c8 at Thu Jun 23 19:09:32 EDT 2016\nNot running due to unset update_identifier\n","


Environment:
instack-undercloud-4.0.0-4.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-11.el7ost.noarch
openstack-puppet-modules-8.1.1-1.el7ost.noarch

Steps to reproduce:
1. Deploy 9.0 
2. Attempt to update to async version.


Result:
After a long time:
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
ERROR: Authentication failed: Authentication required



[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| resource_name                                 | physical_resource_id                          | resource_type                                                                   | resource_status | updated_time        | stack_name                                                                                                                                        |
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| CephStorage                                   | 54319496-df48-4cca-97b0-ea4faf165be1          | OS::Heat::ResourceGroup                                                         | UPDATE_FAILED   | 2016-06-24T01:36:44 | overcloud                                                                                                                                         |
| 0                                             | 7484a4e5-c270-42e8-98c8-4b193d1d9115          | OS::TripleO::CephStorage                                                        | UPDATE_FAILED   | 2016-06-24T01:36:47 | overcloud-CephStorage-y7oejcga2avx                                                                                                                |
| Controller                                    | 8aa5c7e5-2830-463d-9df7-b301aa75a8e3          | OS::Heat::ResourceGroup                                                         | UPDATE_FAILED   | 2016-06-24T01:37:03 | overcloud                                                                                                                                         |
| 2                                             | 89e6dcac-181c-491e-9d2b-6025907db94b          | OS::TripleO::Controller                                                         | UPDATE_FAILED   | 2016-06-24T01:37:13 | overcloud-Controller-g4klenufuufc                                                                                                                 |
| UpdateDeployment                              | 5abc7bac-5a5c-483f-ae1f-941dfad602f3          | OS::Heat::SoftwareDeployment                                                    | UPDATE_FAILED   | 2016-06-24T01:59:23 | overcloud-Controller-g4klenufuufc-2-xpht4e4owh7w                                                                                                  |
+-----------------------------------------------+-------------------------------

[stack@instack ~]$ heat deployment-show 5abc7bac-5a5c-483f-ae1f-941dfad602f3
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "IN_PROGRESS",
  "server_id": "cb7f31ba-49f6-4641-be53-6411010c88c8",
  "config_id": "6776f747-918e-444e-bbb3-f5f0ab7223a0",
  "output_values": {
    "deploy_stdout": "Started yum_update.sh on server cb7f31ba-49f6-4641-be53-6411010c88c8 at Thu Jun 23 19:09:32 EDT 2016\nNot running due to unset update_identifier\n",
    "deploy_stderr": "",
    "update_managed_packages": "false",
    "deploy_status_code": 0
  },
  "creation_time": "2016-06-23T23:08:47",
  "updated_time": "2016-06-24T01:59:23",
  "input_values": {},
  "action": "UPDATE",
  "status_reason": "Deploy data available",
  "id": "5abc7bac-5a5c-483f-ae1f-941dfad602f3"
}


Expected result:
Successful update.

Comment 3 Alexander Chuzhoy 2016-06-27 21:42:20 UTC
Reproduced.

Takes a long time until the update fails.

Comment 4 Marios Andreou 2016-06-29 16:59:36 UTC
Hi Sasha, as promised and just in time for 'Wednesday' squeezed in a quick first look&triage today. I think the error you have pasted about the unset update identifier is a red herring (notice it is stdout and not stderr). During minor update  UpdateIdentifier is only set when it is time for the given node to be updated. Looking at controller-0 for example /var/log/messages, there is one occurrence of it being unset but then another where the update occurs here:
 
        Jun 23 19:09:45 localhost os-collect-config: [2016-06-23 19:09:45,400] (heat-config) [INFO] {"deploy_stdout": "Started yum_update.sh on server e85cb0d5-f5cd-4e81-b96b-6d451cc9ce5e at Thu Jun 23 19:09:45 EDT 2016\nNot running due to unset update_identifier\n", "deploy_stderr": "", "update_managed_packages": "false", "deploy_status_code": 0}

        ...
        Jun 24 01:47:40 overcloud-controller-0 os-collect-config: [2016-06-24 01:47:40,348] (heat-config) [INFO] {"deploy_stdout": "Started yum_update.sh on server e85cb0d5-f5cd-4e81-b96b-6d451cc9ce5e at Fri Jun 24 01:40:44 UTC 2016\nPacemaker running, stopping cluster node and doing full package update\nStopping Cluster (pacemaker)... Stopping Cluster (corosync)...\nRunning: yum -q -y update \nDelta RP

I also see the following in the trace, which is https://bugzilla.redhat.com/show_bug.cgi?id=1343905 afaics (also OSP9 minor update and is fixed upstream in the resource agents). Not clear this is the root yet:

        Jun 24 01:45:24 overcloud-controller-0 rabbitmq-cluster(rabbitmq)[19434]: INFO: node failed to join even after reseting local data. Check SELINUX policy
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [ Error: {aborted,{no_exists,[rabbit_user, ]
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [                             [{{internal_user,'$1','_','_'}, ]
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [                               [{'/=','$1',<<"guest">>}], ]
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [                               ['$_']}]]}} ]
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
        Jun 24 01:45:24 overcloud-controller-0 lrmd[18505]:  notice: rabbitmq_start_0:19434:stderr [ User defined signal 2 ]
        Jun 24 01:45:24 overcloud-controller-0 crmd[18508]:  notice: Operation rabbitmq_start_0: unknown error (node=overcloud-controller-0, call=245, rc=1, cib-update=66, confirmed=true)
        Jun 24 01:45:24 overcloud-controller-0 crmd[18508]:  notice: overcloud-controller-0-rabbitmq_start_0:245 [ User defined signal 2\nUser defined signal 2\nError: {aborted,{no_exists,[rabbit_user,\n                            [{{internal_user,'$1','_','_'},\n                              [{'/=','$1',<<"guest">>}],\n                              ['$_']}]]}}\nUser defined signal 2\nUser defined signal 2\n ]
        Jun 24 01:45:25 overcloud-controller-0 galera(galera)[23083]: ERROR: MySQL is not running


As always with these bugs can you also please include the full deployment and update commands you used (can't recover these from the sos files, though I could guess but you could also just tell me :) )

thanks, marios

Comment 5 Alexander Chuzhoy 2016-06-30 21:29:04 UTC
I noticed that yum update ran on only one machine.

I see the following on it:
[root@overcloud-controller-2 ~]# yum history       
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
ID     | Login user               | Date and time    | Action(s)      | Altered                                
-------------------------------------------------------------------------------                                
    14 | System <unset>           | 2016-06-30 13:50 | I, U           |  108 **                                
    13 | Cloud User <heat-admin>  | 2016-06-30 13:41 | Install        |    1                                   
    12 | Cloud User <heat-admin>  | 2016-06-30 13:41 | Install        |    1                                   
    11 | root <root>              | 2016-06-16 18:19 | Install        |    1                                   
    10 | root <root>              | 2016-06-16 18:18 | Install        |    7 EE                                
     9 | root <root>              | 2016-06-16 18:17 | Install        |    1                                   
     8 | root <root>              | 2016-06-16 18:15 | Install        |   35                                   
     7 | root <root>              | 2016-06-16 17:38 | Install        |  622 EE                                
     6 | root <root>              | 2016-06-16 17:33 | Install        |    2                                   
     5 | root <root>              | 2016-06-16 17:23 | Install        |  119                                   
     4 | root <root>              | 2016-06-16 17:22 | Install        |    2                                   
     3 | root <root>              | 2016-06-16 17:22 | Erase          |    1                                   
     2 | root <root>              | 2016-06-16 17:21 | Install        |   13                                   
     1 | System <unset>           | 2016-06-16 16:36 | Install        |  338                                   
Warning: RPMDB altered outside of yum.                                                                         
** Found 63 pre-existing rpmdb problem(s), 'yum check' output follows:                                         
1:NetworkManager-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-1.0.6-29.el7_2.x86_64              
1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-config-server-1.0.6-29.el7_2.x86_64
1:NetworkManager-libnm-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-libnm-1.0.6-29.el7_2.x86_64                
1:NetworkManager-team-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-team-1.0.6-29.el7_2.x86_64                  
1:NetworkManager-tui-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-tui-1.0.6-29.el7_2.x86_64                    
corosync-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosync-2.3.4-7.el7_2.1.x86_64                                          
corosynclib-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosynclib-2.3.4-7.el7_2.1.x86_64                                    
7:device-mapper-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-1.02.107-5.el7_2.2.x86_64                      
7:device-mapper-event-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-1.02.107-5.el7_2.2.x86_64          
7:device-mapper-event-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64
7:device-mapper-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-libs-1.02.107-5.el7_2.2.x86_64            
device-mapper-multipath-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-0.4.9-85.el7_2.4.x86_64          
device-mapper-multipath-libs-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64
dracut-033-360.el7_2.1.x86_64 is a duplicate with dracut-033-360.el7_2.x86_64                                                
dracut-config-generic-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-generic-033-360.el7_2.x86_64                  
dracut-config-rescue-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-rescue-033-360.el7_2.x86_64                    
glusterfs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-3.7.5-19.el7.x86_64                                              
glusterfs-api-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-api-3.7.5-19.el7.x86_64                                      
glusterfs-client-xlators-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-client-xlators-3.7.5-19.el7.x86_64                
glusterfs-libs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-libs-3.7.5-19.el7.x86_64                                    
iproute-3.10.0-54.el7_2.1.x86_64 is a duplicate with iproute-3.10.0-54.el7.x86_64
iscsi-initiator-utils-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-6.2.0.873-32.el7.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64
kernel-headers-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-headers-3.10.0-327.18.2.el7.x86_64
kernel-tools-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-3.10.0-327.18.2.el7.x86_64
kernel-tools-libs-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-libs-3.10.0-327.18.2.el7.x86_64
kpartx-0.4.9-85.el7_2.5.x86_64 is a duplicate with kpartx-0.4.9-85.el7_2.4.x86_64
libgudev1-219-19.el7_2.11.x86_64 is a duplicate with libgudev1-219-19.el7_2.9.x86_64
libvirt-client-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-client-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-lxc-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64
libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64
libxml2-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-2.9.1-6.el7_2.2.x86_64
libxml2-devel-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-devel-2.9.1-6.el7_2.2.x86_64
libxml2-python-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-python-2.9.1-6.el7_2.2.x86_64
7:lvm2-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-2.02.130-5.el7_2.2.x86_64
7:lvm2-libs-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-libs-2.02.130-5.el7_2.2.x86_64
1:nfs-utils-1.3.0-0.21.el7_2.1.x86_64 is a duplicate with 1:nfs-utils-1.3.0-0.21.el7_2.x86_64
1:openstack-nova-cert-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-cert-13.0.0-8.el7ost.noarch
1:openstack-nova-common-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-common-13.0.0-8.el7ost.noarch
1:openstack-nova-conductor-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-conductor-13.0.0-8.el7ost.noarch
1:openstack-nova-scheduler-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-scheduler-13.0.0-8.el7ost.noarch
os-apply-config-0.1.32-6.el7ost.noarch is a duplicate with os-apply-config-0.1.32-5.el7ost.noarch
os-collect-config-0.1.37-4.el7ost.noarch is a duplicate with os-collect-config-0.1.37-3.el7ost.noarch
os-refresh-config-0.1.11-4.el7ost.noarch is a duplicate with os-refresh-config-0.1.11-3.el7ost.noarch
polkit-0.112-7.el7_2.x86_64 is a duplicate with polkit-0.112-6.el7_2.x86_64
python-gnocchi-2.1.3-2.el7ost.noarch is a duplicate with python-gnocchi-2.1.3-1.el7ost.noarch
1:python-nova-13.1.0-1.el7ost.noarch is a duplicate with 1:python-nova-13.0.0-8.el7ost.noarch
python-oslo-concurrency-3.7.1-1.el7ost.noarch is a duplicate with python-oslo-concurrency-3.6.0-1.el7ost.noarch
python-perf-3.10.0-327.22.2.el7.x86_64 is a duplicate with python-perf-3.10.0-327.18.2.el7.x86_64
python-webtest-1.3.4-6.el7.noarch is a duplicate with python-webtest-1.3.4-5.el7.noarch
rpcbind-0.2.0-33.el7_2.1.x86_64 is a duplicate with rpcbind-0.2.0-33.el7_2.x86_64
selinux-policy-3.13.1-60.el7_2.7.noarch is a duplicate with selinux-policy-3.13.1-60.el7_2.3.noarch
systemd-219-19.el7_2.11.x86_64 is a duplicate with systemd-219-19.el7_2.9.x86_64
systemd-libs-219-19.el7_2.11.x86_64 is a duplicate with systemd-libs-219-19.el7_2.9.x86_64
systemd-sysv-219-19.el7_2.11.x86_64 is a duplicate with systemd-sysv-219-19.el7_2.9.x86_64
tzdata-2016e-1.el7.noarch is a duplicate with tzdata-2016d-1.el7.noarch
history list

Comment 6 Marios Andreou 2016-07-01 12:14:54 UTC
(In reply to Alexander Chuzhoy from comment #5)
> I noticed that yum update ran on only one machine.
> 
> I see the following on it:
> [root@overcloud-controller-2 ~]# yum history       

Hi Sasha, I am not entirely clear what you mean here. Do you mean the package update only ran on controller-2? Because from my comment #4 you can see it ran on controller-0.

Do you mean you tried to reproduce and on this second attempt 'noticed that yum update ran on only one machine'? 

As always with these bugs can you also please include the full deployment and update commands you used (can't recover these from the sos files, though I could guess but you could also just tell me :) )

I will poke a bit at the environment you sent

thanks, marios



> Loaded plugins: product-id, search-disabled-repos, subscription-manager
> This system is not registered to Red Hat Subscription Management. You can
> use subscription-manager to register.
> ID     | Login user               | Date and time    | Action(s)      |
> Altered                                
> -----------------------------------------------------------------------------
> --                                
>     14 | System <unset>           | 2016-06-30 13:50 | I, U           |  108
> **                                
>     13 | Cloud User <heat-admin>  | 2016-06-30 13:41 | Install        |    1
> 
>     12 | Cloud User <heat-admin>  | 2016-06-30 13:41 | Install        |    1
> 
>     11 | root <root>              | 2016-06-16 18:19 | Install        |    1
> 
>     10 | root <root>              | 2016-06-16 18:18 | Install        |    7
> EE                                
>      9 | root <root>              | 2016-06-16 18:17 | Install        |    1
> 
>      8 | root <root>              | 2016-06-16 18:15 | Install        |   35
> 
>      7 | root <root>              | 2016-06-16 17:38 | Install        |  622
> EE                                
>      6 | root <root>              | 2016-06-16 17:33 | Install        |    2
> 
>      5 | root <root>              | 2016-06-16 17:23 | Install        |  119
> 
>      4 | root <root>              | 2016-06-16 17:22 | Install        |    2
> 
>      3 | root <root>              | 2016-06-16 17:22 | Erase          |    1
> 
>      2 | root <root>              | 2016-06-16 17:21 | Install        |   13
> 
>      1 | System <unset>           | 2016-06-16 16:36 | Install        |  338
> 
> Warning: RPMDB altered outside of yum.                                      
> 
> ** Found 63 pre-existing rpmdb problem(s), 'yum check' output follows:      
> 
> 1:NetworkManager-1.0.6-30.el7_2.x86_64 is a duplicate with
> 1:NetworkManager-1.0.6-29.el7_2.x86_64              
> 1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64 is a duplicate with
> 1:NetworkManager-config-server-1.0.6-29.el7_2.x86_64
> 1:NetworkManager-libnm-1.0.6-30.el7_2.x86_64 is a duplicate with
> 1:NetworkManager-libnm-1.0.6-29.el7_2.x86_64                
> 1:NetworkManager-team-1.0.6-30.el7_2.x86_64 is a duplicate with
> 1:NetworkManager-team-1.0.6-29.el7_2.x86_64                  
> 1:NetworkManager-tui-1.0.6-30.el7_2.x86_64 is a duplicate with
> 1:NetworkManager-tui-1.0.6-29.el7_2.x86_64                    
> corosync-2.3.4-7.el7_2.3.x86_64 is a duplicate with
> corosync-2.3.4-7.el7_2.1.x86_64                                          
> corosynclib-2.3.4-7.el7_2.3.x86_64 is a duplicate with
> corosynclib-2.3.4-7.el7_2.1.x86_64                                    
> 7:device-mapper-1.02.107-5.el7_2.5.x86_64 is a duplicate with
> 7:device-mapper-1.02.107-5.el7_2.2.x86_64                      
> 7:device-mapper-event-1.02.107-5.el7_2.5.x86_64 is a duplicate with
> 7:device-mapper-event-1.02.107-5.el7_2.2.x86_64          
> 7:device-mapper-event-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with
> 7:device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64
> 7:device-mapper-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with
> 7:device-mapper-libs-1.02.107-5.el7_2.2.x86_64            
> device-mapper-multipath-0.4.9-85.el7_2.5.x86_64 is a duplicate with
> device-mapper-multipath-0.4.9-85.el7_2.4.x86_64          
> device-mapper-multipath-libs-0.4.9-85.el7_2.5.x86_64 is a duplicate with
> device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64
> dracut-033-360.el7_2.1.x86_64 is a duplicate with
> dracut-033-360.el7_2.x86_64                                                
> dracut-config-generic-033-360.el7_2.1.x86_64 is a duplicate with
> dracut-config-generic-033-360.el7_2.x86_64                  
> dracut-config-rescue-033-360.el7_2.1.x86_64 is a duplicate with
> dracut-config-rescue-033-360.el7_2.x86_64                    
> glusterfs-3.7.9-10.el7.x86_64 is a duplicate with
> glusterfs-3.7.5-19.el7.x86_64                                              
> glusterfs-api-3.7.9-10.el7.x86_64 is a duplicate with
> glusterfs-api-3.7.5-19.el7.x86_64                                      
> glusterfs-client-xlators-3.7.9-10.el7.x86_64 is a duplicate with
> glusterfs-client-xlators-3.7.5-19.el7.x86_64                
> glusterfs-libs-3.7.9-10.el7.x86_64 is a duplicate with
> glusterfs-libs-3.7.5-19.el7.x86_64                                    
> iproute-3.10.0-54.el7_2.1.x86_64 is a duplicate with
> iproute-3.10.0-54.el7.x86_64
> iscsi-initiator-utils-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with
> iscsi-initiator-utils-6.2.0.873-32.el7.x86_64
> iscsi-initiator-utils-iscsiuio-6.2.0.873-33.el7_2.1.x86_64 is a duplicate
> with iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64
> kernel-headers-3.10.0-327.22.2.el7.x86_64 is a duplicate with
> kernel-headers-3.10.0-327.18.2.el7.x86_64
> kernel-tools-3.10.0-327.22.2.el7.x86_64 is a duplicate with
> kernel-tools-3.10.0-327.18.2.el7.x86_64
> kernel-tools-libs-3.10.0-327.22.2.el7.x86_64 is a duplicate with
> kernel-tools-libs-3.10.0-327.18.2.el7.x86_64
> kpartx-0.4.9-85.el7_2.5.x86_64 is a duplicate with
> kpartx-0.4.9-85.el7_2.4.x86_64
> libgudev1-219-19.el7_2.11.x86_64 is a duplicate with
> libgudev1-219-19.el7_2.9.x86_64
> libvirt-client-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-client-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-lxc-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64
> libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 is a duplicate with
> libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64
> libxml2-2.9.1-6.el7_2.3.x86_64 is a duplicate with
> libxml2-2.9.1-6.el7_2.2.x86_64
> libxml2-devel-2.9.1-6.el7_2.3.x86_64 is a duplicate with
> libxml2-devel-2.9.1-6.el7_2.2.x86_64
> libxml2-python-2.9.1-6.el7_2.3.x86_64 is a duplicate with
> libxml2-python-2.9.1-6.el7_2.2.x86_64
> 7:lvm2-2.02.130-5.el7_2.5.x86_64 is a duplicate with
> 7:lvm2-2.02.130-5.el7_2.2.x86_64
> 7:lvm2-libs-2.02.130-5.el7_2.5.x86_64 is a duplicate with
> 7:lvm2-libs-2.02.130-5.el7_2.2.x86_64
> 1:nfs-utils-1.3.0-0.21.el7_2.1.x86_64 is a duplicate with
> 1:nfs-utils-1.3.0-0.21.el7_2.x86_64
> 1:openstack-nova-cert-13.1.0-1.el7ost.noarch is a duplicate with
> 1:openstack-nova-cert-13.0.0-8.el7ost.noarch
> 1:openstack-nova-common-13.1.0-1.el7ost.noarch is a duplicate with
> 1:openstack-nova-common-13.0.0-8.el7ost.noarch
> 1:openstack-nova-conductor-13.1.0-1.el7ost.noarch is a duplicate with
> 1:openstack-nova-conductor-13.0.0-8.el7ost.noarch
> 1:openstack-nova-scheduler-13.1.0-1.el7ost.noarch is a duplicate with
> 1:openstack-nova-scheduler-13.0.0-8.el7ost.noarch
> os-apply-config-0.1.32-6.el7ost.noarch is a duplicate with
> os-apply-config-0.1.32-5.el7ost.noarch
> os-collect-config-0.1.37-4.el7ost.noarch is a duplicate with
> os-collect-config-0.1.37-3.el7ost.noarch
> os-refresh-config-0.1.11-4.el7ost.noarch is a duplicate with
> os-refresh-config-0.1.11-3.el7ost.noarch
> polkit-0.112-7.el7_2.x86_64 is a duplicate with polkit-0.112-6.el7_2.x86_64
> python-gnocchi-2.1.3-2.el7ost.noarch is a duplicate with
> python-gnocchi-2.1.3-1.el7ost.noarch
> 1:python-nova-13.1.0-1.el7ost.noarch is a duplicate with
> 1:python-nova-13.0.0-8.el7ost.noarch
> python-oslo-concurrency-3.7.1-1.el7ost.noarch is a duplicate with
> python-oslo-concurrency-3.6.0-1.el7ost.noarch
> python-perf-3.10.0-327.22.2.el7.x86_64 is a duplicate with
> python-perf-3.10.0-327.18.2.el7.x86_64
> python-webtest-1.3.4-6.el7.noarch is a duplicate with
> python-webtest-1.3.4-5.el7.noarch
> rpcbind-0.2.0-33.el7_2.1.x86_64 is a duplicate with
> rpcbind-0.2.0-33.el7_2.x86_64
> selinux-policy-3.13.1-60.el7_2.7.noarch is a duplicate with
> selinux-policy-3.13.1-60.el7_2.3.noarch
> systemd-219-19.el7_2.11.x86_64 is a duplicate with
> systemd-219-19.el7_2.9.x86_64
> systemd-libs-219-19.el7_2.11.x86_64 is a duplicate with
> systemd-libs-219-19.el7_2.9.x86_64
> systemd-sysv-219-19.el7_2.11.x86_64 is a duplicate with
> systemd-sysv-219-19.el7_2.9.x86_64
> tzdata-2016e-1.el7.noarch is a duplicate with tzdata-2016d-1.el7.noarch
> history list

Comment 7 Marios Andreou 2016-07-01 13:36:07 UTC
Hi Sasha:

just had spent some time on the env you sent me via email. So I answered my own question from comment 6, indeed it looks like yum update only ran on controller-2 - so *must* be a redeployment?

Before getting into the why (still not clear) can we please step back and be clear what this bug is about; for example I propose the title is changed since this isn't to do with unset update identifier, afaics and from my comment #4. Furthermore can we have a clear reproducer; deploy with OSP 9 like _foo_ and then update by doing _bar_.

For this second run, did you start from a clean environment? Looking at the logs, I can see a few  different errors that are from before the yum update on controller-2. Best I can tell and using [1] for reference:

cluster stop on control-2 line 54 @[1]:

    54028 Jun 30 09:49:21 localhost systemd: Stopped Pacemaker High Availability Cluster Manager.
    54029 Jun 30 09:49:21 localhost systemd: Stopping Corosync Cluster Engine...
    54044 Jun 30 09:49:21 localhost corosync[6724]: [MAIN  ] Corosync Cluster Engine exiting normally

package update on control-2 line 68 @[1]

    54099 Jun 30 09:50:30 localhost yum[15213]: Updated: glusterfs-libs-3.7.9-10.el7.x86_64
    54173 Jun 30 09:50:38 localhost yum[15213]: Updated: 7:lvm2-2.02.130-5.el7_2.5.x86_64

I dont see any errors after this update... just before it there is:

    53472:Jun 30 09:47:38 localhost ceilometer-collector: error: [Errno 4] Interrupted system call
    53559:Jun 30 09:47:46 localhost nova-conductor: Exception TypeError: "'NoneType' object is not callable" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:4> ignored

As i said earlier, there are many *other* errors here but they come from way earlier, June 29th... for example this is in all controllers [2]:

    10861:Jun 29 20:45:29 localhost gnocchi-metricd: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0

And this cinder error on controllers 1 and 2:

    40578:Jun 30 09:19:56 localhost cinder-scheduler: 2016-06-30 13:19:25.163 11498 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1205, u'Lock wait timeout exceeded; try restarting transaction')
    40656:Jun 30 09:19:56 localhost cinder-scheduler: 2016-06-30 13:19:56.019 11498 ERROR cinder.service DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') [SQL: u'UPDATE services SET updated_at=%s, report_count=%s WHERE services.id = %s'] [parameters: (datetime.datetime(2016, 6, 30, 13, 18, 34, 872488), 13501, 1)]


Finally on controller0 there is also this ceph error - but again, isn't this one day before the yum update on controller-2:

    5403:Jun 29 20:37:23 localhost os-collect-config: #033[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.admin]/Exec[ceph-injectkey-client.admin]/unless: Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph   --name 'mon.'   --keyring '/var/lib/ceph/mon/ceph-overcloud-controller-0/keyring'  auth get client.admin | grep AQAtYXRXAAAAABAAFa36Lk0MKfIH8zTjcPtWnw==" exceeded timeout#033[0m
    5404:Jun 29 20:37:23 localhost os-collect-config: #033[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.openstack]/Exec[ceph-injectkey-client.openstack]/unless: Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph   --name 'mon.'   --keyring '/var/lib/ceph/mon/ceph-overcloud-controller-0/keyring'  auth get client.openstack | grep AQAtYXRXAAAAABAAz4vP2YaVJrsxX9cvBNNtSQ==" exceeded timeout#033[0m
    5405:Jun 29 20:37:23 localhost os-collect-config: #033[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.bootstrap-osd]/Exec[ceph-injectkey-client.bootstrap-osd]/unless: Check "/bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph   --name 'mon.'   --keyring '/var/lib/ceph/mon/ceph-overcloud-controller-0/keyring'  auth get client.bootstrap-osd | grep AQAtYXRXAAAAABAAFa36Lk0MKfIH8zTjcPtWnw==" exceeded timeout#033[0m
    12922:Jun 29 20:50:11 localhost gnocchi-metricd: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0


So can we please be clear about what we are debugging otherwise I'm really not sure what I'm looking for here.

thanks, marios


[1] https://github.com/openstack/tripleo-heat-templates/blob/c1958cc723f58c1f7f586eab79e36f2fbebe7492/extraconfig/tasks/yum_update.sh

[2] [stack@instack ~]$ for i in $(nova list|grep ctlplane|awk -F' ' '{ print $12 }'|awk -F'=' '{ print $2 }'); do ssh heat-admin@$i 'hostname; echo "*******"; echo "" ;sudo grep -ni "error:" /var/log/messages' | grep -v "demote in redis"; echo ""; done

Comment 8 Alexander Chuzhoy 2016-07-06 14:11:43 UTC
Hi Marios,


The gnocci errors are there upon deployment. I see the following after a clean deployment:
Jul  5 11:10:13 localhost gnocchi-metricd: raise ImportError("Your rados python module does not support "
Jul  5 11:10:13 localhost gnocchi-metricd: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0

Comment 9 Alexander Chuzhoy 2016-07-06 16:52:33 UTC
[stack@instack ~]$ openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml  -e network-environment.yaml
starting package update on stack overcloud
IN_PROGRESS
WAITING
not_started: [u'overcloud-controller-1', u'overcloud-controller-0']
on_breakpoint: [u'overcloud-controller-2', u'overcloud-cephstorage-0', u'overcloud-novacompute-0']
Breakpoint reached, continue? Regexp or Enter=proceed (will clear 8ed88fa4-5d65-4def-b2d6-5bde3a8ccd3b), no=cancel update, C-c=quit interactive mode:
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS


WAITING
completed: [u'overcloud-novacompute-0']
on_breakpoint: [u'overcloud-controller-1', u'overcloud-controller-2', u'overcloud-cephstorage-0', u'overcloud-controller-0']
Breakpoint reached, continue? Regexp or Enter=proceed (will clear 6d6d97f2-330e-41e6-8e4c-2a7f142c8b34), no=cancel update, C-c=quit interactive mode: IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS


The above is what I get.
After many repeatitions of "IN_PROGRESS" the update then fails without being able to complete the rest of the OC nodes.


I checked the OC nodes. On all but one the yum history shows that the update wasn't attempted.

On overcloud-novacompute-0 the yum wasn't attempted and I was able to run yum update manually on it - without issues.

On overcloud-controller0 I see that yum update was already done, but I see the following:


[root@overcloud-controller-0 ~]# yum history                                                                                                                                                                         
Loaded plugins: product-id, search-disabled-repos, subscription-manager                                                                                                                                              
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.                                                                                                      
ID     | Login user               | Date and time    | Action(s)      | Altered                                                                                                                                      
-------------------------------------------------------------------------------                                                                                                                                      
    14 | System <unset>           | 2016-07-06 14:54 | I, U           |  123 **                                                                                                                                      
    13 | Cloud User <heat-admin>  | 2016-07-06 14:16 | Install        |    1                                                                                                                                         
    12 | Cloud User <heat-admin>  | 2016-07-06 14:16 | Install        |    1                                                                                                                                         
    11 | root <root>              | 2016-06-16 18:19 | Install        |    1                                                                                                                                         
    10 | root <root>              | 2016-06-16 18:18 | Install        |    7 EE                                                                                                                                      
     9 | root <root>              | 2016-06-16 18:17 | Install        |    1                                                                                                                                         
     8 | root <root>              | 2016-06-16 18:15 | Install        |   35                                                                                                                                         
     7 | root <root>              | 2016-06-16 17:38 | Install        |  622 EE                                                                                                                                      
     6 | root <root>              | 2016-06-16 17:33 | Install        |    2                                                                                                                                         
     5 | root <root>              | 2016-06-16 17:23 | Install        |  119                                                                                                                                         
     4 | root <root>              | 2016-06-16 17:22 | Install        |    2                                                                                                                                         
     3 | root <root>              | 2016-06-16 17:22 | Erase          |    1                                                                                                                                         
     2 | root <root>              | 2016-06-16 17:21 | Install        |   13                                                                                                                                         
     1 | System <unset>           | 2016-06-16 16:36 | Install        |  338                                                                                                                                         
Warning: RPMDB altered outside of yum.                                                                                                                                                                               
** Found 78 pre-existing rpmdb problem(s), 'yum check' output follows:                                                                                                                                               
1:NetworkManager-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-1.0.6-29.el7_2.x86_64                                                                                                                    
1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-config-server-1.0.6-29.el7_2.x86_64                                                                                        
1:NetworkManager-libnm-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-libnm-1.0.6-29.el7_2.x86_64                                                                                                        
1:NetworkManager-team-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-team-1.0.6-29.el7_2.x86_64                                                                                                          
1:NetworkManager-tui-1.0.6-30.el7_2.x86_64 is a duplicate with 1:NetworkManager-tui-1.0.6-29.el7_2.x86_64                                                                                                            
corosync-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosync-2.3.4-7.el7_2.1.x86_64                                                                                                                                  
corosynclib-2.3.4-7.el7_2.3.x86_64 is a duplicate with corosynclib-2.3.4-7.el7_2.1.x86_64                                                                                                                            
7:device-mapper-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-1.02.107-5.el7_2.2.x86_64                                                                                                              
7:device-mapper-event-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-1.02.107-5.el7_2.2.x86_64                                                                                                  
7:device-mapper-event-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64                                                                                        
7:device-mapper-libs-1.02.107-5.el7_2.5.x86_64 is a duplicate with 7:device-mapper-libs-1.02.107-5.el7_2.2.x86_64                                                                                                    
device-mapper-multipath-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-0.4.9-85.el7_2.4.x86_64                                                                                                  
device-mapper-multipath-libs-0.4.9-85.el7_2.5.x86_64 is a duplicate with device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64                                                                                        
dracut-033-360.el7_2.1.x86_64 is a duplicate with dracut-033-360.el7_2.x86_64                                                                                                                                        
dracut-config-generic-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-generic-033-360.el7_2.x86_64                                                                                                          
dracut-config-rescue-033-360.el7_2.1.x86_64 is a duplicate with dracut-config-rescue-033-360.el7_2.x86_64                                                                                                            
glusterfs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-3.7.5-19.el7.x86_64                                                                                                                                      
glusterfs-api-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-api-3.7.5-19.el7.x86_64                                                                                                                              
glusterfs-client-xlators-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-client-xlators-3.7.5-19.el7.x86_64                                                                                                        
glusterfs-libs-3.7.9-10.el7.x86_64 is a duplicate with glusterfs-libs-3.7.5-19.el7.x86_64                                                                                                                            
iproute-3.10.0-54.el7_2.1.x86_64 is a duplicate with iproute-3.10.0-54.el7.x86_64                                                                                                                                    
iscsi-initiator-utils-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-6.2.0.873-32.el7.x86_64                                                                                                  
iscsi-initiator-utils-iscsiuio-6.2.0.873-33.el7_2.1.x86_64 is a duplicate with iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64                                                                                
kernel-headers-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-headers-3.10.0-327.18.2.el7.x86_64                                                                                                              
kernel-tools-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-3.10.0-327.18.2.el7.x86_64                                                                                                                  
kernel-tools-libs-3.10.0-327.22.2.el7.x86_64 is a duplicate with kernel-tools-libs-3.10.0-327.18.2.el7.x86_64                                                                                                        
kpartx-0.4.9-85.el7_2.5.x86_64 is a duplicate with kpartx-0.4.9-85.el7_2.4.x86_64                                                                                                                                    
libgudev1-219-19.el7_2.11.x86_64 is a duplicate with libgudev1-219-19.el7_2.9.x86_64                                                                                                                                 
libvirt-client-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-client-1.2.17-13.el7_2.4.x86_64                                                                                                                  
libvirt-daemon-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-1.2.17-13.el7_2.4.x86_64                                                                                                                  
libvirt-daemon-driver-interface-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-interface-1.2.17-13.el7_2.4.x86_64                                                                                
libvirt-daemon-driver-lxc-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-lxc-1.2.17-13.el7_2.4.x86_64                                                                                            
libvirt-daemon-driver-network-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-network-1.2.17-13.el7_2.4.x86_64                                                                                    
libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nodedev-1.2.17-13.el7_2.4.x86_64                                                                                    
libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-nwfilter-1.2.17-13.el7_2.4.x86_64                                                                                  
libvirt-daemon-driver-qemu-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-qemu-1.2.17-13.el7_2.4.x86_64                                                                                          
libvirt-daemon-driver-secret-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-secret-1.2.17-13.el7_2.4.x86_64                                                                                      
libvirt-daemon-driver-storage-1.2.17-13.el7_2.5.x86_64 is a duplicate with libvirt-daemon-driver-storage-1.2.17-13.el7_2.4.x86_64                                                                                    
libxml2-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-2.9.1-6.el7_2.2.x86_64                                                                                                                                    
libxml2-devel-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-devel-2.9.1-6.el7_2.2.x86_64
libxml2-python-2.9.1-6.el7_2.3.x86_64 is a duplicate with libxml2-python-2.9.1-6.el7_2.2.x86_64
7:lvm2-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-2.02.130-5.el7_2.2.x86_64
7:lvm2-libs-2.02.130-5.el7_2.5.x86_64 is a duplicate with 7:lvm2-libs-2.02.130-5.el7_2.2.x86_64
1:mariadb-galera-common-5.5.42-2.el7ost.x86_64 is a duplicate with 1:mariadb-galera-common-5.5.42-1.el7ost.x86_64
1:mariadb-galera-server-5.5.42-2.el7ost.x86_64 is a duplicate with 1:mariadb-galera-server-5.5.42-1.el7ost.x86_64
1:nfs-utils-1.3.0-0.21.el7_2.1.x86_64 is a duplicate with 1:nfs-utils-1.3.0-0.21.el7_2.x86_64
1:openstack-ceilometer-api-6.1.0-1.el7ost.noarch is a duplicate with 1:openstack-ceilometer-api-6.0.0-2.el7ost.noarch
1:openstack-ceilometer-common-6.1.0-1.el7ost.noarch is a duplicate with 1:openstack-ceilometer-common-6.0.0-2.el7ost.noarch
openstack-gnocchi-common-2.1.3-2.el7ost.noarch is a duplicate with openstack-gnocchi-common-2.1.3-1.el7ost.noarch
openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch is a duplicate with openstack-gnocchi-statsd-2.1.3-1.el7ost.noarch
1:openstack-neutron-common-8.1.0-8.el7ost.noarch is a duplicate with 1:openstack-neutron-common-8.1.0-6.el7ost.noarch
1:openstack-neutron-metering-agent-8.1.0-8.el7ost.noarch is a duplicate with 1:openstack-neutron-metering-agent-8.1.0-6.el7ost.noarch
1:openstack-nova-api-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-api-13.0.0-8.el7ost.noarch
1:openstack-nova-cert-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-cert-13.0.0-8.el7ost.noarch
1:openstack-nova-common-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-common-13.0.0-8.el7ost.noarch
1:openstack-nova-conductor-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-conductor-13.0.0-8.el7ost.noarch
1:openstack-nova-novncproxy-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-novncproxy-13.0.0-8.el7ost.noarch
1:openstack-nova-scheduler-13.1.0-1.el7ost.noarch is a duplicate with 1:openstack-nova-scheduler-13.0.0-8.el7ost.noarch
1:openstack-puppet-modules-8.1.2-1.el7ost.noarch is a duplicate with 1:openstack-puppet-modules-8.1.1-1.el7ost.noarch
os-apply-config-0.1.32-6.el7ost.noarch is a duplicate with os-apply-config-0.1.32-5.el7ost.noarch
os-collect-config-0.1.37-5.el7ost.noarch is a duplicate with os-collect-config-0.1.37-3.el7ost.noarch
os-refresh-config-0.1.11-4.el7ost.noarch is a duplicate with os-refresh-config-0.1.11-3.el7ost.noarch
polkit-0.112-7.el7_2.x86_64 is a duplicate with polkit-0.112-6.el7_2.x86_64
1:python-ceilometer-6.1.0-1.el7ost.noarch is a duplicate with 1:python-ceilometer-6.0.0-2.el7ost.noarch
1:python-django-horizon-9.0.1-1.el7ost.noarch is a duplicate with 1:python-django-horizon-9.0.0-5.el7ost.noarch
python-gnocchi-2.1.3-2.el7ost.noarch is a duplicate with python-gnocchi-2.1.3-1.el7ost.noarch
1:python-neutron-8.1.0-8.el7ost.noarch is a duplicate with 1:python-neutron-8.1.0-6.el7ost.noarch
1:python-nova-13.1.0-1.el7ost.noarch is a duplicate with 1:python-nova-13.0.0-8.el7ost.noarch
python-oslo-concurrency-3.7.1-1.el7ost.noarch is a duplicate with python-oslo-concurrency-3.6.0-1.el7ost.noarch
python-perf-3.10.0-327.22.2.el7.x86_64 is a duplicate with python-perf-3.10.0-327.18.2.el7.x86_64
python-webtest-1.3.4-6.el7.noarch is a duplicate with python-webtest-1.3.4-5.el7.noarch
rpcbind-0.2.0-33.el7_2.1.x86_64 is a duplicate with rpcbind-0.2.0-33.el7_2.x86_64
selinux-policy-3.13.1-60.el7_2.7.noarch is a duplicate with selinux-policy-3.13.1-60.el7_2.3.noarch
selinux-policy-targeted-3.13.1-60.el7_2.7.noarch is a duplicate with selinux-policy-targeted-3.13.1-60.el7_2.3.noarch
systemd-219-19.el7_2.11.x86_64 is a duplicate with systemd-219-19.el7_2.9.x86_64
systemd-libs-219-19.el7_2.11.x86_64 is a duplicate with systemd-libs-219-19.el7_2.9.x86_64
systemd-sysv-219-19.el7_2.11.x86_64 is a duplicate with systemd-sysv-219-19.el7_2.9.x86_64
tzdata-2016e-1.el7.noarch is a duplicate with tzdata-2016d-1.el7.noarch
history list
[root@overcloud-controller-0 ~]# yum update
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No packages marked for update
[root@overcloud-controller-0 ~]#

Comment 15 Jiri Stransky 2016-07-13 09:29:11 UTC
Got an opportunity to investigate an environment with this issue today. The hypothesis we discussed yesterday on IRC was correct, it's indeed caused by bug 1350489.

There was an update of os-collect-config done by os-collect-config, and the postun script of the older RPM postun script restarted os-collect-config, resulting in the update never being reported to Heat and never finishing. See bug 1350489 for further details and links.

To work around this for the time being, run 'sudo yum -y update os-collect-config' right before you trigger the minor update via Heat. The root cause is already fixed in os-collect-config RPM -- if you do the initial deploy with os-collect-config-0.1.37-5 or newer, this problem will not happen. However, we still need to deal with how to safely update os-collect-config from the broken version, but we'll do that earlier than in OSP 9 minor update (we'll do that during OSP 8 -> OSP 9 major upgrade or earlier within OSP 8).

I'm not closing this as a duplicate of bug 1350489 (yet?) because that one is about major upgrades. It would still be nice to test that minor updates can succeed too when we start on os-collect-config-0.1.37-5 or newer, so i'm keeping this open.


Following up with snippets to show that the root cause is indeed the same as bug 1350489:

[root@overcloud-controller-0 ~]# yum history info 15 | grep os-collect-config -A1
    Updated     os-collect-config-0.1.37-3.el7ost.noarch                   @rhos-9.0-director-puddle
    Update                        0.1.37-5.el7ost.noarch                   installed
--
   7 warning: %postun(os-collect-config-0.1.37-3.el7ost.noarch) scriptlet failed, signal 15
history info


A snippet from os-collect-config log that shows it got restarted during yum update:

Jul 12 19:28:48 overcloud-controller-0.localdomain yum[22424]: Updated: python-webtest-1.3.4-6.el7.noarch
Jul 12 19:28:49 overcloud-controller-0.localdomain yum[22424]: Updated: tzdata-2016f-1.el7.noarch
Jul 12 19:29:08 overcloud-controller-0.localdomain yum[22424]: Installed: kernel-devel-3.10.0-327.22.2.el7.x86_64
Jul 12 19:29:15 overcloud-controller-0.localdomain yum[22424]: Updated: 1:openstack-puppet-modules-8.1.2-1.el7ost.noarch
Jul 12 19:29:15 overcloud-controller-0.localdomain yum[22424]: Updated: python-perf-3.10.0-327.22.2.el7.x86_64
Jul 12 19:29:17 overcloud-controller-0.localdomain yum[22424]: Updated: 1:NetworkManager-config-server-1.0.6-30.el7_2.x86_64
Jul 12 19:29:17 overcloud-controller-0.localdomain yum[22424]: Updated: os-apply-config-0.1.32-6.el7ost.noarch
Jul 12 19:29:18 overcloud-controller-0.localdomain yum[22424]: Updated: kernel-headers-3.10.0-327.22.2.el7.x86_64
Jul 12 19:29:18 overcloud-controller-0.localdomain yum[22424]: Updated: os-refresh-config-0.1.11-4.el7ost.noarch
Jul 12 19:29:20 overcloud-controller-0.localdomain systemd[1]: Stopping Collect metadata and run hook commands....
Jul 12 19:29:20 overcloud-controller-0.localdomain systemd[1]: Started Collect metadata and run hook commands..
Jul 12 19:29:20 overcloud-controller-0.localdomain systemd[1]: Starting Collect metadata and run hook commands....
Jul 12 19:29:21 overcloud-controller-0.localdomain os-collect-config[3187]: /var/lib/os-collect-config/local-data not found. Skipping
Jul 12 19:29:21 overcloud-controller-0.localdomain os-collect-config[3187]: No local metadata found (['/var/lib/os-collect-config/local-data'])
Jul 12 19:29:22 overcloud-controller-0.localdomain os-collect-config[3187]: [2016-07-12 19:29:22,030] (os-refresh-config) [INFO] Starting phase pre-configure

Comment 17 Alexander Chuzhoy 2016-08-03 15:51:53 UTC
Verified:
Environment:
os-collect-config-0.1.37-6.el7ost.noarch
The reported issue doesn't reproduce.

Comment 19 errata-xmlrpc 2016-08-11 11:33:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html


Note You need to log in before you can comment on or make changes to this bug.