Bug 1393005 - rhel-osp-director: 9.0GA->latest with rhel7.2->rhel7.3 switch: undercloud yum update doesn't complete: ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.30.
Summary: rhel-osp-director: 9.0GA->latest with rhel7.2->rhel7.3 switch: undercloud y...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 9.0 (Mitaka)
Assignee: Lukas Bezdicka
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On: 1393045
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-08 16:43 UTC by Alexander Chuzhoy
Modified: 2017-05-05 16:45 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-05 16:45:55 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Alexander Chuzhoy 2016-11-08 16:43:02 UTC 
rhel-osp-director:   9.0GA->latest with rhel7.2->rhel7.3 switch: undercloud yum update doesn't complete:  ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.30.

Environment:
libselinux-utils-2.5-6.el7.x86_64
openstack-puppet-modules-8.1.8-3.el7ost.noarch
selinux-policy-targeted-3.13.1-102.el7_3.4.noarch
openstack-tripleo-heat-templates-2.0.0-36.el7ost.noarch
selinux-policy-devel-3.13.1-102.el7_3.4.noarch
selinux-policy-3.13.1-102.el7_3.4.noarch
libselinux-2.2.2-6.el7.x86_64
libselinux-ruby-2.2.2-6.el7.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
libselinux-python-2.5-6.el7.x86_64
instack-undercloud-4.0.0-15.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-36.el7ost.noarch
openstack-selinux-0.7.11-1.el7ost.noarch
openstack-puppet-modules-8.1.8-2.el7ost.noarch
libselinux-ruby-2.5-6.el7.x86_64
libselinux-2.5-6.el7.x86_64
openstack-tripleo-heat-templates-liberty-2.0.0-33.el7ost.noarch

Steps to reproduce:
1. Deploy 9.0GA on 7.2
2.Try to minor update to 9.0 latest wwith rhel7.3

Result:

The yum update never completes and there's an error:
16:02:05 ERROR: policydb version 30 does not match my version range 15-29
16:02:05 ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.30.
16:02:05 Traceback (most recent call last):
16:02:05   File "/sbin/semanage", line 32, in <module>
16:02:05     import seobject
16:02:05   File "/usr/lib/python2.7/site-packages/seobject/__init__.py", line 27, in <module>
16:02:05     import sepolicy
16:02:05   File "/usr/lib64/python2.7/site-packages/sepolicy/__init__.py", line 798, in <module>
16:02:05 ERROR: policydb version 30 does not match my version range 15-29
16:02:05 ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.30.
16:02:05 Traceback (most recent call last):
16:02:05   File "/sbin/semanage", line 32, in <module>
16:02:05     import seobject
16:02:05   File "/usr/lib/python2.7/site-packages/seobject/__init__.py", line 27, in <module>
16:02:05     import sepolicy
16:02:05   File "/usr/lib64/python2.7/site-packages/sepolicy/__init__.py", line 798, in <module>
16:02:05         raise e
16:02:05 raise e
16:02:05 ValueErrorValueError: : Failed to read //etc/selinux/targeted/policy/policy.30 policy fileFailed to read //etc/selinux/targeted/policy/policy.30 policy file
16:02:05 


It looks as following:

16:01:38   Updating   : 7:device-mapper-event-1.02.135-1.el7_3.1.x86_64          153/732 
16:01:38   Updating   : 7:lvm2-libs-2.02.166-1.el7_3.1.x86_64                    154/732 
16:01:38   Updating   : numad-0.5-17.20150602git.el7.x86_64                      155/732 
16:01:38   Updating   : mesa-libgbm-11.2.2-2.20160614.el7.x86_64                 156/732 
16:02:05   Updating   : selinux-policy-targeted-3.13.1-102.el7_3.4.noarch        157/732 
16:02:05 warning: /etc/selinux/targeted/seusers created as /etc/selinux/targeted/seusers.rpmnew
16:02:05 ‘/etc/selinux/targeted/modules/active/booleans.local’ -> ‘/etc/selinux/targeted/active/booleans.local’
16:02:05 ‘/etc/selinux/targeted/modules/active/file_contexts.local’ -> ‘/etc/selinux/targeted/active/file_contexts.local’
16:02:05 ‘/etc/selinux/targeted/modules/active/ports.local’ -> ‘/etc/selinux/targeted/active/ports.local’
16:02:05 ‘/etc/selinux/targeted/modules/active/seusers’ -> ‘/etc/selinux/targeted/active/seusers.local’
16:02:05 Full path required for exclude: net:[4026532236].
16:02:05 Full path required for exclude: net:[4026532236].
16:02:05 Full path required for exclude: net:[4026532236].
16:02:05 Full path required for exclude: net:[4026532236].
16:02:05 ERROR: policydb version 30 does not match my version range 15-29
16:02:05 ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.30.
16:02:05 Traceback (most recent call last):
16:02:05   File "/sbin/semanage", line 32, in <module>
16:02:05     import seobject
16:02:05   File "/usr/lib/python2.7/site-packages/seobject/__init__.py", line 27, in <module>
16:02:05     import sepolicy
16:02:05   File "/usr/lib64/python2.7/site-packages/sepolicy/__init__.py", line 798, in <module>
16:02:05 ERROR: policydb version 30 does not match my version range 15-29
16:02:05 ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.30.
16:02:05 Traceback (most recent call last):
16:02:05   File "/sbin/semanage", line 32, in <module>
16:02:05     import seobject
16:02:05   File "/usr/lib/python2.7/site-packages/seobject/__init__.py", line 27, in <module>
16:02:05     import sepolicy
16:02:05   File "/usr/lib64/python2.7/site-packages/sepolicy/__init__.py", line 798, in <module>
16:02:05         raise e
16:02:05 raise e
16:02:05 ValueErrorValueError: : Failed to read //etc/selinux/targeted/policy/policy.30 policy fileFailed to read //etc/selinux/targeted/policy/policy.30 policy file
16:02:05 
16:02:05 Full path required for exclude: net:[4026532236].
16:02:05 Full path required for exclude: net:[4026532236].
16:02:10   Updating   : diskimage-builder-1.13.0-2.el7ost.noarch                 158/732 
16:02:10   Updating   : openstack-tripleo-image-elements-0.9.9-7.el7ost.noarch   159/732 
16:02:10   Updating   : 1:wpa_supplicant-2.0-20.el7.x86_64                       160/732 
16:02:10   Updating   : python-rtslib-2.1.fb57-5.el7.noarch                      161/732 
16:02:10   Updating   : openvswitch-2.5.0-14.git20160727.el7fdp.x86_64           162/732 
16:02:10   Updating   : OpenIPMI-modalias-2.0.19-15.el7.x86_64                   163/732 
16:02:10   Updating   : ipmitool-1.8.15-7.el7.x86_64                             164/732 
16:02:10   Updating   : iptables-services-1.4.21-17.el7.x86_64                   165/732 
16:02:10   Updating   : mdadm-3.4-14.el7.x86_64                                  166/732 
16:02:12   Updating   : 10:qemu-kvm-common-rhev-2.6.0-27.el7.x86_64              167/732 
16:02:12   Updating   : 1:mariadb-5.5.52-1.el7.x86_64                            168/732 
16:02:12   Updating   : cryptsetup-1.7.2-1.el7.x86_64                            169/732 
16:02:15   Updating   : libnfsidmap-0.25-15.el7.x86_64                           170/732 
16:02:15   Updating   : python-pycurl-7.19.0-19.el7.x86_64                       171/732 
16:02:15   Updating   : python-urlgrabber-3.10-8.el7.noarch                      172/732 
16:02:15   Installing : 40:libcacard-2.5.2-2.el7.x86_64                          173/732 
16:02:15   Updating   : spice-server-0.12.4-19.el7.x86_64                        174/732 
16:02:15   Updating   : syslinux-extlinux-4.05-13.el7.x86_64                     175/732 
16:02:15   Updating   : libteam-1.25-4.el7.x86_64                                176/732 
16:02:15   Updating   : teamd-1.25-4.el7.x86_64                                  177/732 
16:02:15   Updating   : usbredir-0.7.1-1.el7.x86_64                              178/732 
16:02:15   Updating   : python-aodh-tests-2.0.5-1.el7ost.noarch                  179/732 
16:02:15   Updating   : 1:python-neutron-tests-8.1.2-5.el7ost.noarch             180/732 
16:02:17   Updating   : fontconfig-2.10.95-10.el7.x86_64                         181/732 
16:02:17   Updating   : python-openvswitch-2.5.0-14.git20160727.el7fdp.noarch    182/732 
16:02:17   Updating   : libselinux-python-2.5-6.el7.x86_64                       183/732 
16:02:17   Installing : python-magic-5.11-33.el7.noarch                          184/732 
16:02:17   Updating   : python-perf-3.10.0-514.el7.x86_64                        185/732 
16:02:17   Updating   : libsemanage-python-2.5-4.el7.x86_64                      186/732 
16:02:17   Updating   : python-devel-2.7.5-48.el7.x86_64                         187/732 
16:02:17   Updating   : audit-libs-python-2.6.5-3.el7.x86_64                     188/732 
16:02:18   Installing : python2-pyasn1-0.1.9-7.el7.noarch                        189/732 
16:02:18   Updating   : python-ipaddress-1.0.16-2.el7.noarch                     190/732 
16:02:18   Updating   : python-cffi-1.6.0-5.el7.x86_64                           191/732 
16:02:19   Installing : python2-cryptography-1.3.1-3.el7.x86_64                  192/732 
16:02:19   Updating   : 1:openstack-heat-common-6.0.0-11.el7ost.noarch           193/732 
16:02:19 warning: /etc/heat/heat.conf created as /etc/heat/heat.conf.rpmnew
16:02:26   Updating   : 1:python-heat-tests-6.0.0-11.el7ost.noarch               194/732 
16:02:26   Updating   : 1:python-nova-13.1.1-7.el7ost.noarch                     195/732 
16:02:26   Updating   : 1:openstack-nova-common-13.1.1-7.el7ost.noarch           196/732 
16:02:26 warning: /etc/nova/nova.conf created as /etc/nova/nova.conf.rpmnew
16:02:26   Updating   : 1:openstack-nova-console-13.1.1-7.el7ost.noarch          197/732 
16:02:26   Updating   : 1:openstack-nova-api-13.1.1-7.el7ost.noarch              198/732 
16:02:26   Updating   : 1:openstack-nova-scheduler-13.1.1-7.el7ost.noarch        199/732 
16:02:26   Updating   : 1:openstack-nova-novncproxy-13.1.1-7.el7ost.noarch       200/732 
16:02:26   Updating   : 1:openstack-nova-cert-13.1.1-7.el7ost.noarch             201/732 
16:02:26   Updating   : 1:openstack-nova-cells-13.1.1-7.el7ost.noarch            202/732 
16:02:26   Updating   : 1:openstack-nova-conductor-13.1.1-7.el7ost.noarch        203/732 
16:02:26   Updating   : 1:openstack-nova-network-13.1.1-7.el7ost.noarch          204/732 
16:02:26   Updating   : libsoup-2.48.1-6.el7.x86_64                              205/732 
16:02:26   Updating   : btrfs-progs-4.4.1-1.el7.x86_64                           206/732 
16:02:26   Updating   : perl-Net-SSLeay-1.55-4.el7.x86_64                        207/732 
16:02:33   Updating   : grubby-8.28-18.el7.x86_64                                208/732 
16:02:33   Installing : kernel-3.10.0-514.el7.x86_64                             209/732 
16:02:33 grubby fatal error: unable to find a suitable template
16:02:34   Updating   : netcf-libs-0.2.8-2.el7.x86_64                            210/732 
16:02:34   Updating   : httpd-tools-2.4.6-45.el7.x86_64                          211/732 
16:02:34   Updating   : ldns-1.6.16-10.el7.x86_64                                212/732 
16:02:52   Updating   : unbound-libs-1.4.20-28.el7.x86_64                        213/732 
16:02:53   Updating   : gnutls-dane-3.3.24-1.el7.x86_64                          214/732 
16:02:54   Updating   : gnutls-utils-3.3.24-1.el7.x86_64                         215/732 
16:02:54   Updating   : libvirt-client-2.0.0-10.el7.x86_64                       216/732 
16:02:54   Updating   : libvirt-python-2.0.0-2.el7.x86_64                        217/732 
16:02:54   Updating   : alsa-lib-1.1.1-1.el7.x86_64                              218/732 
16:02:55   Updating   : logrotate-3.8.6-12.el7.x86_64                            219/732 
16:02:55   Updating   : openstack-tripleo-heat-templates-liberty-2.0.0-36.el7o   220/732 
16:02:55   Updating   : openstack-tripleo-heat-templates-2.0.0-36.el7ost.noarc   221/732 
16:02:56   Updating   : pinentry-0.8.1-17.el7.x86_64                             222/732 
16:02:56   Updating   : gnupg2-2.0.22-4.el7.x86_64                               223/732 
16:02:56   Updating   : rpm-build-libs-4.11.3-21.el7.x86_64                      224/732 
16:02:58   Updating   : rpm-python-4.11.3-21.el7.x86_64                          225/732 
16:02:58   Updating   : yum-3.4.3-150.el7.noarch                                 226/732 
16:02:58   Updating   : rhn-client-tools-2.0.2-8.el7.noarch                      227/732 
16:02:58   Updating   : rhn-setup-2.0.2-8.el7.noarch                             228/732 
16:02:58   Updating   : yum-rhn-plugin-2.0.1-6.el7.noarch                        229/732 
16:02:58   Updating   : yum-utils-1.1.31-40.el7.noarch                           230/732 
16:02:58   Updating   : python-rhsm-1.17.9-1.el7.x86_64                          231/732 
16:02:58   Updating   : redhat-support-lib-python-0.9.7-6.el7.noarch             232/732 
16:02:58   Updating   : fuse-2.9.2-7.el7.x86_64                                  233/732 
16:02:58   Updating   : mozjs17-17.0.0-19.el7.x86_64                             234/732 
16:02:59   Updating   : polkit-0.112-9.el7.x86_64                                235/732 
16:02:59   Updating   : libvirt-daemon-2.0.0-10.el7.x86_64                       236/732 
16:02:59   Updating   : libvirt-daemon-driver-network-2.0.0-10.el7.x86_64        237/732 
16:02:59   Updating   : libvirt-daemon-driver-nodedev-2.0.0-10.el7.x86_64        238/732 
16:02:59   Updating   : libvirt-daemon-driver-interface-2.0.0-10.el7.x86_64      239/732 
16:02:59   Updating   : libvirt-daemon-driver-nwfilter-2.0.0-10.el7.x86_64       240/732 
16:02:59   Updating   : libvirt-daemon-driver-secret-2.0.0-10.el7.x86_64         241/732 
16:02:59   Updating   : setools-libs-3.3.8-1.1.el7.x86_64                        242/732 
16:03:00   Updating   : policycoreutils-python-2.5-8.el7.x86_64                  243/732 
16:03:05   Updating   : policycoreutils-devel-2.5-8.el7.x86_64                   244/732 
16:03:29   Updating   : selinux-policy-devel-3.13.1-102.el7_3.4.noarch           245/732 
16:03:29   Updating   : libstdc++-devel-4.8.5-11.el7.x86_64                      246/732 
16:03:29   Updating   : device-mapper-persistent-data-0.6.3-1.el7.x86_64         247/732 
16:03:30   Updating   : 7:lvm2-2.02.166-1.el7_3.1.x86_64                         248/732 
16:03:30 warning: /etc/lvm/lvm.conf created as /etc/lvm/lvm.conf.rpmnew
16:03:33   Updating   : 1:openstack-cinder-8.1.1-2.el7ost.noarch                 249/732 
16:03:33   Updating   : 1:python-cinder-tests-8.1.1-2.el7ost.noarch              250/732 
16:03:33   Updating   : boost-system-1.53.0-26.el7.x86_64                        251/732 
16:03:33   Updating   : libss-1.42.9-9.el7.x86_64                                252/732 
16:03:33   Updating   : e2fsprogs-1.42.9-9.el7.x86_64                            253/732 
16:03:33   Updating   : supermin5-5.1.16-4.el7.x86_64                            254/732 
16:03:33   Updating   : cpp-4.8.5-11.el7.x86_64                                  255/732 
16:03:33   Installing : GeoIP-1.5.0-11.el7.x86_64                                256/732 
16:03:33   Updating   : 32:bind-libs-lite-9.9.4-38.el7_3.x86_64                  257/732 
16:03:33   Updating   : 12:dhclient-4.2.5-47.el7.x86_64                          258/732 
16:03:33   Updating   : dracut-network-033-463.el7.x86_64                        259/732 
16:03:33   Updating   : psmisc-22.20-11.el7.x86_64                               260/732 
16:03:34   Updating   : libseccomp-2.3.1-2.el7.x86_64                            261/732 
16:03:34   Updating   : glibc-headers-2.17-157.el7.x86_64                        262/732 
16:03:34   Updating   : glibc-devel-2.17-157.el7.x86_64                          263/732 
16:03:34   Updating   : fuse-libs-2.9.2-7.el7.x86_64                             264/732 
16:03:35   Updating   : kernel-tools-libs-3.10.0-514.el7.x86_64                  265/732 
16:03:35   Updating   : lcms2-2.6-3.el7.x86_64                                   266/732 
16:03:35   Updating   : libnetfilter_cthelper-1.0.0-9.el7.x86_64                 267/732 
16:03:35   Updating   : conntrack-tools-1.4.3-1.el7.x86_64                       268/732 
16:03:35   Updating   : ipset-libs-6.19-6.el7.x86_64                             269/732 
16:03:35   Updating   : ipset-6.19-6.el7.x86_64                                  270/732 
16:03:35   Updating   : libpath_utils-0.2.1-27.el7.x86_64                        271/732 
16:03:35   Updating   : libini_config-1.3.0-27.el7.x86_64                        272/732 
16:03:35   Updating   : gssproxy-0.4.1-13.el7.x86_64                             273/732 
16:03:36   Updating   : 1:nfs-utils-1.3.0-0.33.el7.x86_64                        274/732 
16:03:36   Updating   : libvirt-daemon-driver-storage-2.0.0-10.el7.x86_64        275/732 
16:03:36   Updating   : libvirt-daemon-driver-qemu-2.0.0-10.el7.x86_64           276/732 
16:03:37   Updating   : libndp-1.2-7.el7.x86_64                                  277/732 
16:03:37   Updating   : 1:NetworkManager-1.4.0-12.el7.x86_64                     278/732 
16:03:37   Updating   : libquadmath-4.8.5-11.el7.x86_64                          279/732 
16:03:37   Updating   : lsscsi-0.27-4.el7.x86_64                                 280/732 
16:03:38   Updating   : lm_sensors-libs-3.4.0-4.20160601gitf9185e5.el7.x86_64    281/732 
16:03:38   Updating   : keepalived-1.2.13-8.el7.x86_64                           282/732 
16:03:38   Updating   : libtalloc-2.1.6-1.el7.x86_64                             283/732 
16:03:38   Updating   : sg3_utils-libs-1.37-9.el7.x86_64                         284/732 
16:03:40   Updating   : sg3_utils-1.37-9.el7.x86_64                              285/732 
16:03:40   Updating   : libgomp-4.8.5-11.el7.x86_64                              286/732 
16:03:41   Updating   : gcc-4.8.5-11.el7.x86_64                                  287/732 
16:03:42   Updating   : seabios-bin-1.9.1-5.el7.noarch                           288/732 
16:03:42   Updating   : kbd-misc-1.15.5-12.el7.noarch                            289/732 
16:03:42   Updating   : libX11-common-1.6.3-3.el7.noarch                         290/732 
16:03:42   Updating   : libX11-1.6.3-3.el7.x86_64                                291/732 
16:03:44   Updating   : pulseaudio-libs-6.0-8.el7.x86_64                         292/732 
16:03:44   Updating   : 10:qemu-kvm-rhev-2.6.0-27.el7.x86_64                     293/732 
16:03:44   Updating   : libvirt-daemon-kvm-2.0.0-10.el7.x86_64                   294/732 
16:03:44   Updating   : 1:libguestfs-1.32.7-3.el7.x86_64                         295/732 
16:03:44   Updating   : 1:python-libguestfs-1.32.7-3.el7.x86_64                  296/732 
16:03:45   Updating   : 1:openstack-nova-compute-13.1.1-7.el7ost.noarch          297/732 
16:03:48   Updating   : 1:openstack-nova-13.1.1-7.el7ost.noarch                  298/732 
16:03:53   Updating   : 1:python-nova-tests-13.1.1-7.el7ost.noarch               299/732 
16:03:53   Updating   : 1:openstack-tempest-10.0.0-3.b4a056dgit.el7ost.noarch    300/732 
16:03:53   Updating   : mesa-libEGL-11.2.2-2.20160614.el7.x86_64                 301/732 
16:03:54   Updating   : mesa-libGL-11.2.2-2.20160614.el7.x86_64                  302/732 
16:03:55   Updating   : ghostscript-9.07-20.el7.x86_64                           303/732 
16:03:57   Updating   : kbd-1.15.5-12.el7.x86_64                                 304/732 
16:03:57   Updating   : gcc-c++-4.8.5-11.el7.x86_64                              305/732 
16:03:57   Updating   : libtevent-0.9.28-1.el7.x86_64                            306/732 
16:03:57   Updating   : 1:openstack-neutron-8.1.2-5.el7ost.noarch                307/732 
16:03:57   Updating   : libgfortran-4.8.5-11.el7.x86_64                          308/732 
16:03:57   Updating   : 1:NetworkManager-tui-1.4.0-12.el7.x86_64                 309/732 
16:03:57   Updating   : 1:NetworkManager-team-1.4.0-12.el7.x86_64                310/732 
16:03:57   Updating   : 1:openstack-neutron-openvswitch-8.1.2-5.el7ost.noarch    311/732 
16:03:57   Updating   : kernel-tools-3.10.0-514.el7.x86_64                       312/732 
16:03:57   Updating   : kexec-tools-2.0.7-50.el7.x86_64                          313/732 
16:03:57   Updating   : boost-thread-1.53.0-26.el7.x86_64                        314/732 
16:03:57   Updating   : instack-undercloud-4.0.0-15.el7ost.noarch                315/732 
16:04:50   Updating   : openstack-selinux-0.7.11-1.el7ost.noarch                 316/732 
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50 Full path required for exclude: net:[4026532236].
16:04:50   Updating   : tuned-2.7.1-3.el7.noarch                                 317/732 
16:04:50   Updating   : redhat-support-tool-0.9.8-6.el7.noarch                   318/732 
16:04:50   Updating   : subscription-manager-1.17.15-1.el7.x86_64                319/732 
16:04:50   Updating   : rhn-check-2.0.2-8.el7.noarch                             320/732 
16:04:51   Updating   : rsyslog-7.4.7-16.el7.x86_64                              321/732 
16:04:51   Updating   : httpd-2.4.6-45.el7.x86_64                                322/732 
16:04:51   Updating   : perl-IO-Socket-SSL-1.94-5.el7.noarch                     323/732 
16:04:51   Updating   : 1:openstack-heat-api-6.0.0-11.el7ost.noarch              324/732 
16:04:51   Updating   : 1:openstack-heat-api-cfn-6.0.0-11.el7ost.noarch          325/732 
16:04:51   Updating   : 1:openstack-heat-engine-6.0.0-11.el7ost.noarch           326/732 
16:04:54   Updating   : python-virtualenv-1.10.1-3.el7.noarch                    327/732 
16:04:55   Updating   : 1:mariadb-server-5.5.52-1.el7.x86_64                     328/732 
16:04:55   Updating   : 1:grub2-2.02-0.44.el7.x86_64                             329/732 
16:04:55   Updating   : 1:facter-2.4.4-4.el7.x86_64                              330/732 
16:04:56   Updating   : openssh-server-6.6.1p1-31.el7.x86_64                     331/732 
16:04:56   Updating   : 1:openstack-neutron-ml2-8.1.2-5.el7ost.noarch            332/732 
16:04:56   Updating   : 2:xinetd-2.3.15-13.el7.x86_64                            333/732 
16:04:56   Updating   : openstack-aodh-listener-2.0.5-1.el7ost.noarch            334/732 
16:04:56   Updating   : openstack-aodh-api-2.0.5-1.el7ost.noarch                 335/732 
16:04:56   Updating   : openstack-aodh-evaluator-2.0.5-1.el7ost.noarch           336/732 
16:04:58   Updating   : openstack-aodh-notifier-2.0.5-1.el7ost.noarch            337/732 
16:04:58   Updating   : authconfig-6.2.8-14.el7.x86_64                           338/732 
16:04:58   Updating   : audit-2.6.5-3.el7.x86_64                                 339/732 
16:04:58   Updating   : memcached-1.4.15-10.el7.x86_64                           340/732 
16:04:58   Updating   : 3:irqbalance-1.0.7-6.el7.x86_64                          341/732 
16:04:58   Updating   : chrony-2.1.1-3.el7.x86_64                                342/732 
16:04:58   Updating   : 10:qemu-guest-agent-2.5.0-3.el7.x86_64                   343/732 
16:04:58   Updating   : at-3.1.13-22.el7.x86_64                                  344/732 
16:06:11   Updating   : 2:microcode_ctl-2.1-16.el7.x86_64                        345/732 
16:06:11   Updating   : tftp-server-5.2-13.el7.x86_64                            346/732 
16:06:11   Updating   : dracut-config-generic-033-463.el7.x86_64                 347/732 
16:06:11   Updating   : dracut-config-rescue-033-463.el7.x86_64                  348/732 
16:06:11   Updating   : postgresql-libs-9.2.18-1.el7.x86_64                      349/732 
16:06:12   Updating   : 1:cups-client-1.6.3-26.el7.x86_64                        350/732 
16:06:18   Updating   : MySQL-python-1.2.5-1.el7.x86_64                          351/732 
16:06:18   Updating   : 1:openstack-puppet-modules-8.1.8-3.el7ost.noarch         352/732 
16:06:19   Updating   : python-dns-1.12.0-2.20150617git465785f.el7.noarch        353/732 
16:06:19   Updating   : sos-3.3-4.el7.noarch                                     354/732 
16:06:19   Updating   : avahi-autoipd-0.6.31-17.el7.x86_64                       355/732 
16:06:29   Updating   : 1:perl-Pod-Escapes-1.04-291.el7.noarch                   356/732 
16:06:40   Installing : kernel-devel-3.10.0-514.el7.x86_64                       357/732 
16:06:40   Updating   : xfsprogs-4.5.0-8.el7.x86_64                              358/732 
16:06:40   Updating   : xz-devel-5.2.2-1.el7.x86_64                              359/732 
16:06:41   Updating   : wget-1.14-13.el7.x86_64                                  360/732 
16:06:41   Updating   : xdg-utils-1.1.0-0.17.20120809git.el7.noarch              361/732 
16:06:41   Updating   : libproxy-0.4.11-10.el7.x86_64                            362/732 
16:06:41   Updating   : zlib-devel-1.2.7-17.el7.x86_64                           363/732 
16:06:41   Updating   : libselinux-ruby-2.5-6.el7.x86_64                         364/732 
16:06:41   Updating   : lksctp-tools-1.0.17-2.el7.x86_64                         365/732 
16:06:41   Updating   : 3:traceroute-2.0.22-2.el7.x86_64                         366/732 
16:06:41   Updating   : ipxe-bootimgs-20160127-5.git6366fa7a.el7.noarch          367/732 
16:06:43   Updating   : 1:NetworkManager-config-server-1.4.0-12.el7.x86_64       368/732 
16:06:43   Updating   : iwl7265-firmware-22.0.7.0-49.el7.noarch                  369/732 
16:06:43   Cleanup    : httpd-2.4.6-40.el7_2.4.x86_64                            370/732 
16:06:43   Cleanup    : openssh-server-6.6.1p1-25.el7_2.x86_64                   371/732 
16:06:47   Cleanup    : 1:mariadb-server-5.5.50-1.el7_2.x86_64                   372/732 
16:06:47   Cleanup    : 1:cups-client-1.6.3-22.el7.x86_64                        373/732 
16:06:47   Cleanup    : ghostscript-9.07-18.el7.x86_64                           374/732 
16:06:47   Cleanup    : 1:cups-libs-1.6.3-22.el7.x86_64                          375/732 
16:06:47   Cleanup    : 1:mariadb-5.5.50-1.el7_2.x86_64                          376/732 
16:06:47   Cleanup    : 1:NetworkManager-tui-1.0.6-31.el7_2.x86_64               377/732 
16:07:16   Cleanup    : kexec-tools-2.0.7-38.el7_2.1.x86_64                      378/732 
16:07:16   Cleanup    : selinux-policy-devel-3.13.1-60.el7_2.9.noarch            379/732 
16:07:16   Cleanup    : policycoreutils-devel-2.2.5-20.el7.x86_64                380/732 
16:07:16   Cleanup    : MySQL-python-1.2.3-11.el7.x86_64                         381/732 
16:07:16   Cleanup    : 1:openstack-tempest-10.0.0-2.b4a056dgit.el7ost.noarch    382/732 
16:07:16   Cleanup    : rsyslog-7.4.7-12.el7.x86_64                              383/732 
16:07:16   Cleanup    : mesa-libGL-10.6.5-3.20150824.el7.x86_64                  384/732 
16:07:17   Cleanup    : audit-2.4.1-5.el7.x86_64                                 385/732 
16:07:18   Cleanup    : 2:xinetd-2.3.15-12.el7.x86_64                            386/732 
16:07:18   Cleanup    : mesa-libEGL-10.6.5-3.20150824.el7.x86_64                 387/732 
16:07:25   Cleanup    : 1:openstack-neutron-8.1.2-4.el7ost.noarch                388/732 
16:07:26   Cleanup    : keepalived-1.2.13-7.el7.x86_64                           389/732 
16:07:26   Cleanup    : subscription-manager-1.15.9-15.el7.x86_64                390/732 
16:07:26   Cleanup    : python-rhsm-1.15.4-5.el7.x86_64                          391/732 
16:07:26   Cleanup    : 1:NetworkManager-team-1.0.6-31.el7_2.x86_64              392/732 
16:07:26   Cleanup    : 1:NetworkManager-1.0.6-31.el7_2.x86_64                   393/732 
16:07:26   Cleanup    : 1:NetworkManager-libnm-1.0.6-31.el7_2.x86_64             394/732 
16:07:27   Cleanup    : 1:wpa_supplicant-2.0-17.el7_1.x86_64                     395/732 
16:07:27   Cleanup    : postgresql-libs-9.2.15-1.el7_2.x86_64                    396/732 
16:07:27   Cleanup    : tuned-2.5.1-4.el7_2.6.noarch                             397/732 
16:07:27   Cleanup    : libgudev1-219-19.el7_2.13.x86_64                         398/732 
16:07:27   Cleanup    : teamd-1.17-7.el7_2.x86_64                                399/732 
16:07:27   Cleanup    : 1:mariadb-libs-5.5.50-1.el7_2.x86_64                     400/732 
16:07:27   Cleanup    : memcached-1.4.15-9.4.el7ost.x86_64                       401/732 
16:07:27   Cleanup    : gcc-c++-4.8.5-4.el7.x86_64                               402/732 
16:07:27   Cleanup    : gcc-4.8.5-4.el7.x86_64                                   403/732 
16:07:28   Cleanup    : 10:qemu-guest-agent-2.3.0-4.el7.x86_64                   404/732 
16:07:28   Cleanup    : boost-thread-1.53.0-25.el7.x86_64                        405/732 
16:07:28   Cleanup    : authconfig-6.2.8-10.el7.x86_64                           406/732 
16:07:28   Cleanup    : wget-1.14-10.el7_0.1.x86_64                              407/732 
16:07:29   Cleanup    : 1:openstack-neutron-openvswitch-8.1.2-4.el7ost.noarch    408/732 
16:07:29   Cleanup    : instack-undercloud-4.0.0-13.el7ost.noarch                409/732 
16:07:30   Cleanup    : openvswitch-2.4.0-2.el7_2.x86_64                         410/732 
16:07:30   Cleanup    : fontconfig-2.10.95-7.el7.x86_64                          411/732 
16:07:30   Cleanup    : kbd-1.15.5-11.el7.x86_64                                 412/732 
16:07:31   Cleanup    : at-3.1.13-20.el7.x86_64                                  413/732 
16:07:31   Cleanup    : kernel-tools-3.10.0-327.36.3.el7.x86_64                  414/732 
16:07:31   Cleanup    : httpd-tools-2.4.6-40.el7_2.4.x86_64                      415/732 
16:07:31   Cleanup    : perl-IO-Socket-SSL-1.94-3.el7.noarch                     416/732 
16:07:31   Cleanup    : 1:facter-1.7.6-2.1.el7sat.x86_64                         417/732 
16:07:31   Cleanup    : dracut-network-033-360.el7_2.1.x86_64                    418/732 
16:07:31   Cleanup    : 12:dhclient-4.2.5-42.el7.x86_64                          419/732 
16:07:31   Cleanup    : 1:grub2-2.02-0.34.el7_2.x86_64                           420/732 
16:07:31   Cleanup    : openstack-selinux-0.7.3-3.el7ost.noarch                  421/732 
16:07:31   Cleanup    : policycoreutils-python-2.2.5-20.el7.x86_64               422/732 
16:07:31   Cleanup    : 1:grub2-tools-2.02-0.34.el7_2.x86_64                     423/732 
16:07:31   Cleanup    : 32:bind-libs-lite-9.9.4-29.el7_2.4.x86_64                424/732 
16:07:31   Cleanup    : setools-libs-3.3.7-46.el7.x86_64                         425/732 
16:07:31   Cleanup    : perl-Net-SSLeay-1.55-3.el7.x86_64                        426/732 
16:07:31   Cleanup    : os-prober-1.58-5.el7.x86_64                              427/732 
16:07:31   Cleanup    : boost-system-1.53.0-25.el7.x86_64                        428/732 
16:07:31   Cleanup    : libteam-1.17-7.el7_2.x86_64                              429/732 
16:07:31   Cleanup    : libsoup-2.48.1-3.el7.x86_64                              430/732 
16:07:31   Cleanup    : mesa-libgbm-10.6.5-3.20150824.el7.x86_64                 431/732 
16:07:31   Cleanup    : libgfortran-4.8.5-4.el7.x86_64                           432/732 
16:07:32   Cleanup    : chrony-2.1.1-1.el7.x86_64                                433/732 
16:07:32   Cleanup    : xfsprogs-3.2.2-2.el7.x86_64                              434/732 
16:07:32   Cleanup    : libproxy-0.4.11-8.el7.x86_64                             435/732 
16:07:32   Cleanup    : audit-libs-python-2.4.1-5.el7.x86_64                     436/732 
16:07:32   Cleanup    : libselinux-python-2.2.2-6.el7.x86_64                     437/732 
16:07:32   Cleanup    : libsemanage-python-2.1.10-18.el7.x86_64                  438/732 
16:07:32   Cleanup    : pciutils-3.2.1-4.el7.x86_64                              439/732 
16:07:32   Cleanup    : ipset-6.19-4.el7.x86_64                                  440/732 
16:07:32   Cleanup    : cpp-4.8.5-4.el7.x86_64                                   441/732 
16:07:32   Cleanup    : python-perf-3.10.0-327.36.3.el7.x86_64                   442/732 
16:07:32   Cleanup    : conntrack-tools-1.4.2-9.el7.x86_64                       443/732 
16:07:32   Cleanup    : logrotate-3.8.6-7.el7_2.x86_64                           444/732 
16:07:32 warning: /var/lib/logrotate.status saved as /var/lib/logrotate.status.rpmsave
16:07:32   Cleanup    : 3:irqbalance-1.0.7-5.el7.x86_64                          445/732 
16:07:32   Cleanup    : selinux-policy-targeted-3.13.1-60.el7_2.9.noarch         446/732 
16:07:32 warning: /etc/selinux/targeted/modules/active/seusers.final saved as /etc/selinux/targeted/modules/active/seusers.final.rpmsave
16:07:33   Cleanup    : openstack-tripleo-image-elements-0.9.9-6.el7ost.noarch   447/732 
16:07:33   Cleanup    : diskimage-builder-1.13.0-1.el7ost.noarch                 448/732 
16:07:33   Cleanup    : glibc-devel-2.17-106.el7_2.8.x86_64                      449/732 
16:07:33   Cleanup    : 1:openstack-heat-engine-6.0.0-9.el7ost.noarch            450/732 
16:07:33   Cleanup    : rhn-check-2.0.2-6.el7.noarch                             451/732 
16:07:33   Cleanup    : yum-rhn-plugin-2.0.1-5.el7.noarch                        452/732 
16:07:33   Cleanup    : 1:openstack-heat-api-cfn-6.0.0-9.el7ost.noarch           453/732 
16:07:34   Cleanup    : 1:openstack-heat-api-6.0.0-9.el7ost.noarch               454/732 
16:07:34   Cleanup    : rhn-setup-2.0.2-6.el7.noarch                             455/732 
16:07:34   Cleanup    : rhn-client-tools-2.0.2-6.el7.noarch                      456/732 
16:07:36   Cleanup    : glibc-headers-2.17-106.el7_2.8.x86_64                    457/732 
16:07:36   Cleanup    : selinux-policy-3.13.1-60.el7_2.9.noarch                  458/732 
16:07:37   Cleanup    : openstack-tripleo-heat-templates-2.0.0-33.el7ost.noarc   459/732 
16:07:37   Cleanup    : python-aodh-tests-2.0.3-2.el7ost.noarch                  460/732 
16:07:37   Cleanup    : 1:python-cinder-tests-8.1.0-1.el7ost.noarch              461/732 
16:07:38   Cleanup    : 1:openstack-cinder-8.1.0-1.el7ost.noarch                 462/732 
16:07:38   Cleanup    : 1:python-cinder-8.1.0-1.el7ost.noarch                    463/732 
16:07:38   Cleanup    : python-rtslib-2.1.fb57-3.el7.noarch                      464/732 
16:07:38   Cleanup    : 1:python-heat-tests-6.0.0-9.el7ost.noarch                465/732 
16:07:38   Cleanup    : 1:openstack-heat-common-6.0.0-9.el7ost.noarch            466/732 
16:07:38   Cleanup    : 1:python-neutron-tests-8.1.2-4.el7ost.noarch             467/732 
16:07:38   Cleanup    : 1:python-nova-tests-13.1.1-2.el7ost.noarch               468/732 
16:07:38   Cleanup    : 1:openstack-nova-13.1.1-2.el7ost.noarch                  469/732 


Expected result:
The yum update on UC should complete.
Comment 1 Alexander Chuzhoy 2016-11-08 16:44:51 UTC 
The issue always reproduces.
Comment 2 Marios Andreou 2016-11-08 17:48:19 UTC 
could be related to https://bugzilla.redhat.com/show_bug.cgi?id=1392010 as discussed on lifecycle scrum today (suggested by lbezdick)
Comment 3 Sofer Athlan-Guyot 2016-11-09 15:38:50 UTC 
Hi,

just a comment here.  In major upgrade procedure we have to stop all services before doing the yum upgrade (ie done in undercloud upgrade).

So here what is done doing upgrade:

    sudo systemctl stop 'openstack-*'
    sudo systemctl stop 'neutron-*'

You could try this as a workaround.
Comment 6 Lukas Bezdicka 2016-11-10 11:40:12 UTC 
It's stuck in cleanup phase when post scripts happen and usually that's restarts of services. It's stuck on openstack-nova restart. I'd look closely there.
Comment 7 Lukas Bezdicka 2016-11-10 14:43:14 UTC 
stack    16889  0.0  0.0 113120  1504 ?        Ss   Nov08   0:00      \_ bash -c sudo yum localinstall -y http://download.eng.bos.redhat.com/brewroot/work/tasks/5116/12055116/selinux-policy-targeted-3.13.1-105.el7.1.noarch.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/5116/12055116/selinux-policy-3.13.1-105.el7.1.noarch.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/5116/12055116/selinux-policy-devel-3.13.1-105.el7.1.noarch.rpm && sudo yum update -y
root     20037  0.0  0.0 191252  2280 ?        S    Nov08   0:00          \_ sudo yum update -y
root     20038  0.1  2.0 577432 162056 ?       S    Nov08   3:01              \_ /usr/bin/python /bin/yum update -y
root      3144  0.0  0.0 113120  1368 ?        S    Nov08   0:00                  \_ /bin/sh /var/tmp/rpm-tmp.g9SnEP 1
root      3159  0.0  0.0 128408  1296 ?        S    Nov08   0:01                      \_ systemctl try-restart openstack-nova-compute.service
Comment 8 Lukas Bezdicka 2016-11-10 14:43:42 UTC 
2016-11-10 09:41:08.954 3183 ERROR oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] AMQP server on 192.168.0.1:5672 is unreachable: timed out. Trying again in 32 seconds.
2016-11-10 09:41:46.010 3183 DEBUG oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabb
it.py:678
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     return self.connection
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     self._connection = self._establish_connection()
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     conn = self.transport.establish_connection()
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     conn = self.Connection(**opts)
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     self.transport = self.Transport(host, connect_timeout, ssl)
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     return create_transport(host, connect_timeout, ssl)
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     return TCPTransport(host, connect_timeout)
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit     raise socket.error(last_err)
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit error: timed out
2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit 
2016-11-10 09:41:46.013 3183 ERROR oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] AMQP server on 192.168.0.1:5672 is unreachable: timed out. Trying again in 32 seconds.
2016-11-10 09:42:23.072 3183 DEBUG oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:678
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     return self.connection
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     self._connection = self._establish_connection()
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     conn = self.transport.establish_connection()
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     conn = self.Connection(**opts)
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     self.transport = self.Transport(host, connect_timeout, ssl)
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     return create_transport(host, connect_timeout, ssl)
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     return TCPTransport(host, connect_timeout)
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit     raise socket.error(last_err)
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit error: timed out
2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit 
2016-11-10 09:42:23.075 3183 ERROR oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] AMQP server on 192.168.0.1:5672 is unreachable: timed out. Trying again in 32 seconds.
2016-11-10 09:43:00.130 3183 DEBUG oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:678
Comment 9 Lukas Bezdicka 2016-11-10 14:45:07 UTC 
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/rabbitmq-server.service.d
           └─limits.conf
   Active: active (running) since Fri 2016-09-02 14:08:41 EDT; 2 months 7 days ago
 Main PID: 30323 (beam.smp)
   Status: "Initialized"
   CGroup: /system.slice/rabbitmq-server.service
           ├─30323 /usr/lib64/erlang/erts-7.3.1/bin/beam.smp -W w -A 64 -P 1048576 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.3/ebin -noshell -noinput -s...
           ├─30525 inet_gethost 4
           └─30526 inet_gethost 4

Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Starting RabbitMQ broker...
Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: RabbitMQ 3.6.3. Copyright (C) 2007-2016 Pivotal Software, Inc.
Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ##  ##
Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ##########  Logs: /var/log/rabbitmq/rabbit
Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ######  ##        /var/log/rabbitmq/rabbit
Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ##########
Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: Starting broker...
Sep 02 14:08:41 undercloud72.localdomain systemd[1]: Started RabbitMQ broker.
Sep 02 14:08:41 undercloud72.localdomain rabbitmq-server[30323]: completed with 6 plugins.
Comment 10 Lukas Bezdicka 2016-11-10 14:45:35 UTC 
root@undercloud72 ~]# netstat -anp | grep 5672 
tcp        0      0 192.168.0.1:15672       0.0.0.0:*               LISTEN      30323/beam.smp      
tcp        0      0 192.168.0.1:5672        0.0.0.0:*               LISTEN      30323/beam.smp      
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:48304       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44734       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40348       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44440       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43526       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43534       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40400       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:48568       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40398       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:34366       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40384       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46626       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:58106       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46760       ESTABLISHED 30323/beam.smp      
tcp        0      1 10.19.184.237:38988     192.168.0.1:5672        SYN_SENT    2151/python         
tcp        0      0 192.168.0.1:5672        192.168.0.1:43542       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40346       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44430       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:58168       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44916       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:52368       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46610       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46758       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:58822       ESTABLISHED 30323/beam.smp      
tcp        0      1 10.19.184.237:38990     192.168.0.1:5672        SYN_SENT    30419/python2       
tcp        0      0 192.168.0.1:5672        192.168.0.1:44752       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40390       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44428       ESTABLISHED 30323/beam.smp      
tcp        0      1 10.19.184.237:39000     192.168.0.1:5672        SYN_SENT    2154/python         
tcp        0      0 192.168.0.1:5672        192.168.0.1:47412       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47416       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:59360       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47426       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44762       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47418       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:48024       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40350       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47168       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46774       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43558       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:60146       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40388       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:60150       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46612       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:59318       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40392       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:60148       ESTABLISHED 30323/beam.smp      
tcp        0      1 10.19.184.237:38996     192.168.0.1:5672        SYN_SENT    2150/python         
tcp        0      0 192.168.0.1:5672        192.168.0.1:47410       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47528       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47236       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46728       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44444       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47420       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46608       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46736       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44432       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47474       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:49150       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:48282       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:50406       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43540       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47424       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44434       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43368       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:48746       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46724       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44732       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44874       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40386       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43638       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47414       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43364       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:43532       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:46624       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44442       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47470       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:48894       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:44436       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40402       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:48110       ESTABLISHED 30323/beam.smp      
tcp        0      1 10.19.184.237:38992     192.168.0.1:5672        SYN_SENT    31880/python2       
tcp        0      0 192.168.0.1:5672        192.168.0.1:43510       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:47406       ESTABLISHED 30323/beam.smp      
tcp        0      1 10.19.184.237:38998     192.168.0.1:5672        SYN_SENT    31881/python2       
tcp        0      0 192.168.0.1:5672        192.168.0.1:40394       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:55530       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:40426       ESTABLISHED 30323/beam.smp      
tcp        0      0 192.168.0.1:5672        192.168.0.1:45080       ESTABLISHED 30323/beam.smp      
tcp        0      1 10.19.184.237:39002     192.168.0.1:5672        SYN_SENT    2149/python         
tcp        0      0 192.168.0.1:5672        192.168.0.1:46780       ESTABLISHED 30323/beam.smp
Comment 11 Lukas Bezdicka 2016-11-10 14:46:47 UTC 
=INFO REPORT==== 2-Sep-2016::16:10:29 ===
accepting AMQP connection <0.28727.1> (192.168.0.1:40388 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:29 ===
accepting AMQP connection <0.28730.1> (192.168.0.1:40384 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:29 ===
accepting AMQP connection <0.28760.1> (192.168.0.1:40390 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:29 ===
accepting AMQP connection <0.28763.1> (192.168.0.1:40392 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:29 ===
accepting AMQP connection <0.28785.1> (192.168.0.1:40394 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:29 ===
accepting AMQP connection <0.28809.1> (192.168.0.1:40398 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:29 ===
accepting AMQP connection <0.28823.1> (192.168.0.1:40400 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:30 ===
accepting AMQP connection <0.28841.1> (192.168.0.1:40402 -> 192.168.0.1:5672)

=INFO REPORT==== 2-Sep-2016::16:10:31 ===
accepting AMQP connection <0.28864.1> (192.168.0.1:40426 -> 192.168.0.1:5672)

=ERROR REPORT==== 8-Nov-2016::14:56:53 ===
closing AMQP connection <0.9199.0> (192.168.0.1:45220 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:56:54 ===
closing AMQP connection <0.9210.0> (192.168.0.1:45222 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:57:00 ===
closing AMQP connection <0.7639.0> (192.168.0.1:44510 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:57:12 ===
closing AMQP connection <0.9383.0> (192.168.0.1:45262 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:57:44 ===
closing AMQP connection <0.7496.0> (192.168.0.1:44494 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:57:53 ===
closing AMQP connection <0.9182.0> (192.168.0.1:45218 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:57:59 ===
closing AMQP connection <0.7636.0> (192.168.0.1:44508 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:58:03 ===
closing AMQP connection <0.7696.0> (192.168.0.1:44536 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s

=ERROR REPORT==== 8-Nov-2016::14:58:12 ===
closing AMQP connection <0.9400.0> (192.168.0.1:45264 -> 192.168.0.1:5672):
missed heartbeats from client, timeout: 60s
Comment 12 Lukas Bezdicka 2016-11-10 14:48:19 UTC 
Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Starting RabbitMQ broker...
Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Starting Erlang Port Mapper Daemon Activation Socket.
Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Unit epmd.0.0.socket entered failed state.
Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Failed to listen on Erlang Port Mapper Daemon Activation Socket.
Sep 02 14:08:29 undercloud72.localdomain systemd[1]: epmd.0.0.socket failed to listen on sockets: Address already in use
Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding 
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Reloading.
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic API service...
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Started OpenStack Ironic API service.
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding 
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Reloading.
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic Conductor service...
Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Started OpenStack Ironic Conductor service.
Comment 13 Lukas Bezdicka 2016-11-10 14:48:57 UTC 
Sep 02 13:55:09 undercloud72.localdomain yum[30561]: Installed: rabbitmq-server-3.6.3-5.el7ost.noarch
Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding 
Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Reloading.
Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding 
Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Reloading.
Sep 02 13:55:07 undercloud72.localdomain useradd[30595]: new user: name=rabbitmq, UID=988, GID=986, home=/var/lib/rabbitmq, shell=/sbin/nologin
Sep 02 13:55:06 undercloud72.localdomain groupadd[30590]: new group: name=rabbitmq, GID=986
Sep 02 13:55:06 undercloud72.localdomain groupadd[30590]: group added to /etc/gshadow: name=rabbitmq
Sep 02 13:55:06 undercloud72.localdomain groupadd[30590]: group added to /etc/group: name=rabbitmq, GID=986
Comment 14 Lukas Bezdicka 2016-11-10 14:51:09 UTC 
Nov 08 14:57:22 undercloud72.localdomain nova-compute[3183]: Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".
Nov 08 14:57:22 undercloud72.localdomain nova-compute[3183]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
Nov 08 14:57:18 undercloud72.localdomain object-server[3184]: Object audit (ZBF) "forever" mode completed: 0.52s. Total quarantined: 0, Total errors: 0, Total files/sec: 51.46, Total bytes/sec: 0.00, Auditing time: 0.50, Rate: 0.95
Nov 08 14:57:18 undercloud72.localdomain object-server[3184]: Object audit (ZBF). Since Tue Nov  8 19:57:18 2016: Locally: 1 passed, 0 quarantined, 0 errors, files/sec: 311.43, bytes/sec: 0.00, Total time: 0.00, Auditing time: 0.00, Rate:
Nov 08 14:57:18 undercloud72.localdomain object-server[3184]: Begin object audit "forever" mode (ZBF)
Nov 08 14:57:17 undercloud72.localdomain systemd[1]: Starting OpenStack Nova Compute Server...
Nov 08 14:57:17 undercloud72.localdomain systemd[1]: openstack-nova-compute.service failed.
Nov 08 14:57:17 undercloud72.localdomain systemd[1]: Unit openstack-nova-compute.service entered failed state.
Nov 08 14:57:17 undercloud72.localdomain systemd[1]: openstack-nova-compute.service: main process exited, code=killed, status=14/ALRM
Comment 15 Lukas Bezdicka 2016-11-10 14:59:27 UTC 
Sep 02 13:55:09 Installed: rabbitmq-server-3.6.3-5.el7ost.noarch
Sep 02 13:59:39 Installed: 1:openstack-nova-compute-13.1.1-2.el7ost.noarch
Sep 02 16:00:58 Updated: 1:openstack-nova-compute-13.1.1-7.el7ost.noarch
Comment 16 Lukas Bezdicka 2016-11-10 15:49:58 UTC 
I suggest running systemctl stop openstack* ; openstack undercloud upgrade
Comment 18 Alexander Chuzhoy 2016-11-11 03:22:26 UTC 
1. sudo yum -y update python-tripleoclient didn't update the rpm - no newer version.

openstack undercloud upgrade gets stuck as seen below:

01:03:50 Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-lua.conf]/ensure: removed
01:03:50 Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-dav.conf]/ensure: removed

01:03:52 Notice: /Stage[main]/Main/Nova_config[DEFAULT/my_ip]/value: value changed '192.168.0.1' to '10.19.184.237'

01:03:54 Notice: /Stage[main]/Nova::Deps/Anchor[nova::config::end]: Triggered 'refresh' from 1 events

01:08:54 Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Failed to call refresh: Command exceeded timeout
01:08:54 Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Command exceeded timeout
01:08:54 Wrapped exception:
01:08:54 execution expired

01:13:54 Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: Failed to call refresh: Command exceeded timeout
01:13:54 Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: Command exceeded timeout
01:13:54 Wrapped exception:
01:13:54 execution expired
01:13:54 Notice: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: Triggered 'refresh' from 1 events

01:13:58 Notice: /Stage[main]/Nova::Cert/Nova::Generic_service[cert]/Service[nova-cert]/ensure: ensure changed 'stopped' to 'running'

01:14:04 Notice: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]/ensure: ensure changed 'stopped' to 'running'
Comment 19 Lukas Bezdicka 2016-11-11 08:40:14 UTC 
same thing:
Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Starting RabbitMQ broker...
Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Starting Erlang Port Mapper Daemon Activation Socket.
Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Unit epmd.0.0.socket entered failed state.
Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Failed to listen on Erlang Port Mapper Daemon Activation Socket.
Sep 02 14:08:12 undercloud72.localdomain systemd[1]: epmd.0.0.socket failed to listen on sockets: Address already in use
Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding 
Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Reloading.
Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic API service...
Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Started OpenStack Ironic API service.
Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding 
Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Reloading.
Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Failed to receive notification message: Resource temporarily unavailable
Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic Conductor service...
Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Started OpenStack Ironic Conductor service.
Comment 20 Lukas Bezdicka 2016-11-11 08:49:24 UTC 
but I was still wrong, this time recovery was to return back lost ip to eth1 - services couldnt connect to 192.168.0.1
Comment 21 Lukas Bezdicka 2016-11-11 08:58:44 UTC 
ep 02 16:05:01 undercloud72.localdomain sudo[356]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
Sep 02 16:04:59 undercloud72.localdomain neutron-openvswitch-agent[325]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
Sep 02 16:04:58 undercloud72.localdomain neutron-openvswitch-agent[325]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Sep 02 16:04:57 undercloud72.localdomain systemd[1]: [/usr/lib/systemd/system/epmd@.service:18] Failed to parse resource value, ignoring: 0
Sep 02 16:04:57 undercloud72.localdomain systemd[1]: Reloading.
Sep 02 16:04:56 undercloud72.localdomain neutron-openvswitch-agent[325]: Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUS
Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Starting OpenStack Neutron Open vSwitch Agent...
Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Started OpenStack Neutron Open vSwitch Agent.
Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Started Open vSwitch.
Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Starting Open vSwitch...
Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Started Open vSwitch Internal Unit.
Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Enabling remote OVSDB managers [  OK  ]
Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Starting ovs-vswitchd [  OK  ]
Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info>  (br-ctlplane): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 10)
Sep 02 16:04:53 undercloud72.localdomain kernel: device br-ctlplane entered promiscuous mode
Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info>  (eth1): enslaved to non-master-type device ovs-system; ignoring
Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info>  (br-int): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 9)
Sep 02 16:04:53 undercloud72.localdomain kernel: device eth1 entered promiscuous mode
Sep 02 16:04:53 undercloud72.localdomain kernel: device br-int entered promiscuous mode
Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info>  (tap27cc6be3-a5): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 8)
Sep 02 16:04:53 undercloud72.localdomain kernel: device tap27cc6be3-a5 entered promiscuous mode
Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Configuring Open vSwitch system IDs [  OK  ]
Sep 02 16:04:53 undercloud72.localdomain ovs-vsctl[32763]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.5.0 "external-ids:system-id=\"0a1bba84-6321-4832-b11e-74468c9b120f\"" "system-type=\"RedHatEnte
Sep 02 16:04:53 undercloud72.localdomain ovs-vsctl[32758]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1
Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Starting ovsdb-server [  OK  ]
Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Starting Open vSwitch Internal Unit...
Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32598]: Exiting ovsdb-server (29098) [  OK  ]
Sep 02 16:04:52 undercloud72.localdomain kernel: device eth1 left promiscuous mode
Sep 02 16:04:52 undercloud72.localdomain NetworkManager[485]: <warn>  (br-ctlplane): failed to disable userspace IPv6LL address handling
Sep 02 16:04:52 undercloud72.localdomain kernel: device br-ctlplane left promiscuous mode
Sep 02 16:04:52 undercloud72.localdomain kernel: device tap27cc6be3-a5 left promiscuous mode
Sep 02 16:04:52 undercloud72.localdomain systemd-udevd[17454]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Sep 02 16:04:52 undercloud72.localdomain NetworkManager[485]: <warn>  (br-int): failed to disable userspace IPv6LL address handling
Sep 02 16:04:52 undercloud72.localdomain kernel: device br-int left promiscuous mode
Sep 02 16:04:52 undercloud72.localdomain ovs-ctl[32598]: Exiting ovs-vswitchd (29109) [  OK  ]
Sep 02 16:04:52 undercloud72.localdomain systemd[1]: Stopping Open vSwitch Internal Unit...
Sep 02 16:04:52 undercloud72.localdomain systemd[1]: Stopping Open vSwitch...


since than no connectivity to local services on 192.168.0.1
Comment 23 Miroslav Grepl 2016-11-11 15:17:23 UTC 
It looks like the policy was broken somehow on the updated system?

Are we able to reproduce it?

Could you try to reinstall the policy on updated system?

# yum reinstall selinux-policy-targeted
Comment 25 Adriano Petrich 2016-11-14 10:27:35 UTC 
I was unable to reproduce it, trying again. Anyone had any luck reproducing this?
Comment 27 Marios Andreou 2016-11-14 13:28:14 UTC 
Lucas filed BZ 1394778 - Upgrade of openvswitch-2.4.0-1.el7 makes ip disappears

to track the issue seen after working around the selinux-targetted issue.
Comment 29 Alexander Chuzhoy 2016-11-24 15:01:32 UTC 
Verified:
The reported issue doesn't reproduce.
Note You need to log in before you can comment on or make changes to this bug.