Bug 1393005
Summary: | rhel-osp-director: 9.0GA->latest with rhel7.2->rhel7.3 switch: undercloud yum update doesn't complete: ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.30. | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
Component: | rhosp-director | Assignee: | Lukas Bezdicka <lbezdick> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Alexander Chuzhoy <sasha> |
Severity: | urgent | Docs Contact: | |
Priority: | urgent | ||
Version: | 9.0 (Mitaka) | CC: | apetrich, dbecker, jcoufal, jjoyce, lvrabec, mandreou, mburns, mgrepl, morazi, rcritten, rhel-osp-director-maint, sasha, sathlang |
Target Milestone: | rc | Keywords: | Regression, Triaged |
Target Release: | 9.0 (Mitaka) | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-05-05 16:45:55 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1393045 | ||
Bug Blocks: |
Description
Alexander Chuzhoy
2016-11-08 16:43:02 UTC
The issue always reproduces. could be related to https://bugzilla.redhat.com/show_bug.cgi?id=1392010 as discussed on lifecycle scrum today (suggested by lbezdick) Hi, just a comment here. In major upgrade procedure we have to stop all services before doing the yum upgrade (ie done in undercloud upgrade). So here what is done doing upgrade: sudo systemctl stop 'openstack-*' sudo systemctl stop 'neutron-*' You could try this as a workaround. It's stuck in cleanup phase when post scripts happen and usually that's restarts of services. It's stuck on openstack-nova restart. I'd look closely there. stack 16889 0.0 0.0 113120 1504 ? Ss Nov08 0:00 \_ bash -c sudo yum localinstall -y http://download.eng.bos.redhat.com/brewroot/work/tasks/5116/12055116/selinux-policy-targeted-3.13.1-105.el7.1.noarch.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/5116/12055116/selinux-policy-3.13.1-105.el7.1.noarch.rpm http://download.eng.bos.redhat.com/brewroot/work/tasks/5116/12055116/selinux-policy-devel-3.13.1-105.el7.1.noarch.rpm && sudo yum update -y root 20037 0.0 0.0 191252 2280 ? S Nov08 0:00 \_ sudo yum update -y root 20038 0.1 2.0 577432 162056 ? S Nov08 3:01 \_ /usr/bin/python /bin/yum update -y root 3144 0.0 0.0 113120 1368 ? S Nov08 0:00 \_ /bin/sh /var/tmp/rpm-tmp.g9SnEP 1 root 3159 0.0 0.0 128408 1296 ? S Nov08 0:01 \_ systemctl try-restart openstack-nova-compute.service 2016-11-10 09:41:08.954 3183 ERROR oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] AMQP server on 192.168.0.1:5672 is unreachable: timed out. Trying again in 32 seconds. 2016-11-10 09:41:46.010 3183 DEBUG oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabb it.py:678 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit return self.connection 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection() 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection() 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit conn = self.Connection(**opts) 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__ 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit self.transport = self.Transport(host, connect_timeout, ssl) 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit return create_transport(host, connect_timeout, ssl) 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit return TCPTransport(host, connect_timeout) 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__ 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit raise socket.error(last_err) 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit error: timed out 2016-11-10 09:41:46.010 3183 ERROR oslo.messaging._drivers.impl_rabbit 2016-11-10 09:41:46.013 3183 ERROR oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] AMQP server on 192.168.0.1:5672 is unreachable: timed out. Trying again in 32 seconds. 2016-11-10 09:42:23.072 3183 DEBUG oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:678 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit return self.connection 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection() 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection() 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit conn = self.Connection(**opts) 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__ 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit self.transport = self.Transport(host, connect_timeout, ssl) 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit return create_transport(host, connect_timeout, ssl) 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit return TCPTransport(host, connect_timeout) 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__ 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit raise socket.error(last_err) 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit error: timed out 2016-11-10 09:42:23.072 3183 ERROR oslo.messaging._drivers.impl_rabbit 2016-11-10 09:42:23.075 3183 ERROR oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] AMQP server on 192.168.0.1:5672 is unreachable: timed out. Trying again in 32 seconds. 2016-11-10 09:43:00.130 3183 DEBUG oslo.messaging._drivers.impl_rabbit [req-589bf545-b5da-4e6d-ac2d-2962ad78b54d - - - - -] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:678 ● rabbitmq-server.service - RabbitMQ broker Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/rabbitmq-server.service.d └─limits.conf Active: active (running) since Fri 2016-09-02 14:08:41 EDT; 2 months 7 days ago Main PID: 30323 (beam.smp) Status: "Initialized" CGroup: /system.slice/rabbitmq-server.service ├─30323 /usr/lib64/erlang/erts-7.3.1/bin/beam.smp -W w -A 64 -P 1048576 -K true -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.3/ebin -noshell -noinput -s... ├─30525 inet_gethost 4 └─30526 inet_gethost 4 Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Starting RabbitMQ broker... Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: RabbitMQ 3.6.3. Copyright (C) 2007-2016 Pivotal Software, Inc. Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ## ## Licensed under the MPL. See http://www.rabbitmq.com/ Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ## ## Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ########## Logs: /var/log/rabbitmq/rabbit Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ###### ## /var/log/rabbitmq/rabbit Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: ########## Sep 02 14:08:33 undercloud72.localdomain rabbitmq-server[30323]: Starting broker... Sep 02 14:08:41 undercloud72.localdomain systemd[1]: Started RabbitMQ broker. Sep 02 14:08:41 undercloud72.localdomain rabbitmq-server[30323]: completed with 6 plugins. root@undercloud72 ~]# netstat -anp | grep 5672 tcp 0 0 192.168.0.1:15672 0.0.0.0:* LISTEN 30323/beam.smp tcp 0 0 192.168.0.1:5672 0.0.0.0:* LISTEN 30323/beam.smp tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:48304 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44734 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40348 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44440 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43526 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43534 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40400 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:48568 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40398 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:34366 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40384 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46626 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:58106 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46760 ESTABLISHED 30323/beam.smp tcp 0 1 10.19.184.237:38988 192.168.0.1:5672 SYN_SENT 2151/python tcp 0 0 192.168.0.1:5672 192.168.0.1:43542 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40346 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44430 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:58168 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44916 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:52368 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46610 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46758 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:58822 ESTABLISHED 30323/beam.smp tcp 0 1 10.19.184.237:38990 192.168.0.1:5672 SYN_SENT 30419/python2 tcp 0 0 192.168.0.1:5672 192.168.0.1:44752 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40390 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44428 ESTABLISHED 30323/beam.smp tcp 0 1 10.19.184.237:39000 192.168.0.1:5672 SYN_SENT 2154/python tcp 0 0 192.168.0.1:5672 192.168.0.1:47412 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47416 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:59360 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47426 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44762 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47418 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:48024 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40350 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47168 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46774 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43558 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:60146 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40388 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:60150 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46612 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:59318 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40392 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:60148 ESTABLISHED 30323/beam.smp tcp 0 1 10.19.184.237:38996 192.168.0.1:5672 SYN_SENT 2150/python tcp 0 0 192.168.0.1:5672 192.168.0.1:47410 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47528 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47236 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46728 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44444 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47420 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46608 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46736 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44432 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47474 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:49150 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:48282 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:50406 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43540 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47424 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44434 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43368 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:48746 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46724 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44732 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44874 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40386 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43638 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47414 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43364 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:43532 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:46624 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44442 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47470 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:48894 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:44436 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40402 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:48110 ESTABLISHED 30323/beam.smp tcp 0 1 10.19.184.237:38992 192.168.0.1:5672 SYN_SENT 31880/python2 tcp 0 0 192.168.0.1:5672 192.168.0.1:43510 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:47406 ESTABLISHED 30323/beam.smp tcp 0 1 10.19.184.237:38998 192.168.0.1:5672 SYN_SENT 31881/python2 tcp 0 0 192.168.0.1:5672 192.168.0.1:40394 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:55530 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:40426 ESTABLISHED 30323/beam.smp tcp 0 0 192.168.0.1:5672 192.168.0.1:45080 ESTABLISHED 30323/beam.smp tcp 0 1 10.19.184.237:39002 192.168.0.1:5672 SYN_SENT 2149/python tcp 0 0 192.168.0.1:5672 192.168.0.1:46780 ESTABLISHED 30323/beam.smp =INFO REPORT==== 2-Sep-2016::16:10:29 === accepting AMQP connection <0.28727.1> (192.168.0.1:40388 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:29 === accepting AMQP connection <0.28730.1> (192.168.0.1:40384 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:29 === accepting AMQP connection <0.28760.1> (192.168.0.1:40390 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:29 === accepting AMQP connection <0.28763.1> (192.168.0.1:40392 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:29 === accepting AMQP connection <0.28785.1> (192.168.0.1:40394 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:29 === accepting AMQP connection <0.28809.1> (192.168.0.1:40398 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:29 === accepting AMQP connection <0.28823.1> (192.168.0.1:40400 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:30 === accepting AMQP connection <0.28841.1> (192.168.0.1:40402 -> 192.168.0.1:5672) =INFO REPORT==== 2-Sep-2016::16:10:31 === accepting AMQP connection <0.28864.1> (192.168.0.1:40426 -> 192.168.0.1:5672) =ERROR REPORT==== 8-Nov-2016::14:56:53 === closing AMQP connection <0.9199.0> (192.168.0.1:45220 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:56:54 === closing AMQP connection <0.9210.0> (192.168.0.1:45222 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:57:00 === closing AMQP connection <0.7639.0> (192.168.0.1:44510 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:57:12 === closing AMQP connection <0.9383.0> (192.168.0.1:45262 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:57:44 === closing AMQP connection <0.7496.0> (192.168.0.1:44494 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:57:53 === closing AMQP connection <0.9182.0> (192.168.0.1:45218 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:57:59 === closing AMQP connection <0.7636.0> (192.168.0.1:44508 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:58:03 === closing AMQP connection <0.7696.0> (192.168.0.1:44536 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s =ERROR REPORT==== 8-Nov-2016::14:58:12 === closing AMQP connection <0.9400.0> (192.168.0.1:45264 -> 192.168.0.1:5672): missed heartbeats from client, timeout: 60s Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Starting RabbitMQ broker... Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Starting Erlang Port Mapper Daemon Activation Socket. Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Unit epmd.0.0.socket entered failed state. Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Failed to listen on Erlang Port Mapper Daemon Activation Socket. Sep 02 14:08:29 undercloud72.localdomain systemd[1]: epmd.0.0.socket failed to listen on sockets: Address already in use Sep 02 14:08:29 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Reloading. Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic API service... Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Started OpenStack Ironic API service. Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Reloading. Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic Conductor service... Sep 02 14:08:28 undercloud72.localdomain systemd[1]: Started OpenStack Ironic Conductor service. Sep 02 13:55:09 undercloud72.localdomain yum[30561]: Installed: rabbitmq-server-3.6.3-5.el7ost.noarch Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Reloading. Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding Sep 02 13:55:09 undercloud72.localdomain systemd[1]: Reloading. Sep 02 13:55:07 undercloud72.localdomain useradd[30595]: new user: name=rabbitmq, UID=988, GID=986, home=/var/lib/rabbitmq, shell=/sbin/nologin Sep 02 13:55:06 undercloud72.localdomain groupadd[30590]: new group: name=rabbitmq, GID=986 Sep 02 13:55:06 undercloud72.localdomain groupadd[30590]: group added to /etc/gshadow: name=rabbitmq Sep 02 13:55:06 undercloud72.localdomain groupadd[30590]: group added to /etc/group: name=rabbitmq, GID=986 Nov 08 14:57:22 undercloud72.localdomain nova-compute[3183]: Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications". Nov 08 14:57:22 undercloud72.localdomain nova-compute[3183]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications". Nov 08 14:57:18 undercloud72.localdomain object-server[3184]: Object audit (ZBF) "forever" mode completed: 0.52s. Total quarantined: 0, Total errors: 0, Total files/sec: 51.46, Total bytes/sec: 0.00, Auditing time: 0.50, Rate: 0.95 Nov 08 14:57:18 undercloud72.localdomain object-server[3184]: Object audit (ZBF). Since Tue Nov 8 19:57:18 2016: Locally: 1 passed, 0 quarantined, 0 errors, files/sec: 311.43, bytes/sec: 0.00, Total time: 0.00, Auditing time: 0.00, Rate: Nov 08 14:57:18 undercloud72.localdomain object-server[3184]: Begin object audit "forever" mode (ZBF) Nov 08 14:57:17 undercloud72.localdomain systemd[1]: Starting OpenStack Nova Compute Server... Nov 08 14:57:17 undercloud72.localdomain systemd[1]: openstack-nova-compute.service failed. Nov 08 14:57:17 undercloud72.localdomain systemd[1]: Unit openstack-nova-compute.service entered failed state. Nov 08 14:57:17 undercloud72.localdomain systemd[1]: openstack-nova-compute.service: main process exited, code=killed, status=14/ALRM Sep 02 13:55:09 Installed: rabbitmq-server-3.6.3-5.el7ost.noarch Sep 02 13:59:39 Installed: 1:openstack-nova-compute-13.1.1-2.el7ost.noarch Sep 02 16:00:58 Updated: 1:openstack-nova-compute-13.1.1-7.el7ost.noarch I suggest running systemctl stop openstack* ; openstack undercloud upgrade 1. sudo yum -y update python-tripleoclient didn't update the rpm - no newer version. openstack undercloud upgrade gets stuck as seen below: 01:03:50 Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-lua.conf]/ensure: removed 01:03:50 Notice: /Stage[main]/Apache/File[/etc/httpd/conf.modules.d/00-dav.conf]/ensure: removed 01:03:52 Notice: /Stage[main]/Main/Nova_config[DEFAULT/my_ip]/value: value changed '192.168.0.1' to '10.19.184.237' 01:03:54 Notice: /Stage[main]/Nova::Deps/Anchor[nova::config::end]: Triggered 'refresh' from 1 events 01:08:54 Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Failed to call refresh: Command exceeded timeout 01:08:54 Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Command exceeded timeout 01:08:54 Wrapped exception: 01:08:54 execution expired 01:13:54 Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: Failed to call refresh: Command exceeded timeout 01:13:54 Error: /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api]: Command exceeded timeout 01:13:54 Wrapped exception: 01:13:54 execution expired 01:13:54 Notice: /Stage[main]/Nova::Deps/Anchor[nova::service::begin]: Triggered 'refresh' from 1 events 01:13:58 Notice: /Stage[main]/Nova::Cert/Nova::Generic_service[cert]/Service[nova-cert]/ensure: ensure changed 'stopped' to 'running' 01:14:04 Notice: /Stage[main]/Nova::Api/Nova::Generic_service[api]/Service[nova-api]/ensure: ensure changed 'stopped' to 'running' same thing: Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Starting RabbitMQ broker... Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Starting Erlang Port Mapper Daemon Activation Socket. Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Unit epmd.0.0.socket entered failed state. Sep 02 14:08:12 undercloud72.localdomain systemd[1]: Failed to listen on Erlang Port Mapper Daemon Activation Socket. Sep 02 14:08:12 undercloud72.localdomain systemd[1]: epmd.0.0.socket failed to listen on sockets: Address already in use Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Reloading. Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic API service... Sep 02 14:08:11 undercloud72.localdomain systemd[1]: Started OpenStack Ironic API service. Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/wpa_supplicant.service is marked executable. Please remove executable permission bits. Proceeding anyway. Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Reloading. Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Failed to receive notification message: Resource temporarily unavailable Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Starting OpenStack Ironic Conductor service... Sep 02 14:08:10 undercloud72.localdomain systemd[1]: Started OpenStack Ironic Conductor service. but I was still wrong, this time recovery was to return back lost ip to eth1 - services couldnt connect to 192.168.0.1 ep 02 16:05:01 undercloud72.localdomain sudo[356]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf Sep 02 16:04:59 undercloud72.localdomain neutron-openvswitch-agent[325]: Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications". Sep 02 16:04:58 undercloud72.localdomain neutron-openvswitch-agent[325]: Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future. Sep 02 16:04:57 undercloud72.localdomain systemd[1]: [/usr/lib/systemd/system/epmd@.service:18] Failed to parse resource value, ignoring: 0 Sep 02 16:04:57 undercloud72.localdomain systemd[1]: Reloading. Sep 02 16:04:56 undercloud72.localdomain neutron-openvswitch-agent[325]: Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUS Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Starting OpenStack Neutron Open vSwitch Agent... Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Started OpenStack Neutron Open vSwitch Agent. Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Started Open vSwitch. Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Starting Open vSwitch... Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Started Open vSwitch Internal Unit. Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Enabling remote OVSDB managers [ OK ] Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Starting ovs-vswitchd [ OK ] Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info> (br-ctlplane): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 10) Sep 02 16:04:53 undercloud72.localdomain kernel: device br-ctlplane entered promiscuous mode Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info> (eth1): enslaved to non-master-type device ovs-system; ignoring Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info> (br-int): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 9) Sep 02 16:04:53 undercloud72.localdomain kernel: device eth1 entered promiscuous mode Sep 02 16:04:53 undercloud72.localdomain kernel: device br-int entered promiscuous mode Sep 02 16:04:53 undercloud72.localdomain NetworkManager[485]: <info> (tap27cc6be3-a5): new Generic device (carrier: OFF, driver: 'openvswitch', ifindex: 8) Sep 02 16:04:53 undercloud72.localdomain kernel: device tap27cc6be3-a5 entered promiscuous mode Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Configuring Open vSwitch system IDs [ OK ] Sep 02 16:04:53 undercloud72.localdomain ovs-vsctl[32763]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.5.0 "external-ids:system-id=\"0a1bba84-6321-4832-b11e-74468c9b120f\"" "system-type=\"RedHatEnte Sep 02 16:04:53 undercloud72.localdomain ovs-vsctl[32758]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1 Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32680]: Starting ovsdb-server [ OK ] Sep 02 16:04:53 undercloud72.localdomain systemd[1]: Starting Open vSwitch Internal Unit... Sep 02 16:04:53 undercloud72.localdomain ovs-ctl[32598]: Exiting ovsdb-server (29098) [ OK ] Sep 02 16:04:52 undercloud72.localdomain kernel: device eth1 left promiscuous mode Sep 02 16:04:52 undercloud72.localdomain NetworkManager[485]: <warn> (br-ctlplane): failed to disable userspace IPv6LL address handling Sep 02 16:04:52 undercloud72.localdomain kernel: device br-ctlplane left promiscuous mode Sep 02 16:04:52 undercloud72.localdomain kernel: device tap27cc6be3-a5 left promiscuous mode Sep 02 16:04:52 undercloud72.localdomain systemd-udevd[17454]: Network interface NamePolicy= disabled on kernel command line, ignoring. Sep 02 16:04:52 undercloud72.localdomain NetworkManager[485]: <warn> (br-int): failed to disable userspace IPv6LL address handling Sep 02 16:04:52 undercloud72.localdomain kernel: device br-int left promiscuous mode Sep 02 16:04:52 undercloud72.localdomain ovs-ctl[32598]: Exiting ovs-vswitchd (29109) [ OK ] Sep 02 16:04:52 undercloud72.localdomain systemd[1]: Stopping Open vSwitch Internal Unit... Sep 02 16:04:52 undercloud72.localdomain systemd[1]: Stopping Open vSwitch... since than no connectivity to local services on 192.168.0.1 It looks like the policy was broken somehow on the updated system? Are we able to reproduce it? Could you try to reinstall the policy on updated system? # yum reinstall selinux-policy-targeted I was unable to reproduce it, trying again. Anyone had any luck reproducing this? Lucas filed BZ 1394778 - Upgrade of openvswitch-2.4.0-1.el7 makes ip disappears to track the issue seen after working around the selinux-targetted issue. Verified: The reported issue doesn't reproduce. |