rhosp-director: overcloud minor update fails with: "Warning: RPMDB altered outside of yum.\n/usr/bin/python: symbol lookup error: /lib64/libnsssysinit.so: undefined symbol: PR_GetEnvSecure\n", "update_managed_packages": "false", "deploy_status_code": 127} Environment: openstack-tripleo-heat-templates-liberty-2.0.0-54.el7ost.noarch openstack-tripleo-heat-templates-2.0.0-54.el7ost.noarch openstack-puppet-modules-8.1.10-2.el7ost.noarch instack-undercloud-4.0.0-16.el7ost.noarch Steps to reproduce: Tried to minor update overcloud with: openstack overcloud update stack overcloud -i \ --templates ~/pilot/templates/overcloud \ -e ~/pilot/templates/overcloud/overcloud-resource-registry-puppet.yaml \ -e ~/pilot/templates/overcloud/environments/network-isolation.yaml \ -e ~/pilot/templates/network-environment.yaml \ -e ~/pilot/templates/node-placement.yaml \ -e ~/pilot/templates/overcloud/environments/storage-environment.yaml \ -e ~/pilot/templates/dell-environment.yaml \ -e ~/pilot/templates/overcloud/environments/puppet-pacemaker.yaml The overcloud was deployed using: cd ; openstack overcloud deploy \ --debug \ --log-file ~/pilot/overcloud_deployment.log \ -t 400 \ --stack overcloud \ --templates ~/pilot/templates/overcloud \ -e ~/pilot/templates/overcloud/environments/network-isolation.yaml \ -e ~/pilot/templates/network-environment.yaml \ -e ~/pilot/templates/node-placement.yaml \ -e ~/pilot/templates/overcloud/environments/storage-environment.yaml \ -e ~/pilot/templates/dell-environment.yaml \ -e ~/pilot/templates/overcloud/environments/puppet-pacemaker.yaml \ --control-flavor control \ --compute-flavor compute \ --ceph-storage-flavor ceph-storage \ --swift-storage-flavor swift-storage \ --block-storage-flavor block-storage \ --neutron-public-interface bond1 \ --neutron-network-type vlan \ --neutron-disable-tunneling \ --os-auth-url http://192.168.120.101:5000/v2.0 \ --os-project-name admin \ --os-user-id admin \ --os-password a966628855740e074e62a2c0f4e14e788cd0a6f1 \ --control-scale 3 \ --compute-scale 3 \ --ceph-storage-scale 3 \ --ntp-server 0.centos.pool.ntp.org \ --neutron-network-vlan-ranges physint:201:220,physext \ --neutron-bridge-mappings physint:br-tenant,physext:br-ex Result: The minor update of overcloud fails: [stack@director ~]$ heat resource-list -n5 overcloud|grep -v COMPLE WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | CephStorage | 4bf4e3bc-7d4a-4a99-85c3-ea9e5701bee3 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2017-06-14T15:31:12 | overcloud | | 0 | 07f64f1d-f80c-4de4-9328-015aef9fef6e | OS::TripleO::CephStorage | UPDATE_IN_PROGRESS | 2017-06-14T15:31:16 | overcloud-CephStorage-6iiwtdfnx7z5 | | Compute | 10fb444a-d2bc-4109-aec8-d83677629a79 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2017-06-14T15:31:24 | overcloud | | Controller | 7f932930-bd22-4b9d-bae1-e2817d0ef6ab | OS::Heat::ResourceGroup | UPDATE_FAILED | 2017-06-14T15:31:26 | overcloud | | 1 | 4d3c51d7-b76a-4834-a3d7-ecb8a432debb | OS::TripleO::Compute | UPDATE_IN_PROGRESS | 2017-06-14T15:31:27 | overcloud-Compute-k7eqwlf2pv64 | | 0 | 00890028-d03c-4abf-bc3c-a4d305eafefc | OS::TripleO::Compute | UPDATE_IN_PROGRESS | 2017-06-14T15:31:29 | overcloud-Compute-k7eqwlf2pv64 | | 1 | 4d7747a4-bdab-425d-8396-4d193c5684cf | OS::TripleO::Controller | UPDATE_FAILED | 2017-06-14T15:31:30 | overcloud-Controller-7zmfn5uuv4nc | | 2 | e0555b88-8414-4354-a918-3551a36b672a | OS::TripleO::Compute | UPDATE_IN_PROGRESS | 2017-06-14T15:31:30 | overcloud-Compute-k7eqwlf2pv64 | | 2 | 6a5f7e6c-1e12-4dba-a92b-19ac22712d35 | OS::TripleO::Controller | UPDATE_FAILED | 2017-06-14T15:31:32 | overcloud-Controller-7zmfn5uuv4nc | | 0 | f99d6fb1-b011-4024-a988-6342b76497ad | OS::TripleO::Controller | UPDATE_FAILED | 2017-06-14T15:31:34 | overcloud-Controller-7zmfn5uuv4nc | | UpdateDeployment | f9d5362e-2eea-4de7-8b44-7a35622f34f3 | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2017-06-14T15:34:30 | overcloud-Controller-7zmfn5uuv4nc-1-4jq6hiw2wn6t | +-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ [stack@director ~]$ echo -e `heat deployment-show f9d5362e-2eea-4de7-8b44-7a35622f34f3` WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "272bc481-3c01-430d-a6d2-e504de3a6705", "config_id": "36948756-06fb-4945-95af-5646e9684aa8", "output_values": { "deploy_stdout": "Started yum_update.sh on server 272bc481-3c01-430d-a6d2-e504de3a6705 at Wed Jun 14 15:35:06 UTC 2017 Manual upgrade of openvswitch - restart in postun detected /var/lib/heat-config/heat-config-script/OVS_UPGRADE /var/lib/heat-config/heat-config-script Attempting to downloading latest openvswitch with yumdownloader Loaded plugins: product-id --> Running transaction check ---> Package openvswitch.x86_64 0:2.5.0-14.git20160727.el7fdp will be installed --> Finished Dependency Resolution Updating openvswitch with nopostun option /var/lib/heat-config/heat-config-script Pacemaker running, stopping cluster node and doing full package update Stopping Cluster (pacemaker)... Stopping Cluster (corosync)... Running: yum -q -y update Delta RPMs disabled because /usr/bin/applydeltarpm not installed. warning: /etc/os-release saved as /etc/os-release.rpmsave warning: /etc/shadow created as /etc/shadow.rpmnew warning: /etc/nsswitch.conf created as /etc/nsswitch.conf.rpmnew warning: /etc/sysctl.conf created as /etc/sysctl.conf.rpmnew warning: /etc/aodh/aodh.conf created as /etc/aodh/aodh.conf.rpmnew warning: /etc/heat/heat.conf created as /etc/heat/heat.conf.rpmnew unable to write 'random state' warning: /etc/sudoers created as /etc/sudoers.rpmnew warning: /etc/neutron/neutron.conf created as /etc/neutron/neutron.conf.rpmnew warning: /etc/nova/nova.conf created as /etc/nova/nova.conf.rpmnew warning: /etc/neutron/dhcp_agent.ini created as /etc/neutron/dhcp_agent.ini.rpmnew warning: /etc/neutron/l3_agent.ini created as /etc/neutron/l3_agent.ini.rpmnew warning: /etc/neutron/plugins/ml2/openvswitch_agent.ini created as /etc/neutron/plugins/ml2/openvswitch_agent.ini.rpmnew warning: /etc/cinder/cinder.conf created as /etc/cinder/cinder.conf.rpmnew warning: /etc/neutron/plugins/ml2/ml2_conf.ini created as /etc/neutron/plugins/ml2/ml2_conf.ini.rpmnew warning: file /usr/share/openstack-dashboard/static/dashboard/js/5ac31e075a3b.js: remove failed: No such file or directory warning: /var/lib/logrotate.status saved as /var/lib/logrotate.status.rpmsave Created symlink from /etc/systemd/system/sockets.target.wants/virtlogd.socket to /usr/lib/systemd/system/virtlogd.socket. warning: file /var/run/OpenIPMI: remove failed: No such file or directory 2670 blocks yum return code: 127 Starting cluster node Starting Cluster... HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 36 Galera cluster node is not synced. HTTP/1.1 200 OK Content-Type: text/plain Connection: close Content-Length: 32 Galera cluster node is synced. Waiting for pacemaker cluster to settle Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum Last updated: Wed Jun 14 15:48:14 2017 Last change: Tue Jun 13 23:17:32 2017 by root via cibadmin on overcloud-controller-2 3 nodes and 127 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Full list of resources: ip-192.168.140.121 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.120.127 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-192.168.120.126 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Clone Set: haproxy-clone [haproxy] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: memcached-clone [memcached] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] ip-192.168.190.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.170.120 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.140.120 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Clone Set: rabbitmq-clone [rabbitmq] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-core-clone [openstack-core] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Master/Slave Set: redis-master [redis] Masters: [ overcloud-controller-2 ] Slaves: [ overcloud-controller-0 overcloud-controller-1 ] Clone Set: mongod-clone [mongod] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-l3-agent-clone [neutron-l3-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Clone Set: openstack-heat-engine-clone [openstack-heat-engine] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-api-clone [openstack-heat-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-glance-api-clone [openstack-glance-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-api-clone [openstack-nova-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-sahara-api-clone [openstack-sahara-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-glance-registry-clone [openstack-glance-registry] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-cinder-api-clone [openstack-cinder-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: delay-clone [delay] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-server-clone [neutron-server] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: httpd-clone [httpd] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Finished yum_update.sh on server 272bc481-3c01-430d-a6d2-e504de3a6705 at Wed Jun 14 15:48:14 UTC 2017 ", "deploy_stderr": "Warning: RPMDB altered outside of yum. /usr/bin/python: symbol lookup error: /lib64/libnsssysinit.so: undefined symbol: PR_GetEnvSecure ", "update_managed_packages": "false", "deploy_status_code": 127 }, "creation_time": "2017-06-13T22:35:13", "updated_time": "2017-06-14T15:48:15", "input_values": { "update_identifier": "1497454243" }, "action": "UPDATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 127", "id": "f9d5362e-2eea-4de7-8b44-7a35622f34f3" } Checking the node where this error is reported - seems like yum update has completed. nova list |grep 272bc481-3c01-430d-a6d2-e504de3a6705 | 272bc481-3c01-430d-a6d2-e504de3a6705 | overcloud-controller-1 | ACTIVE | - | Running | ctlplane=192.168.120.131 | [stack@director ~]$ ssh heat-admin.120.131 [heat-admin@overcloud-controller-1 ~]$ sudo yum check-update Loaded plugins: product-id, search-disabled-repos, subscription-manager Security: kernel-3.10.0-514.21.1.el7.x86_64 is an installed security update Security: kernel-3.10.0-327.28.3.el7.x86_64 is the currently running version [heat-admin@overcloud-controller-1 ~]$ sudo yum update Loaded plugins: product-id, search-disabled-repos, subscription-manager No packages marked for update [heat-admin@overcloud-controller-1 ~]$
Hi Sasha, it's a know issue and the root cause is handled in 1458841. We are currently waiting for its resolution.
Successful update of RHOS-9 on RHEL-7.3 to latest on RHEL-7.4 with openstack-tripleo-heat-templates-2.0.0-60.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2813