Bug 1461531

Summary: rhosp-director: overcloud minor update fails with: "Warning: RPMDB altered outside of yum.\n/usr/bin/python: symbol lookup error: /lib64/libnsssysinit.so: undefined symbol: PR_GetEnvSecure\n", "update_managed_packages": "false", "deploy_status_code": 127}
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-heat-templatesAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: high Docs Contact:
Priority: urgent    
Version: 9.0 (Mitaka)CC: dbecker, emacchi, lbezdick, lruzicka, mbultel, mburns, mcornea, morazi, rhel-osp-director-maint, sathlang, yprokule
Target Milestone: zstreamKeywords: Triaged, ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-59.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-27 13:08:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1458841, 1469518, 1469526    
Bug Blocks:    

Description Alexander Chuzhoy 2017-06-14 17:14:47 UTC 
rhosp-director: overcloud minor update fails with: "Warning: RPMDB altered outside of yum.\n/usr/bin/python: symbol lookup error: /lib64/libnsssysinit.so: undefined symbol: PR_GetEnvSecure\n", "update_managed_packages": "false", "deploy_status_code": 127}

Environment:
openstack-tripleo-heat-templates-liberty-2.0.0-54.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-54.el7ost.noarch
openstack-puppet-modules-8.1.10-2.el7ost.noarch
instack-undercloud-4.0.0-16.el7ost.noarch


Steps to reproduce:
Tried to minor update overcloud with:
openstack overcloud update stack overcloud -i \
 --templates ~/pilot/templates/overcloud \
 -e ~/pilot/templates/overcloud/overcloud-resource-registry-puppet.yaml \
 -e ~/pilot/templates/overcloud/environments/network-isolation.yaml \
 -e ~/pilot/templates/network-environment.yaml \
 -e ~/pilot/templates/node-placement.yaml \
 -e ~/pilot/templates/overcloud/environments/storage-environment.yaml \
 -e ~/pilot/templates/dell-environment.yaml \
 -e ~/pilot/templates/overcloud/environments/puppet-pacemaker.yaml



The overcloud was deployed using:
cd ; openstack overcloud deploy \
 --debug \
 --log-file ~/pilot/overcloud_deployment.log \
 -t 400 \
 --stack overcloud \
 --templates ~/pilot/templates/overcloud \
 -e ~/pilot/templates/overcloud/environments/network-isolation.yaml \
 -e ~/pilot/templates/network-environment.yaml \
 -e ~/pilot/templates/node-placement.yaml \
 -e ~/pilot/templates/overcloud/environments/storage-environment.yaml \
 -e ~/pilot/templates/dell-environment.yaml \
 -e ~/pilot/templates/overcloud/environments/puppet-pacemaker.yaml \
 --control-flavor control \
 --compute-flavor compute \
 --ceph-storage-flavor ceph-storage \
 --swift-storage-flavor swift-storage \
 --block-storage-flavor block-storage \
 --neutron-public-interface bond1 \
 --neutron-network-type vlan \
 --neutron-disable-tunneling \
 --os-auth-url http://192.168.120.101:5000/v2.0 \
 --os-project-name admin \
 --os-user-id admin \
 --os-password a966628855740e074e62a2c0f4e14e788cd0a6f1 \
 --control-scale 3 \
 --compute-scale 3 \
 --ceph-storage-scale 3 \
 --ntp-server 0.centos.pool.ntp.org \
 --neutron-network-vlan-ranges physint:201:220,physext \
 --neutron-bridge-mappings physint:br-tenant,physext:br-ex


Result:
The minor update of overcloud fails:
[stack@director ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| resource_name                                 | physical_resource_id                          | resource_type                                                             | resource_status    | updated_time        | stack_name                                                                                                                                        |
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| CephStorage                                   | 4bf4e3bc-7d4a-4a99-85c3-ea9e5701bee3          | OS::Heat::ResourceGroup                                                   | UPDATE_FAILED      | 2017-06-14T15:31:12 | overcloud                                                                                                                                         |
| 0                                             | 07f64f1d-f80c-4de4-9328-015aef9fef6e          | OS::TripleO::CephStorage                                                  | UPDATE_IN_PROGRESS | 2017-06-14T15:31:16 | overcloud-CephStorage-6iiwtdfnx7z5                                                                                                                |
| Compute                                       | 10fb444a-d2bc-4109-aec8-d83677629a79          | OS::Heat::ResourceGroup                                                   | UPDATE_FAILED      | 2017-06-14T15:31:24 | overcloud                                                                                                                                         |
| Controller                                    | 7f932930-bd22-4b9d-bae1-e2817d0ef6ab          | OS::Heat::ResourceGroup                                                   | UPDATE_FAILED      | 2017-06-14T15:31:26 | overcloud                                                                                                                                         |
| 1                                             | 4d3c51d7-b76a-4834-a3d7-ecb8a432debb          | OS::TripleO::Compute                                                      | UPDATE_IN_PROGRESS | 2017-06-14T15:31:27 | overcloud-Compute-k7eqwlf2pv64                                                                                                                    |
| 0                                             | 00890028-d03c-4abf-bc3c-a4d305eafefc          | OS::TripleO::Compute                                                      | UPDATE_IN_PROGRESS | 2017-06-14T15:31:29 | overcloud-Compute-k7eqwlf2pv64                                                                                                                    |
| 1                                             | 4d7747a4-bdab-425d-8396-4d193c5684cf          | OS::TripleO::Controller                                                   | UPDATE_FAILED      | 2017-06-14T15:31:30 | overcloud-Controller-7zmfn5uuv4nc                                                                                                                 |
| 2                                             | e0555b88-8414-4354-a918-3551a36b672a          | OS::TripleO::Compute                                                      | UPDATE_IN_PROGRESS | 2017-06-14T15:31:30 | overcloud-Compute-k7eqwlf2pv64                                                                                                                    |
| 2                                             | 6a5f7e6c-1e12-4dba-a92b-19ac22712d35          | OS::TripleO::Controller                                                   | UPDATE_FAILED      | 2017-06-14T15:31:32 | overcloud-Controller-7zmfn5uuv4nc                                                                                                                 |
| 0                                             | f99d6fb1-b011-4024-a988-6342b76497ad          | OS::TripleO::Controller                                                   | UPDATE_FAILED      | 2017-06-14T15:31:34 | overcloud-Controller-7zmfn5uuv4nc                                                                                                                 |
| UpdateDeployment                              | f9d5362e-2eea-4de7-8b44-7a35622f34f3          | OS::Heat::SoftwareDeployment                                              | UPDATE_FAILED      | 2017-06-14T15:34:30 | overcloud-Controller-7zmfn5uuv4nc-1-4jq6hiw2wn6t                                                                                                  |
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+




[stack@director ~]$ echo -e `heat deployment-show f9d5362e-2eea-4de7-8b44-7a35622f34f3`
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{ "status": "FAILED", "server_id": "272bc481-3c01-430d-a6d2-e504de3a6705", "config_id": "36948756-06fb-4945-95af-5646e9684aa8", "output_values": { "deploy_stdout": "Started yum_update.sh on server 272bc481-3c01-430d-a6d2-e504de3a6705 at Wed Jun 14 15:35:06 UTC 2017
Manual upgrade of openvswitch - restart in postun detected
/var/lib/heat-config/heat-config-script/OVS_UPGRADE /var/lib/heat-config/heat-config-script
Attempting to downloading latest openvswitch with yumdownloader
Loaded plugins: product-id
--> Running transaction check
---> Package openvswitch.x86_64 0:2.5.0-14.git20160727.el7fdp will be installed
--> Finished Dependency Resolution
Updating openvswitch with nopostun option
/var/lib/heat-config/heat-config-script
Pacemaker running, stopping cluster node and doing full package update
Stopping Cluster (pacemaker)... Stopping Cluster (corosync)...
Running: yum -q -y update 
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /etc/os-release saved as /etc/os-release.rpmsave
warning: /etc/shadow created as /etc/shadow.rpmnew
warning: /etc/nsswitch.conf created as /etc/nsswitch.conf.rpmnew
warning: /etc/sysctl.conf created as /etc/sysctl.conf.rpmnew
warning: /etc/aodh/aodh.conf created as /etc/aodh/aodh.conf.rpmnew
warning: /etc/heat/heat.conf created as /etc/heat/heat.conf.rpmnew
unable to write 'random state'
warning: /etc/sudoers created as /etc/sudoers.rpmnew
warning: /etc/neutron/neutron.conf created as /etc/neutron/neutron.conf.rpmnew
warning: /etc/nova/nova.conf created as /etc/nova/nova.conf.rpmnew
warning: /etc/neutron/dhcp_agent.ini created as /etc/neutron/dhcp_agent.ini.rpmnew
warning: /etc/neutron/l3_agent.ini created as /etc/neutron/l3_agent.ini.rpmnew
warning: /etc/neutron/plugins/ml2/openvswitch_agent.ini created as /etc/neutron/plugins/ml2/openvswitch_agent.ini.rpmnew
warning: /etc/cinder/cinder.conf created as /etc/cinder/cinder.conf.rpmnew
warning: /etc/neutron/plugins/ml2/ml2_conf.ini created as /etc/neutron/plugins/ml2/ml2_conf.ini.rpmnew
warning: file /usr/share/openstack-dashboard/static/dashboard/js/5ac31e075a3b.js: remove failed: No such file or directory
warning: /var/lib/logrotate.status saved as /var/lib/logrotate.status.rpmsave
Created symlink from /etc/systemd/system/sockets.target.wants/virtlogd.socket to /usr/lib/systemd/system/virtlogd.socket.
warning: file /var/run/OpenIPMI: remove failed: No such file or directory
2670 blocks
yum return code: 127
Starting cluster node
Starting Cluster...
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36

Galera cluster node is not synced.
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32

Galera cluster node is synced.
Waiting for pacemaker cluster to settle
Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
Last updated: Wed Jun 14 15:48:14 2017          Last change: Tue Jun 13 23:17:32 2017 by root via cibadmin on overcloud-controller-2

3 nodes and 127 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.168.140.121     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-192.168.120.127     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 ip-192.168.120.126     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: galera-master [galera]
 Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.190.5       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-192.168.170.120     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-192.168.140.120     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 Clone Set: rabbitmq-clone [rabbitmq]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
 Masters: [ overcloud-controller-2 ]
 Slaves: [ overcloud-controller-0 overcloud-controller-1 ]
 Clone Set: mongod-clone [mongod]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
 Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Daemon Status:
 corosync: active/enabled
 pacemaker: active/enabled
 pcsd: active/enabled
Finished yum_update.sh on server 272bc481-3c01-430d-a6d2-e504de3a6705 at Wed Jun 14 15:48:14 UTC 2017
", "deploy_stderr": "Warning: RPMDB altered outside of yum.
/usr/bin/python: symbol lookup error: /lib64/libnsssysinit.so: undefined symbol: PR_GetEnvSecure
", "update_managed_packages": "false", "deploy_status_code": 127 }, "creation_time": "2017-06-13T22:35:13", "updated_time": "2017-06-14T15:48:15", "input_values": { "update_identifier": "1497454243" }, "action": "UPDATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 127", "id": "f9d5362e-2eea-4de7-8b44-7a35622f34f3" }



Checking the node where this error is reported - seems like yum update has completed.

 nova list |grep 272bc481-3c01-430d-a6d2-e504de3a6705
| 272bc481-3c01-430d-a6d2-e504de3a6705 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.168.120.131 |

[stack@director ~]$ ssh heat-admin.120.131

[heat-admin@overcloud-controller-1 ~]$ sudo yum check-update
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Security: kernel-3.10.0-514.21.1.el7.x86_64 is an installed security update
Security: kernel-3.10.0-327.28.3.el7.x86_64 is the currently running version
[heat-admin@overcloud-controller-1 ~]$ sudo yum update
Loaded plugins: product-id, search-disabled-repos, subscription-manager
No packages marked for update
[heat-admin@overcloud-controller-1 ~]$
Comment 2 Sofer Athlan-Guyot 2017-06-15 09:41:08 UTC 
Hi Sasha,

it's a know issue and the root cause is handled in 1458841.  We are currently waiting for its resolution.
Comment 5 Yurii Prokulevych 2017-09-22 07:36:38 UTC 
Successful update of RHOS-9 on RHEL-7.3 to latest on RHEL-7.4 with openstack-tripleo-heat-templates-2.0.0-60.el7ost.noarch
Comment 6 errata-xmlrpc 2017-09-27 13:08:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2813