Description of problem: While deployment of OSP14 puddle I got error failed running docker-puppet.py for OVN-controller. nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password --env NAME=nova_libvirt --env HOSTNAME=compute-0 --env NO_ARCHIVE= --env STEP=6 --volume /etc/localtime:/etc/localtime:ro --volume /tmp/tmpyUwbNe:/etc/config.pp:ro,z --volume /etc/puppet/:/tmp/puppet-etc/:ro,z --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro,z --volume /var/lib/config-data:/var/lib/config-data/:z --volume tripleo_logs:/var/log/tripleo/ --volume /dev/log:/dev/log --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:z --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro 192.168.24.1:8787/rhosp14/openstack-nova-compute:2018-08-20.1", "2018-08-23 06:31:27,357 ERROR: 19154 -- Failed running docker-puppet.py for ovn_controller", "2018-08-23 06:31:27,358 ERROR: 19154 -- Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend", "Notice: Compiled catalog for compute-0.localdomain in environment production in 1.21 seconds", " Skipped: 3", " Total: 3", " Config retrieval: 1.43", " Total: 1.43", " Last run: 1535005887", " Config: 1535005885", "2018-08-23 06:31:27,358 ERROR: 19154 -- + mkdir -p /etc/puppet", "+ '[' -n file,file_line,concat,augeas,cron,vs_config,exec ']'", "+ TAGS='--tags file,file_line,concat,augeas,cron,vs_config,exec'", "+ origin_of_time=/var/lib/config-data/ovn_controller.origin_of_time", "+ touch /var/lib/config-data/ovn_controller.origin_of_time", The stack creation completed by the configuration of ovn-controoler failed. openstack stack list +--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+ | 9af261ab-6ac7-4a00-bfa4-9623fae3049b | overcloud | 20aec16013ea4b19b5d1fb84fbcdce7d | CREATE_COMPLETE | 2018-08-23T06:05:43Z | None | undercloud) [stack@undercloud-0 ~]$ openstack workflow execution list --filter state=ERROR +--------------------------------------+--------------------------------------+----------+-------+------------------------------+---------------------+---------------------+ | ID | Workflow ID | Workflow name | Workflow namespace | Description | Task Execution ID | Root Execution ID | State | State info | Created at | Updated at | +--------------------------------------+--------------------------------------+----------+-------+------------------------------+---------------------+---------------------+ | 1339aaf3-714d-4c7f-ba5a-585e51f69650 | 216a1ba9-f7fa-49cc-8e2e-b6a4ea330169 | tripleo.deployment.v1.config_download_deploy | | | <none> | <none> | ERROR | Failure caused by error i... | 2018-08-23 06:22:08 | 2018-08-23 06:34:23 | | c07561d5-6df7-49ea-a94b-ad60afca95f1 | b40068a3-91a4-42ee-a021-4c38fd017446 | tripleo.messaging.v1.send | | sub-workflow execution | e059441a-9b76-4f9d-bdd3-62bc7295a5c8 | 1339aaf3-714d-4c7f-ba5a-585e51f69650 | ERROR | Workflow failed due to me... | 2018-08-23 06:34:18 | 2018-08-23 06:34:22 | +--------------------------------------+--------------------------------------+- Version-Release number of selected component (if applicable): OSP14 -p 2018-08-21.2 (undercloud) [stack@undercloud-0 ~]$ rpm -qa |grep -i director rhosp-director-images-14.0-20180820.1.el7ost.noarch rhosp-director-images-ipa-14.0-20180820.1.el7ost.noarch (undercloud) [stack@undercloud-0 ~]$ rpm -qa |grep -i triple openstack-tripleo-puppet-elements-9.0.0-0.20180801001359.d9df3a3.el7ost.noarch openstack-tripleo-image-elements-9.0.0-0.20180801003129.39e37fe.el7ost.noarch ansible-role-tripleo-modify-image-1.0.0-0.20180803113326.3b4e2e6.el7ost.noarch openstack-tripleo-validations-9.2.1-0.20180726214014.7627d15.el7ost.noarch openstack-tripleo-common-9.2.1-0.20180803214330.el7ost.noarch python-tripleoclient-10.4.1-0.20180803113705.1e3bb6e.el7ost.noarch openstack-tripleo-common-containers-9.2.1-0.20180803214330.el7ost.noarch python2-tripleo-common-9.2.1-0.20180803214330.el7ost.noarch openstack-tripleo-heat-templates-9.0.0-0.20180804083746.el7ost.noarch python-tripleoclient-heat-installer-10.4.1-0.20180803113705.1e3bb6e.el7ost.noarch ansible-tripleo-ipsec-8.1.1-0.20180405121919.325d233.el7ost.noarch puppet-tripleo-9.2.1-0.20180731173658.dd67adb.el7ost.noarch (undercloud) [stack@undercloud-0 ~]$ rpm -qa |grep -i ovn puppet-ovn-13.2.0-0.20180802202238.6af8ce1.el7ost.noarch How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
The reason for the failure is because of the warning messages which "ovs-vsctl" displays when puppet vswitch executes the command "/usr/bin/ovs-vsctl list Open_vSwitch .". *** PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5) PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4) PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5) PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4) *** Because of these warning messages, the code here - https://github.com/openstack/puppet-vswitch/blob/master/lib/puppet/provider/vs_config/ovs.rb#L41 is Nil. Ideally ovs-vsctl shouldn't display these warning messages. As a workaround we need to mount "/usr/lib64" to the ovn_controller container during puppet confi step.
Similar issue for neutron-openvswitch-agent in bug #1619387, current workaround is to install that optional library, and work in progress upstream not to die on stderr output
The fix is here - https://review.openstack.org/#/c/597023/
The fix is merged u/s master and u/s rocky in puppet-vswitch repo.
Fix verified: 14.0-RHEL-7/2018-09-06.1 (overcloud) [stack@undercloud-0 ~]$ rpm -qa |grep puppet-vswitch-9. puppet-vswitch-9.3.1-0.20180831182046.20c9fad.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045